Sync with Google Drive creates duplicate files.

[CharIlia]

I am using the donation edition.
I syncronize (with Mirror) a local folder with Google Drive. During the synchronization, Freefilesync creates dublicate files (two or more files with the same name) on the Gdrive. This creates errors during the next synchronization like this:

Cannot find "(file name)". The name "(file name)" is used by more than one item in the folder.

The only solution is to delete the dublicate files (the files with the same names). But this is not a usable solution because my folder has over 18000 subfolders and over 500000 files!

I use 5 parallel file operations for gdrive and 100 for the local drive.(I don't know if this creates the problem but fewer than this makes the synchronization very slow)

The above problem makes Freefilesync unusable for backup! Isn't any solution to this? Why Freefilesync can't check if the files already exist on gdrive?

Please provide any help!

[BWal]

I came here to report the same bug, As best I can tell what is happening is I have multiple computers trying to update the file at the same time and google drive is making duplicate copies. Google drive stores files in a weird way and so having multiple files with the same name is technically allowed because it is just storing a record with a name associated with it.

This happens to me after a couple of days and I periodically need to clean up any duplicate files on google drive to have the process continue. The simple solution for this would be for the sync tool to just delete the oldest, generally the files are identical.

I was hoping this would be fixed by now but it has persisted for well over 15 versions.

[Zenju]

I was hoping this would be fixed by now but it has persisted for well over 15 versions.

There is nothing to fix here. The behavior is exactly as Google wants it to be: https://webapps.stackexchange.com/questions/144039/google-drive-weirdly-allow-files-folder-with-the-same-name-in-the-same-location

Best you can do is to somehow make sure that only a single sync is running against a particular Google Drive folder at a time.

PS: If it were a local sync, FFS could work around by placing sync.ffs_lock files in the base directories to serialize access of multiple FFS instances. But this requires transactional file creation and is therefore not possible for Gdrive.

[CharIlia]

After this I stoped using Google drive for backup. I now use a ftp service provider.

[BWal]

I was hoping this would be fixed by now but it has persisted for well over 15 versions.

There is nothing to fix here. The behavior is exactly as Google wants it to be: https://webapps.stackexchange.com/questions/144039/google-drive-weirdly-allow-files-folder-with-the-same-name-in-the-same-location

Best you can do is to somehow make sure that only a single sync is running against a particular Google Drive folder at a time.

The behavior of allowing multiple files on google drive isn't the issue, Google gonna google. I'm not asking you to fix google. The problem is that it becomes a blocking issue for FreeFileSync as it's unable to perform any sync functions when there is 2 files with the same name on google drive. I was hoping there was something that we could add to the program to any or multiple of the following
1) set a rule to clean up duplicate names
2) or always overwrite if duplicate names
3) or set a rule that it automatically deletes the older of the 2 files if a duplicate is found.
4) Block FFS from changing files on target when another instance of FFS is currently running a sync operation on the target and try again in a user defined amount of time.

[CharIlia]

Amen!!!

[Zenju]

None of these suggestions work because Google Drive does not offer transactional file creation. There's just a gap between checking if a file exists, and creating a new one. It's a race condition by Google's design.

[BWal]

None of these suggestions work because Google Drive does not offer transactional file creation. There's just a gap between checking if a file exists, and creating a new one. It's a race condition by Google's design.

You are focusing on the preventing it happening part.
While some suggestions where to prevent it happening I understand how that can not be possible.

I am mostly asking for a way to handle/clean up the condition after it happens.
Currently FFS just gets locked in an error condition and requires manual intervention.

I am not sure how the google drive API works at this time without looking into it personally.
But please explain to me why FFS is able to detect that more than one file exists with same name but is unable to delete one or both of those files if that happens. I would 100% be ok with setting a sync rule that just says if duplicate files exist that are older just nuke both and upload new file.

[BWal]

https://rclone.org/commands/rclone_dedupe/

Can something akin to this be integrated or implemented into the program?

"--dedupe-mode newest - removes identical files then keeps the newest one."

[Zenju]

I would 100% be ok with setting a sync rule that just says if duplicate files exist that are older just nuke both and upload new file.

This might be good enough for you, but it isn't good enough as a general rule. Just picking one of a set of equally-named files is like gambling with files.

[BWal]

I would 100% be ok with setting a sync rule that just says if duplicate files exist that are older just nuke both and upload new file.

This might be good enough for you, but it isn't good enough as a general rule. Just picking one of a set of equally-named files is like gambling with files.

Which is why it could be an optional rule. Similar to how I have it set in my custom sync to always overwrite google drive in the event of a difference.

For my user case I'm trying to sync game config files between multiple computers so that I can switch to any computer and my settings will move over, this game lacks any form of cloud saves. There is never a condition that the same file name will be anything else. And I always want the newest file as that means I played on that computer most recently. In the absolute worst cast scenario the worst thing I could have is if an older file overwrites a newer one I would go back to a previous configuration from the day before. Which is a minor inconvenience at best. But what is major inconvenience have the syncs just suddenly stop working for a long period of time and fail silently because I have them running in unattended mode and then have the files get massively out of sync then having to manually log into google drive to find all the files that got duplicated and clean it up so that the syncs will work again.

Could we not have a condition where we chose what to do in the case of a duplicate file exists?
If the user is concerned about losing files/data then they can chose to skip files with duplicate names.

[phpjunkie]

I came here to report the same bug, As best I can tell what is happening is I have multiple computers trying to update the file at the same time and google drive is making duplicate copies. Google drive stores files in a weird way and so having multiple files with the same name is technically allowed because it is just storing a record with a name associated with it.

This happens to me after a couple of days and I periodically need to clean up any duplicate files on google drive to have the process continue. The simple solution for this would be for the sync tool to just delete the oldest, generally the files are identical.

I was hoping this would be fixed by now but it has persisted for well over 15 versions.

It isn't FreeFileSync it is Google Drive, and it is a Revision Control System. It saves older copies of files. If you screw up and overwrite a file you still have a copy of the previous revision. Google doesn't store files like what you would see in your hard drive. Every file you upload to Google Drive is stored in a database and isn't written to any kind of file system.

Screenshot 2024-11-09 142106.png (52.02 KiB) Viewed 387 times

Screenshot 2024-11-09 142344.png (71.28 KiB) Viewed 385 times

Google will automatically remove older revisions of the file itself.

[BWal]

I came here to report the same bug, As best I can tell what is happening is I have multiple computers trying to update the file at the same time and google drive is making duplicate copies. Google drive stores files in a weird way and so having multiple files with the same name is technically allowed because it is just storing a record with a name associated with it.

This happens to me after a couple of days and I periodically need to clean up any duplicate files on google drive to have the process continue. The simple solution for this would be for the sync tool to just delete the oldest, generally the files are identical.

I was hoping this would be fixed by now but it has persisted for well over 15 versions.

It isn't FreeFileSync it is Google Drive, and it is a Revision Control System. It saves older copies of files. If you screw up and overwrite a file you still have a copy of the previous revision. Google doesn't store files like what you would see in your hard drive. Every file you upload to Google Drive is stored in a database and isn't written to any kind of file system.
Screenshot 2024-11-09 142106.png
Screenshot 2024-11-09 142344.png
Google will automatically remove older revisions of the file itself.

That is not what was happening. It was 2 files with exact same name in same "folder" it would break sync if this condition happened. It happened because more than one pc would try to upload a file at the exact same time. It was basically a race condition. I tried several things to prevent this but it would of likely required more strict scheduling and it was 6 computers all trying to sync at staggered intervals.

The solutions that worked was either use ftp site or just use windows file share drive mapping.

I only have the google drive sync run on 1 pc for when I need access outside my own network.