Dealing with Google Drive's horrible duplicate file problem

[superluig164]

Hello there,

I would just like to express some concerns with Google Drive, and also hopefully kickstart something to have FFS work around Drive's stupid problems until such time as the Drive team realizes how stupid they are.

So, basically, if you don't already know, Google Drive allows multiple files to have the same filename. It is supposed to merge them into one file with different version, but as I have seen happen multiple times, this isn't always the case.

First of all, why the hell does a file management platform even allow that in the first place?! It's just confusing no matter which angle you look at it. Either it forces a user to look at the file contents to determine which is which all the time, or it causes computer programs to get confused about which file to use, and this includes FreeFileSync.

Normally, when uploading a file to Google Drive manually, it asks if you'd like to merge it with the existing file or save it as a new file (which appends a (1), like all filesystems should.) However, it's still possible to have multiple files with the same name simply by renaming files or, seemingly, also by using FreeFileSync.

I use FreeFileSync to keep my Minecraft information synced between multiple computers. However, sometimes I will be forced to go into Drive myself and delete duplicates, because FreeFileSync just refuses to download either file, without either a) checking the file contents (in my case, the duplicates are almost always either the same as each other, or one of them is newer) or b) checking the date modified of each file and using the newest file. I could see an option being added to handle this, or at least FreeFileSync giving a prompt during (or right after) sync to ask which file to use. Instead, it just refuses, and this can cause problems.

I'd like to have my Minecraft sync just run in a script every time I close Minecraft, but I can't, because if all the updated world files aren't downloaded correctly on another system, then my world will be messed up, and now I will have conflicts. I really feel that there are more elegant solutions than what is currently happening.

FreeFileSync does handle upload correctly, though. I've only ever seen duplicates get created during batch upload through Google Drive (or using its horrible desktop app.) but sometimes I do see them crop up after FreeFileSync syncs, so I'm not entirely sure. Either way, FreeFileSync has to do something for now, because Drive isn't.

[Zenju]

Apparently Google Drive models a key/value store, which is more generic and thous only partially compatible with a hierarchical file system. Most notably, Google Drive allows duplicate file names (distinguished by item id) and treats all data the same, consequently even a file rename changes the modification time. This has far-reaching consequences, e.g. it's not possible to optimistically create a file and test if it already existed by evaluating the error response. This is mitigated to some degree by Google Drive supporting change notifications, but still such situations can only be detected with "high probability" rather than 100% certainty, as one would except from ordinary file systems.
FreeFileSync tries to make the best out of this situation by primarily making sure to avoid data corruption, i.e. fail with an error upon finding duplicate file names, rather than say, just select any one of the candidates and call it a day and by being internally consistent regarding the file state. Why Google chose to model key/value rather than hierarchical file system is beyond me.

[superluig164]

Yep, I understand how it works. My issue is that it ruins file-sensitive sync operations (like Minecraft worlds). So may I suggest an option or something to allow FreeFileSync, in certain situations, (say, when both files have the exact same data content, which has happened to me before) to just choose one of the files, or perhaps whichever one is newer. Maybe even an option to automatically remove the duplicate to prevent further problems. Either that, or just prompt the user to choose a duplicate and delete/rename the other duplicates after each sync, to prevent further problems.

[superluig164]

Apologies for the bump, but I realise this issue is essentially solved by syncing directly to and from Google Drive using said new feature. FreeFileSync being in charge means no duplicates will ever be uploaded. Before, I was syncing to a folder on my computer, which would then be synced by Google Drive's desktop app, which apparently sucks at this stuff.

The Google Drive team is still stupid, but at least everything worked out now that FFS can sync directly to Drive.