Mirror/Sync with Google Drive File Stream

[jayjlawrence]

I am sync'ing to/from a Google Drive File Stream source on a Mac. This works fine for normal files but there are "cloud" file types that cannot be exported from the File Stream. Files such as .gsheet and .gdoc. These files are "in the Google Cloud" for use by apps "in the Google Cloud" and getting these files has to be done through an export from their related application.

The Google Cloud file types can be identified by extensions. So these we can exclude in FreeFileSync as expected. I have come across other application types where they do not force an extension and will cause sync errors.

12:22:38 Error Cannot open file "/Volumes/GoogleDrive/Team Drives/<path>/<filename>".
Error Code 45: Operation not supported [open]

You can experience this error at the OS level by trying to copy this file to your desktop and receive this error "The operation can’t be completed because it isn’t supported." [ OK ] ... thanks guys! Where's the "[ Not OK ]" button???

Since this is holding up my RealTimeSync job I really only have two options today. First is, on a case by case basis react to this error and give an extension to the file. Second is activate "Ignore errors" feature. Been there, done that, had a f@@king disaster because of it, will never turn that feature on again. @zenju I can tell you offline what happened if you're interested.

In my ideal world I would like to see these two features:
- option to skip Error Code 45 errors
- a log generated of all Error Code 45 files: path and attributes(timestamp, size, etc.)

The first feature is to allow the sync to complete. The second I could feed to some process to either advise users that these files are unavailable and why or feed to some sort of export script that would script the applications to export those files somewhere somehow.

I hope this info helps and if you'd like more info/assistance just ask and I'll be happy to provide.

[Zenju]

Second is activate "Ignore errors" feature. Been there, done that, had a f@@king disaster because of it, will never turn that feature on again. @zenju I can tell you offline what happened if you're interested.

Yes, that would be interesting. "Ignore errors" *should* always be safe in that it just skips the items it cannot process and I'm not aware of any fundamental problems in this regard.

In my ideal world I would like to see these two features:
- option to skip Error Code 45 errors
- a log generated of all Error Code 45 files: path and attributes(timestamp, size, etc.)

Generally I'd say this is best solved via the exclude filter if the problematic files are known beforehand and via "ignore errors" if they are yet unknown. The error codes and paths are then automatically written to the sync log. But I don't know what kind of problems you had with "ignore errors" earlier.

[jayjlawrence]

In my ideal world I would like to see these two features:
- option to skip Error Code 45 errors
- a log generated of all Error Code 45 files: path and attributes(timestamp, size, etc.)

Generally I'd say this is best solved via the exclude filter if the problematic files are known beforehand and via "ignore errors" if they are yet unknown. The error codes and paths are then automatically written to the sync log. But I don't know what kind of problems you had with "ignore errors" earlier.

It's a bit of an edge case but one that could be significant. The problem files were Google Cloud files (which cannot be copied from the drive) that did not have an extension. Therefore I had no rule to filter with. If I had a lot of files that would generate this error code then I could be clicking "Ignore" all day. If you feel so inclined this could be a useful advanced feature setting.

[jayjlawrence]

Second is activate "Ignore errors" feature. Been there, done that, had a f@@king disaster because of it, will never turn that feature on again. @zenju I can tell you offline what happened if you're interested.

Yes, that would be interesting. "Ignore errors" *should* always be safe in that it just skips the items it cannot process and I'm not aware of any fundamental problems in this regard.

I guess I don't mind sharing here. So I am using FreeFileSync for a few interesting tasks. I've used various tools over the years and I do find that FFS is very good at striking a balance between ease of use and being industrial strength.

The problem came about doing a bi-directional sync between two "cloud" folders using an iMac in my office. Normally these cloud folders are resident on your computer and you're doing a simple directory to directory sync. However, Google Drive File Stream only keeps a partial cache of your total cloud drive - and that makes sense. The original folders came from a Nextcloud install that is used to manage file content on a number of webservers and it does this job really quite well.

On D-Day what seems to have happened is that the Google Drive File stream was temporarily disrupted and the entire volume was "unavailable". FreeFileSync flagged this as a major difference and issued the warning to the log. Ignore Errors also means Ignore Warnings. That warning was ignored and the sync proceed as if all the directories were deleted in Google Drive so therefore all the directories were deleted on the other drive too.

The issue caused a bit of chaos but fortunately no data was lost - just some tense moments, a bit time and a slightly hurt sense of pride. Sync cloud resources very carefully. I am sticking to uni-directional mirroring for now.

I have the logs for FFS and Google DriveFS if you'd like to see them.

[Zenju]

On D-Day what seems to have happened is that the Google Drive File stream was temporarily disrupted and the entire volume was "unavailable". [...] That warning was ignored and the sync proceed as if all the directories were deleted in Google Drive so therefore all the directories were deleted on the other drive too.

Wow, that's a fatal bug in Google Drive, namely in that it returns false information (empy folders) instead of an error. What's insidious about this case is that there's no way to detect this from FreeFileSync or any other application. Had Google Drive returned an error during folder traversal as it should have, FFS would have skipped the folders, logged the errors and no data loss had occured.

[jayjlawrence]

Wow, that's a fatal bug in Google Drive, namely in that it returns false information (empy folders) instead of an error. What's insidious about this case is that there's no way to detect this from FreeFileSync or any other application.

There's one thing to bear in mind. The actual target folder was missing as was it's parent. Perhaps consider handling a missing target folder a fatal error that cannot be ignored. If the target folder is created then definitely consider a missing parent folder as a fatal error.

[Zenju]

There's one thing to bear in mind. The actual target folder was missing as was it's parent.

A missing target folder does not lead to data loss with FreeFileSync, same is true for a missing source folder. The bug with Google Drive as I understood you correctly was that there were no missing folders, the base folders existed, but that their folder content incorrectly returned as empty instead of failing due to network connection problems.