Glitch on Sync/Mirror <Item not existing>

[t0ma5]

Hi guys. I use FFS to sync/mirror two Google Team Drives (AKA Shared Drives) and it used to work perfectly, but not anymore, this is my problem:

FFS was working just fine until two days ago when out of the blue, without changing any configuration or setting, FFS stopped reading files correctly in both folders (source and destination) so it tries copy and delete files without any need for it.

FFS claims that many files don't exist on both sides by displaying this message: <Item not existing>
I open the team drives on my browser and I can see all the files that FFS can't see, I can open them and download them without a problem, but FFS says the items are not there.

Also it reads the destination folder with mistakes claiming some files/folders don't exist and it need to copy them (but again, I can see/use those files with my internet browser). So it read both (source and destination) wrongly and suggests actions that are not necessary to perform.

The folders in question (source and destination) have about 1.5TB of data. I am using Windows 10 and FFS v11.3 (same problem with v11.2)

Thanks for your help

EDIT: A quick fix is to delete the DB file in
C:\Users\YOU_NAME\AppData\Roaming\FreeFileSync\GoogleDrive
and run all the scans again to rebuild a new DB that works for some time until it gets corrupted again. Rinse and repeat.

[therube]

the source folder

Is that a "real" (i.e., not some "virtual") local folder on your HDD?

[t0ma5]

the source folder

Is that a "real" (i.e., not some "virtual") local folder on your HDD?

no, both folders are TEAM DRIVES and it worked just fine for weeks, no problem at all (until now).
I even tried GoodSync to see if I get the same reading errors and that's not the case, only with FFS

[t0ma5]

UPDATE: Re-installing FFS solved the issue. I reloaded my saved profiles (untouched) and they worked again as expected. So it was not a miss configuration but some sort of glitch...

[t0ma5]

It keeps happening, after certain time FFS starts to give errors and the only way to "fix" it is re-installing :(

EDIT: A quick fix is to delete the DB file in
C:\Users\YOU_NAME\AppData\Roaming\FreeFileSync\GoogleDrive
and run all the scans again to rebuild a new DB that works for some time until it gets corrupted again. Rinse and repeat.

[Zenju]

UPDATE: Re-installing FFS solved the issue.

By reinstalling you deleted the local google drive database file that apparently got corrupted. You could achive the same effect by opening the Google Drive settings in FreeFileSync, "Disconnect" and "Add connection".

The real question is why the database file got corrupted in first place. Are you able to simplify and replicate this issue? If yes, I can have a look and make sure it gets fixed.

[t0ma5]

The real question is why the database file got corrupted in first place. Are you able to simplify and replicate this issue? If yes, I can have a look and make sure it gets fixed.

Thanks, I will try your tip next time it happens. And no, I can't replicate it. It seems to happen after X time using the application :(

[Zenju]

I don't think this is a random bug. Most likely there is a specific sequence of file operations happening on the shared drives that send out a number of change notifications that are not correctly processed by FFS. How this could be I can't imagine, so a repro is essential. Alternatively, full access to a Google Drive account that exhibits the bug might do, but I understand this is probably not feasible.

[jpgill86]

I have experienced issues with Google Drive a few times like those described by t0ma5, though I was using a Google Shared Drive as a backup for local files, rather syncing two Drives.

The first time it happened, it was very painful and nearly cost me data, since I had a nightly batch script running two-way synchronization. Gigabytes of files were just deleted one day! Luckily, I had other backups! I changed the two-way sync mode on my nightly script to a one-way update instead; this worked fine initially, but sometime later duplicates of files appeared in Drive, since it apparently thought those files weren't on the Drive any more. I deleted those duplicates by hand and wondered what was going on.

Apparently I'm a slow learner, because I let it happen a third time in one-way update mode, so duplicate files appeared in Drive again. When I asked it to do a comparison in two-way mode, it showed that it thought many of the Drive files were deleted, and it wanted to remove them from my local machine. Like the first time, if I had still been using two-way mode for my nightly script, I would have lost files.

A little frustrated, I just walked away from it for a couple weeks, letting those duplicated files sit there. But today I felt like trying to solve the problem, and I quickly stumbled upon this thread.

Following Zenju's suggestion (thank you for this software by the way; despite this post, I absolutely love it, and am already a regular donor!), I tried disconnecting and reconnecting my Google Drive account. When I attempted a two-way comparison again, apparently a new sync.ffs_db was created in the Shared Drive. I received this error:

"Cannot open file <...>\sync.ffs_db. The name sync.ffs_db is used by more than one item in the folder."

Indeed, there were now two copies of the file in the Shared Drive.

I then deleted the newer copy of sync.ffs_db and reran the two-way sync. This time, there was no error, and the file comparison gave the expected results, albeit somehow ignoring the duplicated files on Drive.

The following may be an independent problem, or it might be caused by the changing sync.ffs_db file:

After getting that to work, the file comparison helped me realize that some files needed to be deleted from the Shared Drive; they were mistakenly placed there by a collaborator. As expected, these files were showing in my two-way comparison as being ready to be copied locally. Since I did not want to keep them, I deleted them in the Shared Drive manually, but when I re-ran the comparison, it still claimed these deleted files were in the Shared Drive, ready to be synced. I tried emptying the Shared Drive's trash, but that made no difference. When I tried to actual perform the sync (not just a comparison) on the non-existent file, I got this message:

"Cannot read file "<name of my file>. File not found: <Google Drive id for the file>"

So, next I tried deleting the old sync.ffs_db in Drive also. Now there were not two or one database files, but zero. This had no effect on the comparison, and a new sync.ffs_db was not immediately created in its place.

Finally, I disconnected and reconnected my Google Drive account a second time. This time, the comparison refreshed accurately, and a new sync.ffs_db was created.

As of this writing, all is working as expected. However, since I anticipate there being another issue like this in the future, I have disabled all automatic nightly backups to Google Drive and will monitor for this problem in the future whenever I run a manual backup. That way, I can avoid losing files with a bad two-way sync, or duplicating files with a bad one-way update.

But, of course, I'd love to see the problem identified and fixed! Unfortunately, I don't know how to reproduce it consistently. If it would be helpful to, say, preserve and share my sync.ffs_db file when this happens next before doing the disconnect-and-reconnect workaround, I'm happy to try that.

P.S.: I always try to keep my installation of FreeFileSync up-to-date. Presently, I'm at version 11.4.

[t0ma5]

Hi jpgill86!

I have encountered this problem many times already, the more I run syncs the more often it happens.
For the time being my work-around is to delete the DB file in
C:\Users\YOU_NAME\AppData\Roaming\FreeFileSync\GoogleDrive
and run all the scans again to rebuild a new DB that works for X time until it gets corrupted again.
I have to check every scan result manually before going ahead with the sync to prevent mistakes from happening as you described.

[Zenju]

@jpgill86:
These seem to be different issues than what t0ma5 was describing, which is "FFS reporting <Item not existing>, when the file in fact exists".

I deleted them in the Shared Drive manually, but when I re-ran the comparison, it still claimed these deleted files were in the Shared Drive, ready to be synced.

This is "normal" for Google Drive. It takes a short moment until Google Drive reports changes that happen externally (= outside of FFS), such as file deletions.

I received this error:
"Cannot open file <...>\sync.ffs_db. The name sync.ffs_db is used by more than one item in the folder."
externally (= outside of FFS).

Google Drive allows multiple items that have the same name. So it can happen if two FFS instances are syncing the same folders more or less at the same time that both try to update the sync.ffs_db file:

1. both instances delete the old sync.ffs_db (one succeeds, the second skips with nothing to do)
2. both instances create a sync.ffs_db


The proper solution for both such issues is some form of folder locking that makes sure only one FFS instance is ever accessing a Google drive folder at a time. This is implemented via the sync.ffs_lock file for local drives and network shares. Unfortunately no such mechanism exists for Google Drive.

[KinsleyProk]

maybe you would take a look to GoodSync or Gs RichCopy360 , I think you will not face this problem with these tools , search both

[t0ma5]

maybe you would take a look to GoodSync or Gs RichCopy360 , I think you will not face this problem with these tools , search both

Those options are painfully slow compared to FFS

[Zenju]

EDIT: A quick fix is to delete the DB file in
C:\Users\YOU_NAME\AppData\Roaming\FreeFileSync\GoogleDrive
and run all the scans again to rebuild a new DB that works for some time until it gets corrupted again. Rinse and repeat.

Is there some way to reproduce this, or some test account that I could analyze from my end?

[t0ma5]

Is there some way to reproduce this, or some test account that I could analyze from my end?

This is exactly what I do:

• I have 3 SHARED DRIVES on my Google Drive, let's call them A, B and C
• I manually upload content to A on daily basis. Mostly folders with a few large video files inside (2-5GB)
• I sync A to B and C using FFS and it works without a problem, for some time.
• After a while, something goes wrong and FFS acts as if B and C were empty, it wants to copy everything from A to B and C again. But on the web-browser I can see B and C are not empty, all the files are there but are somehow ignored by FFS
• To fix it I have to delete "mymail@gmail.com.db" and re-scan everything again. Then it works fine for some time until the problem appears again.

[Zenju]

• After a while, something goes wrong and FFS acts as if B and C were empty

This is the interesting part. FFS apparently receives change notifications from Gdrive that report the files as being "removed". Question is: Is this the case and if yes what exactly triggers these change notifications? E.g. is the shared drive disconnected, and connected again, or some permission/owner change of the shared drive or one of these folders. Or something else?

[t0ma5]

E.g. is the shared drive disconnected, and connected again, or some permission/owner change of the shared drive or one of these folders. Or something else?

No idea but I can tell you that I use those extra SHARED DRIVES as mirror backups ONLY. No other app has access to them in any way. The only changes I make to drives B and C are those made by FFS sync, absolutely nothing else. I don't even access them via web browser until FSS tells me they are empty and I go and check manually.

[Zenju]

In case you somehow manage to reproduce this with as *few steps as possible*: Try to send me the list of change notifications that FFS is receiving as follows:

1. Get the current "start page token". This represents a current snapshot of your Google Drive state: https://developers.google.com/drive/api/v3/reference/changes/getStartPageToken

2. Reproduce the issue of FFS not seeing an item, although it is there and note its ID, which is part of the share link.

3. Get the list of changes since step 1: https://developers.google.com/drive/api/v3/reference/changes/list

[Zenju]

No idea but I can tell you that I use those extra SHARE DRIVES as mirror backups ONLY.

Shared drives means someone else is also having access to these drives? Maybe they are doing something with it?

[t0ma5]

Shared drives means someone else is also having access to these drives? Maybe they are doing something with it?

Yes and No. Let me explain.
The SHARED drives belong to me, I gave no access to anyone. Of course the Google Suite admin of each organization has access to them. I see what your logic is here, but the problem with FFS saying that those drives are empty happen at the same time with all the SHARED DRIVES administered by different organizations. This leads me to believe the issue is from FFS' side.

[Zenju]

Whatever it is, in order to get it fixed we either need steps to reprocudce, or at least the list of change notifications (which hopefully and likely give a clue).

[Zenju]

We could even do this retroactively, since Google Drive change notifications are persistent. So the next time you encounter the situation that a file exists but is not shown in FFS, don't touch anything, but retrieve its ID. We'll then try to extract the change notifications via the Gdrive v3 API as outlined above.

[t0ma5]

We could even do this retroactively, since Google Drive change notifications are persistent. So the next time you encounter the situation that a file exists but is not shown in FFS, don't touch anything, but retrieve its ID. We'll then try to extract the change notifications via the Gdrive v3 API as outlined above.

Sorry, the instructions don't mean much to me, there are many fields to complete and I have no idea. I don't know what do with this result

{
 "kind": "drive#changeList",
 "newStartPageToken": "341XXX",
 "changes": []
}


[Zenju]

Do you have a file ID for a file that is not shown as existing in FFS?

[t0ma5]

Do you have a file ID for a file that is not shown as existing in FFS?

not now since I "FIXED" the database yesterday for the last time. In a few days I will face the problem again. What exactly should I do then? Happy to send u the logs but I need further instructions, pls! :)

[Zenju]

No prob. I'll give you further instructions once we're there.

[jpgill86]

Hello again! I've been following along. :-)

Thank you Zenju for replying to my earlier post back in December. While I agree that I had multiple symptoms, suggesting possibly multiple issues, I still think that what t0ma5 has been describing is among them.

This has continued to occur from time to time, unpredictably, for me since I last posted in December, with files occasionally being identified by FFS as removed from my Shared Drive despite them being there. In fact, it just happened for two files.

My approach for dealing with this has been not to delete the database file, as t0ma5 has suggested, but to just disconnect and reconnect to my Google account. I think that has worked every time for me.

I read this thread this morning and hoped to be able to contribute next time it happened to me, so when a few minutes ago FFS reported two files as deleted, I wanted to investigate it a bit more. I checked the timestamps on the supposedly deleted files, as reported by Google Drive, and noticed that the creation dates were April 21, despite the fact that the two files are much older and haven't been touched since before that date. In fact, the last modified dates precede the creation dates.

Intent on getting a file ID, as Zenju suggested, I clicked on the files in my browser to open them, so that I could extract the ID from the URL. Not only did this update the last opened date on Google Drive, but it also fixed the comparison in FFS, so that the files were no longer reported as deleted. I did this for the two files one at a time, and they were fixed one at a time.

I went ahead anyway with using the Google API to retrieve a changes list, but I guess that since my page token was generated after I had inadvertently fixed the problem files and had made no other changes, the list was empty.

Just to see what would happen, I also tried the Files: get API using the file ID of one of the two files, but this didn't provide anything illuminating (just these fields: kind, id, name, mimeType, teamDriveId, driveId).

[jpgill86]

It's happened again! And this time I haven't fixed it yet, so instructions on what to do next are welcome.

I just ran a comparison on a different folder in the Shared Drive, and I'm seeing 5 files with erroneous comparisons. I also ran a comparison on a different Shared Drive (owned by the same organization), and I'm seeing 14 files with bad comparisons there too.

These are by no means the whole file sets. The files indicated as changed seem to just be a random selection, though in one case it's all files in a small folder, along with the folder itself. Curious to see if the issue with the folder was the directory itself or each file within, I merely opened the folder in my browser, and that cleared up the bad comparison for that folder and its contents.

I should correct something I said in my last post. I repeatedly said that FFS reported the files as deleted. However, I had my comparison mode set to Update, saw that it wanted to push the local files to the remote, and assumed that if I switched to Two-Way it would say that the files were deleted. I just tried switching my comparison to Two-Way for these new cases, and it still wants to copy local files to the Shared Drive, as if they are brand new and are absent from the Drive (even though they are not), rather than delete the local files because it thinks the remote's copies were deleted. It's as if those files on the Shared Drive are invisible to FFS, and it has forgotten they were previously copied there, so FFS thinks it needs to copy them now.

Anyway, I will wait to fix these problems (I would normally dis- and re-connect to my Google account), and I look forward to some instructions on how to help further.

[Zenju]

I'm too busy right now to respond in detail, but maybe this highlevel overview is enough and you can figure out the missing parts:

1. get the "start page token" that represents the current state:
viewtopic.php?t=7827&p=29595#p29547

2. get the list of changes (also explained by the link)
as input, use the start page token and make it "somewhat" smaller
how much? I don't know, but you can test this and see what you get for different values
Goal is to make it small enough to get change notifications that include the one that lead FFS into assuming a particular file (note the ID!) was deleted, but large enough to not get flooded with irrelevant changes.

[Zenju]

Use these parameters for the "list" command:

    
pageToken: from step 1

fields: kind,nextPageToken,newStartPageToken,changes(kind,changeType,removed,fileId,file(trashed,name,mimeType,ownedByMe,size,modifiedTime,parents,shortcutDetails(targetId)),driveId,drive(name))
          
includeItemsFromAllDrives: true

pageSize: 1000

spaces: drive

supportsAllDrives: true