Clarification on LockDirectoriesDuringSync / sync.ffs_lock

[literalzebra]

Hi --

I use FFS (on a Mac) to sync one external hard drive to another nightly, both accessed as mounted SMB volumes over a network.

Occasionally, for mysterious OS/network reasons that have nothing to do with FFS, the source volume is mounted but appears entirely empty, and I need to variously reboot my computer and router to get its contents to appear again.

I suddenly worried -- what if this happened nightly, that FFS saw the source as empty, and thus deleted everything in the destination? But this has never actually happened, so I wondered if something was protecting against it.

Then I realized it must be the sync.ffs_lock file -- that FFS must attempt to create it on the "empty" source, but because the SMB interface is failing somehow, the file can't be created and FFS aborts.

So I wanted to double-check that this is actually what is happening, and if so, might suggest some additional clarification in the description of "LockDirectoriesDuringSync" at https://freefilesync.org/manual.php?topic=expert-settings ?

First, it's not entirely clear from documentation that lock files are placed on the *source* directory. The documentation seems to suggest it by referring to "lock files" in the plural, but it's not clear why a lock *would* be needed on a source, as opposed to just a destination. So it might be helpful to readers just to make explicit that instead of "lock files" generally, there are 2 lock files specifically, both on source and destination?

And second, if my understanding is correct, then it might also be helpful to explain a second use case of ensuring that the source volume is actually functioning, in case network errors erroneously make it appear empty, then this protects against that.

(I also wonder if it wouldn't be helpful to have some kind of explicit protection against massive deletion? E.g. to be able to set a sync to abort if it would result in more than 50% of files by count or size being deleted from the destination.)

Thanks as always for the amazing amazing tool!

[Zenju]

The lock files generate a warning at most if creation fails, but don't stop sync if the warning is ignored. The question is whether FFS can read the network directory listing without errors. If the network folder presents itself as empty without giving any error indication - in other words if the folder appears just like a regular empty folder - this means big trouble. The only mechanism left that could help then is FFS's warning that "more than 50% of files have changed or will be deleted". So you wouldn't want to ignore warning messages if such a case could occur. But it's unclear if this ("network location incorrectly presents as empty without any error indication") can actually happen in practice (I haven't yet seen it). FFS logs as always will tell what was going on.

[literalzebra]

Thanks so much for the reply Zenju.

Huh, OK -- very interesting, thanks. In both Finder and Terminal, the folder appears like a regular empty folder as far as I can tell, e.g. "ls" behaves normally except lists nothing.

I didn't know FFS had a 50% change warning, so that's good to know -- is that documented anywhere? A quick Google search seems to reveal there used to be a "WarnSignificantDifference" set to 50% by default in GlobalSettings.xml, but current documentation doesn't show anything.

But I *do* check Ignore Errors on my nightly batch jobs, because they frequently run into small errors from random network timeouts or similar on a single file operation that would prevent the rest of the sync from happening -- so now it seems like maybe I should be worried?

I've only noticed the "volume appears empty" bug a handful of times over the past few months, so it's not something I can reliably test. But the next time I do see it, I'll try running a compare and sync manually (so I'd catch an error) to see what happens, and report back here in case there's anything that could help you.

Thanks!

[literalzebra]

Hi Zenju --

So wanted to follow up here, bad news -- exactly what I was afraid of finally happened. My mounted source drive appeared empty due to the mysterious SMB networking bug, and FFS's sync therefore deleted the entire synced drive's contents (with a handful of errors that left a handful of files -- 6493 items deleted, 13 missed).

The log says:

MediaBackup 06/30/2022 03:00:05
Completed with errors
Errors: 14
Warnings: 2
Items processed: 6,493 (0 bytes)
Items remaining: 13 (0 bytes)
Total time: 00:11:10
Errors and warnings:
03:00:10 Warning: Cannot set directory locks for the following folders:
Cannot write file "/Volumes/Media/sync.ffs_lock".
EIO: Input/output error [open]
03:02:12 Warning: The following folders are significantly different. Please check that the correct folders are selected for synchronization.
/Volumes/Media <->
/Volumes/Media Backup
...

I see that it did warn the "significantly different" part, but as I said, I'd set the batch job to ignore errors because there are so frequently little errors along the way that the sync would never complete at all. E.g. a representative error from this log is:

03:07:04 Error: Cannot delete file "/Volumes/Media Backup/.../._.DS_Store".
ENOENT: No such file or directory [unlink]

Now this isn't the end of the world -- I can re-sync the drives which takes a couple days -- but this means FFS is dangerously unusable for my use case.

So I'd like to ask if it would be possible to split the "ignore errors" checkbox into two -- one for "ignore significantly different warning" and the other "ignore filesystem errors". It seems they serve totally different purposes. I never want to proceed with a backup that is significantly different, but I always need to ignore filesystem errors. (Or even if you're not ready to add a UX checkbox, to provide that ability in a config file?)

(And for more info on the bug that mysteriously makes a network drive appear empty -- it's a mounted SMB volume that seems to not play well with my Mac's sleep mode. Every so often when I open my Mac's clamshell, it seems to lose the SMB connection but not the mount, so the drive is there but appears empty. And this is a fairly new M1 Macbook Air running a fully up-to-date Monterey 12.4.)

Thanks!