.sync.ffs_db queries

[charliebrown]

Hi,

Newbie query here.

I have 4 storage locations.

A. laptop A - EXT4 - Local
B. USD HDD B - BTRFS ZSTD:10 - Offline
C. laptop C - NTFS - Offsite
D. online cloud D (eg. PCloud) - Online

1. When I sync AB it creates .sync.ff_db at root of sync. Is .sync.ff_db duplicated between AB or is there a different version on A and B?

2. When I sync BC will this update the .sync.ff_db or ovewrite? If updating, will it record the different data for ABC and know there are 3 storage locations?

3. Will .sync.ff_db be duplicated between BC now or is there a diff version at B and C?

4. Does .sync.tf_db keep internal records to differentiate ABC storage location data?

5. Do we need a different sync.ff_db for AB and a different one for BC, or can 1 file cover all storage locations used ABC?

6. If I return to sync AB without any change at A, will ABC now be in sync?

7. If I decide to also sync AC via online D cloud (mounted as local fuse drive), will FFS work for this?

8. If I continue using D,B as central locations (offline/online) will this be okay, as .sync.ff_db is keeping track of changes on each?

I hope the above questions make sense?

If not, please ask.

Thanks

[Plerry]

1) each location has its own sync.ffs_db file
2) each location has a single db-file with information relevant to all syncs run with that location
3) not duplicated; see 1)
4) yes, see 2)
5) see 1) and 2)
6) If you (only) sync A<=>B, obviously C will not (yet) be in sync

7) and 8) Hard to say up front.
If the D cloud acts as a local drive on B, that drive uses the file-IDs local to B and accepts a sync.ffs_db file to be written, then D can be kept in sync with B.
But if you also "mount" D as a local drive on C, the sync.ffs_db file will not "work", as C will have file-IDs different from B for the same files.
However, if you sync D directly to/from the cloud and the cloud service has stable file-IDs, this should work

[charliebrown]

Thanks for the answers as very useful. 1-5 are clear now.

Just a few more queries as below:

6. B is the portable USB hdd to sync between site A laptop and site B laptop. So if I sync AB, then BC, then AB again (no changes to A) - then all three should be sync?

7. What I meant was, if I mount cloud service as local fuse drive (eg. PCloud/Google drive etc.) as D, then sync AD and later DC, would this keep ACD in sync?

8. I'm really only syncing 2 laptops A and C. But because they are in different locations using either online D cloud or offline B usb HDD as the middle man to complete the sync. With each having their own sync.ff_db file, would this work?

9. What are the file-ID's derived from? eg. If we have file X with same date/time/size/contents. Would it have the same file-ID on all 4 locations ABCD?

10. What would you suggest is the best way to keep both laptops AC synced. + offline copy on B usb hdd + online copy in cloud D. Is this within FFS remit?

11. Would (10) answer work if I added 3rd laptop E in the sync equation?

12. I understand the ideal scenario would be a STAR situation with only 1 central hub to sync all other devices. this could either be online D or offline B, but just trying to understand the options here as don't want to carry B everywhere always so will try to use D as the central location for speed/efficiency, but would want offline backup B just incase.

Many thanks again.

[charliebrown]

Hi,

Just one comment on issue (6).

a. I did AB sync. Files added to A and B from each other.
b. Then I did BC sync. Files added to b and C from each other.

c. I went back to A (which had been off throughout so no new files added). I did AB sync expecting new files from B (as received from C) to be sent to A, thus having same files on ABC.

Strangely, FFS didn't send ANY new files from B to A.
Instead it wanted to ONLY send files from A to B again, including files that were already on B.

Totally confused where the logic in this is?
Wondering if any date/times/permission were changed in BC sync that makes A want to send to B again?


(9) Does ffs consider file permissions aswell when checking which files have changed? Wondering if the file permissions were possibly changed by BC sync, hence A wants to send files to B again? Does FFS change copy file permission?

A is EXT4, B BTRSF, C NTFS. Does the permissions handling on these create any issues for ffs?

Thanks

[Plerry]

Your question started about the use of sync.ffs_db files.
These files are mandatory in a Two-way sync variant, and optional in some other variants.
The benefit of using sync.ffs_db files is that changes in file- or folder-names, file-moves (within the scope of the sync) will be synced as renames or syncs and not result in renewed copying over, and file deletions in one location will also result in the file(s) being deleted in its counter-location. FFS uses file-IDs for that purpose. These file-ID need to be stable; i.e. not change in between sessions. File-IDs are local to a machine, and not all OSs use stable file-IDs.

As your B is a portable medium, the file-IDs of files on B when connected to machine C will not be the same as when connected to machine machine A (even if they are stable per machine). The same holds for your D when used on machine A versus when used on machine C.
So, in your use-case, the use of sync.ffs_db files (and thus the Two-way sync variant) needs to be discouraged and will not have the expected results.
If changes can originate on both A and C, and thus you need a two-way like sync, you can instead define a suitable Custom sync variant with the following settings:
• Difference category "Item exist on left side only" choose the Action "Copy new item to the right" (green right pointing arrow with plus)
• Difference category "Left item is newer" choose the Action "Update right item" (green right pointing arrow)
• Difference category "Conflict" choose the Action "Leave as unresolved conflict"
• Difference category "Right item is newer" choose the Action "Update left item" (blue left pointing arrow)
• Difference category "Item exist on right side only" choose the Action "Copy new item to the left" (blue left pointing arrow with plus)
and leave "Detect moved files" uncheckmarked (thus, no sync.ffs_db files are used).

Now you should be able to run you syncs without problem, at least via B, and likely also via D, depending how your web-drive handles file-dates.
However, you will not have the benefit of faster syncs when you have renamed or moved files or folders, and any file-deletions need to be handled manually.

[charliebrown]

Thanks so much for the clarification.

If ffs won't be able to detect file deletions, then it would be too cumbersome to do manually each time so I might have to look for another solution.

This is a real pity.

Do you think there might be a way in future to keep a database on portable B, which isn't limited to single machine use?

thanks

[Plerry]

If syncing file deletions is that important to you, you may consider:
• After working on A, Mirror sync A to B (or D)
• When starting to work on C, Mirror sync B (or D) to C
• After working on C, Mirror sync C to B (or D)
• When starting to work on A, Mirror sync B (or D) to A
You may organize this via scheduled tasks (e.g. at startup and at shutdown)

The risk here is that you have to select the proper sync at the proper moment, as it is now asymmetrical.
Use the wrong one and you are in trouble, although the use of Versioning as your Deletion Method can help here.

But perhaps always "working" on D (or on B) is your way to go, whether using machine A or C, and make "backup" syncs to A and/or C.

[charliebrown]

Hi,

Just one comment on issue (6).

a. I did AB sync. Files added to A and B from each other.
b. Then I did BC sync. Files added to b and C from each other.

c. I went back to A (which had been off throughout so no new files added). I did AB sync expecting new files from B (as received from C) to be sent to A, thus having same files on ABC.

Strangely, FFS didn't send ANY new files from B to A.
Instead it wanted to ONLY send files from A to B again, including files that were already on B.

Totally confused where the logic in this is?

Just realised this was my silly error.

When I went back to AB, for some reason the setting file wasn't actually pointing at B properly hence the confusion.

I corrected the B path and now FFS is acting perfectly by sending all new files from B (as taken from C), into A.

This is exactly what I wanted, so now ABC will all be the same.

Moving onto the issue you stated about ff_db file only being for 1 machine, I will have to see if this creates problems in the future. For now it seems to work fine.

I'm hoping I'll be able to sync machines A,C using offline hdd B and/or online cloud D.

If there are problems, might consider Unison as its manual says it can sync ABCD fine because its logic is to work in pairs only. AB, BC, CD, DA. If FFS could do this, perhaps it would overcome the current limitations on multi machine sync.

Will test further and update.

Thanks

[charliebrown]

Hi,

Just slightly confused by your answers which suggest that ABC can't be sync properly as ff_db is particular to each machine.

After setting the right path for B (my silly error - so ignore my problem post), I've done a few tests and found the following all work:

1. AB - sync
2. BC - manually move files, then sync.
3. AB - sync again, and log shows that files were moved. Not deleted/added.

If (3) is correct, then it seems FFS is figuring everything out perfectly and I can use the usb disk B to easily keep laptops A and C in sync, with moving, deleting, adding all working fine?

[charliebrown]

Ignore above post.

I've realised that ffs is NOT able to sync between multiple machines properly like you said as ff_db is unique to each machine.

Could I please propose the following solution:

1. When ffs is run by machine A, create ff_db on A and another on B. However, append machine A network MAC address to filename so it reads ff_db_MAC. This would therefore be a unique ff_db to machine A only.

2. When ffs is run by machine B, a new uniquely named ff_db_MAC would be created, which is unique to machine B.

This way, B would have 2 ff_db_MAC files and stll be able to truly sync with each machine including ability to move/delete without confusion.

I guess its similar in some ways to how Unison handles sync by creating a source/destination profile per pair.

Wouldn't above solution be quite simple and feasible to improve ffs so it can truly sync between multiple computers without issue?

Thanks

[charliebrown]

Pending an update with feature as requested above, I've created a manual solution.

1. AB - after sync is complete rename B .sync.ffs_db to another name such as machineA.sync.ffs_db

2. BC - after sync is complete rename B .sync.ffs_db to another name such as machineC.sync.ffs_db

3. AB - When syncing AB again, rename machineA.sync.ffs_db to .sync.ffs_db

4. BC - When syncing BC again, rename machineC.sync.ffs_db to .sync.ffs_db
.
This would allow B to be a central portable disk between A and C, while keeping the benefits of acknowledging moves/deletes.

Would be better if we could have .sync.ffs_db auto named with unique name for each machine. eg. by prefixing the machine MAC address or something like that.

Until then, will have to do this manual override to make ffs work with syncing multiple computers.

If you see any problem with this setup please let me know, and if possible to add this request that would be great.

thanks