New sync concept in FreeFileSync 13

[Zenju]

The next FreeFileSync version generalizes the way sync configurations are set up: In addition to "comparison results", it will be possible to set sync directions based on "changes":

sync-cfg.png (40.73 KiB) Viewed 5361 times

Of the three sync variants that FreeFileSync offers by default (two way, mirror, update) it always bothered me that "update" wasn't as fundamental and useful as the other two.
After meditating on the issue, I think I found out why. "Update" was making it's decision based on the result categories after comparison. Thous it's unable to distinguish whether a new file was created on the source, or if an old file was deleted on the target. But these two should get different handling: The most common "update" case is making a copy of photos from a smartphone to some backup location. When photos on the smartphone are deleted, they should *not* be deleted on the backup, obviously. The user is just making free space for new photos. On the other hand, photos deleted on the backup, should *not* get them copied over again during the next sync: The user is just cleaning up unwanted photos. Unfortunately this is precisely what the results-category based "update" does. Additionally there's the issue where a file on the source side is renamed: without a database this change cannot be detected, so "update" needlessly copies a duplicate file to the backup location.

The solution? The sync directions should be determined based on "changes" compared to the last sync, in the exact same way that "two way" operates, by using a database file. But why stop there? Why not let the user make specific decisions for each individual change "create, update, or delete", like it was always possible with the result categories: The "two way" and "update" variants just become special cases. Even "mirror" can be expressed in terms of "changes". One might be tempted to go further and assume "changes" is the more fundamental concept, and get rid of the old results categories altogether, if there wasn't a catch: The requirement of sync.ffs_db files. "Mirror" can be expressed in both terms, but doesn't need a DB file if specified via result categories. "Mirror without DB" is an important enough use case that should be kept.

So what gives? FreeFileSync now supports two fundamentally different ways to set up sync directions. The "changes"-based settings have always been hidden inside "two way" but are now exposed. On first sight, this slightly increases the complexity of the tool, but on the other hand opens up for new possibilities (like a proper "update"), and maybe/hopefully other kinds of sync scenarios, that I'm not seeing right now (e.g. perhaps a "changes"-based "mirror" that "does nothing" if the changes are occurring on the target side? That's not quite the "conflict" that some users would like to see, but better than nothing).

sync-variant-update.PNG (28.58 KiB) Viewed 5361 times

[Plerry]

Based on the above, it is still quite unclear (at least to me ...) what will change in the next FFS release (13).
It seems clear that the present five Difference categories (assuming a Compare by time+size) will be replaced with three (or perhaps actually six) Change categories.
How this is achieved is unclear, but it seems that, although you apparently intend to keep the "Mirror without DB", that the future "proper" Update-variant, unlike the present Update variant, would require the use of a DB-file, which in turn would require FFS to have write access to the source/left base location.
At least for me that would be an absolute no-go.
I run all my Mirror and Update backups under a dedicated user-account that only has read-access to the source/left base-location and is the only user-account that has write-access to the target/right base-location and to the Versioning location; second to adequate prevention, probably the single most effective way to protect yourself against potential consequences of ransomware.
So, I hope that next to a "Mirror without DB" there will also still be an "Update without DB", or at least the option to define a Custom variant that does not require a DB.
If not, I may see myself forced to stick to FFS 12

[Zenju]

The "< FFS 13 update" is demoted to a "custom" variant. A quick way to get this variant would be to click on "Update >" and uncheck "Use database file to detect changes".

Another important change: the "conflict" category is not configurable as a sync direction anymore:
In general there are two kinds of conflicts, which are handled separately in FFS 13:
1. invalid time: it's unclear which side is newer/older
2. other conflicts like "error during directory traversal"

"< FFS 13" both kinds could be resolved by automatically assigning a sync direction, but it's really only safe to do so in case 1, not in 2.
FFS 13 will always require manual resolution in case 2, and automatically handle case 1: If sync config is such that both "left newer" and "right newer" are assigned the same sync direction (e.g. as is the case for mirror), then it doesn't matter if file time is invalid.

sync-cfg-mirror.PNG (39.08 KiB) Viewed 5338 times

[Zenju]

Here's the current beta version for testing: https://www.mediafire.com/file/1tg1bacfmlwuyqr/FreeFileSync_13.0_%255BBeta%255D_Windows_Setup.exe

[patrickdrd]

is this change gonna be backwards compatible?
or do we have to change our scripts?
I'm using not the photos mode you mentioned, but the other one, deletions synced etc, do I have to do anything?

[patrickdrd]

@zenju??

[xCSxXenon]

It might be worth trying the beta version to confirm. From what he has said, nothing is changing in regards to how FFS works. The update is just going to make some things visible/changeable that were locked 'under the hood' previously.

[patrickdrd]

a, ok,v great, thanks

[patrickdrd]

after updating to v13, all my *.ffs files were edited and
the following was added to them:

        <Changes>
            <Left Create="right" Update="right" Delete="right"/>
            <Right Create="left" Update="left" Delete="left"/>
        </Changes>


I guess this was in order to preserve backwards compatibility

[mikoy]

I don't quite understand icons showed in mirror mode with database on in version 13.
FFS shows something like this [left/right]:
create: ->+ / trash bin
update: -> / ->
delete: trash bin / ->+

I thought mirror should be rather:
create: ->+ / ->+
update: -> / ->
delete: trash bin / trash bin

Is this some kind of mistake (wrong icons) or I just don't get the idea? I'm using mirror for a long time, but those icons showed recently and made me wonder if anything has changed how mirror works.

[xCSxXenon]

Each row corresponds to an action. If you create something on the right side, it will be deleted as shown in the matching row/column intercept. Updating a file in either the left or right side will cause FFS to mirror the left file onto the right side. And on and on

[mikoy]

Ok, thanks, now I get it. Left and right row/action should be watched separately, not as combined process such as if I create something left it will be deleted right ;)

[David.P]

Thanks for the awesome new sync concept!

Meanwhile, is there a way to recognize and sync renamed folders?

I am syncing from a local folder to a network share, and it seems that renamed folders are not recognized. Instead, FreeFileSync wants to delete the existing folders and replace them with the renamed folders.

[xCSxXenon]

I was not able to replicate your issue personally. I synced a folder and subfolders to a Windows network share. There were Windows ISOs in it, quite large an noticeable if they are re-copying rather than moving. I was able to move subfolders around, even into other subfolders, and it moved everything. I will say, it looks like the folder itself is re-created, then the files are moved, and then the old folder is deleted, but empty folder creation and deletion is negligible

EDIT: I also tested renaming the subfolder and it performed the same actions: Create new folder, move files, delete empty original folder.

[David.P]

In my case, unfortunately, the original folder is even synchronized backwards and the renamed folder is synchronized forwards, so that afterwards both the original folder and the renamed folder are present on both sides 😮

However, this could be related to the fact that the folder is on SharePoint on one side (via OneDrive shortcut in Windows Explorer) and on a file server on the other side. SharePoint is known to increase the file size for most files as it seems to write some kind of metadata into the files🤦🏼‍♂️

PS: I just see that at least some of the moved files (or renamed folders) seem to be recognized. However, the folders to be renamed or files to be moved are not displayed horizontally side by side in the preview window, but in separate lists one below the other. This makes it almost impossible to see if the synchronization will be done correctly.

[David.P]

I also tested renaming the subfolder and it performed the same actions: Create new folder, move files, delete empty original folder.

Do these actions each show up in the same respective rows in the sync preview?

Meaning, do you get one row for each file that is going to be moved, or two different rows?

[xCSxXenon]

I can confirm I get two rows, but that the icon in the middle is the icon for move/rename

[David.P]

I see, thanks.

In my view, that's a bit unfortunate since you basically have to blindly trust that the moving/renaming will work properly, at least when many files have been moved or renamed, because there is no way to see what is going to happen to which file (apart from checking each file manually).

What's more, it seems that moved/renamed files or folders are shown together with changed files in the same view. That makes it even harder to recognize what is going to happen when syncing.

[xCSxXenon]

That is discussed here:
viewtopic.php?t=10584

I chimed in that lumping it into 'update' was fine, but I am starting to change my mind. It gets full down there with more granular icons

[David.P]

Thanks, I also chimed in over there.

[LarsErikOsterud]

I can't get mirror to work anymore. A file has changed on the PC but still is not on the list to be copied even if I have the setup as before (left newer -> copy to right). But still does not show up if the left is newver (have checked the date and time). To me the new version doesn't work and I can't trust it

[xCSxXenon]

I can't get mirror to work anymore. A file has changed on the PC but still is not on the list to be copied even if I have the setup as before (left newer -> copy to right). But still does not show up if the left is newver (have checked the date and time). To me the new version doesn't work and I can't trust it

Sounds more like your locations aren't set up correctly or there are filters interfering

[Daniel22]

sync-cfg-mirror.PNG

Please change the part where you can set what happens on certain differences back to pre ffs13.

It was so easy to understand what you do with these icons in previous versions because of the tooltips and so on, now i dont get it at all.

This was perfect: https://web.archive.org/web/20230102234323/https://freefilesync.org/manual.php?topic=synchronization-settings