"Detect moved files" questions/feature requests (Warning! long post :P)

[rrhuffy]

Hello, let's say I have following simplified file structure:

.
├── External
└── Internal
    ├── folderToBackup1
    │   └── fileFromFolder1.txt
    ├── folderToBackup2
    │   └── fileFromFolder2.txt
    └── folderToNotBackup1
        └── fileFromFolderToNotBackup1.txt


I want to backup 2 folders from internal storage, but without 1 unwanted one (a real-life example is using the whole D drive with multiple wanted and unwanted folders)
To achieve this I've prepared a configuration like in attachment:

2018-12-22 12_43_26-_FreeFileSync 10.7 – Folder Comparison and Synchronisation.png (58.65 KiB) Viewed 2848 times

Most important ones are "Mirror" + "Detect moved files". After synchronization file structure looks like this:

.
├── External
│   ├── folderToBackup1
│   │   ├── fileFromFolder1.txt
│   │   └── sync.ffs_db
│   └── folderToBackup2
│       ├── fileFromFolder2.txt
│       └── sync.ffs_db
└── Internal
    ├── folderToBackup1
    │   ├── fileFromFolder1.txt
    │   └── sync.ffs_db
    ├── folderToBackup2
    │   ├── fileFromFolder2.txt
    │   └── sync.ffs_db
    └── folderToNotBackup1
        └── fileFromFolderToNotBackup1.txt


Now if I move
Internal\folderToBackup1\fileFromFolder1.txt -> Internal\folderToBackup1\newFolder\fileFromFolder1.txt
Then it is properly detected:

2018-12-22 12_51_28-_FreeFileSync 10.7 – Folder Comparison and Synchronisation.png (54.08 KiB) Viewed 2848 times

But if I'll (instead of moving above) move file like this (so between two folders and also between two "synchronization lines/folders" in FreeFileSync):
Internal\folderToBackup1\fileFromFolder1.txt -> Internal\folderToBackup2\fileFromFolder1.txt
With file structure like this:

Internal
├── folderToBackup1
│   └── sync.ffs_db
├── folderToBackup2
│   ├── fileFromFolder1.txt
│   ├── fileFromFolder2.txt
│   └── sync.ffs_db
└── folderToNotBackup1
    └── fileFromFolderToNotBackup1.txt


Then file move is not properly detected and I'm getting 1x remove and 1x copy (which is not a problem for small files but imagine a few 5GB files being unnecessarily copied instead of internally renamed/moved):

2018-12-22 12_55_09-_FreeFileSync 10.7 – Folder Comparison and Synchronisation.png (52.81 KiB) Viewed 2848 times

I expect that if the files are equal, so they have the same time and size (like selected in "Comparison (F6)" tab) then they shouldn't be copied and should be moved internally instead.

Mine assumptions:
If files are considered equal due to for example "File size" selected in settings then there can be false positives in both scenarios: comparing file from internal/file1.txt vs external/file1.txt (because we could rename the same size file2->file1) and in detecting moved files from internal/file1.txt vs external/file2.txt (because they can coincidentally be the same size)...but it is only a matter of user preferences when selecting which "file equality" variant user will select in settings. I think that default one ("File time and size") won't make many false detections compared to just mistakenly recognition two different files as equal in normal mode (without file move detection)

And now questions time ;)
1. What technical problems are causing that we are forced to use 'differences file' (sync.ffs_db) during sync to use file move detection feature?
2. Why we can't we use the same settings (from "Comparison (F6)" tab: "File time and size", "File content", "File size") which we are currently using to compare files to be used also to detect file moves (instead of sync.ffs_db file)?

Best Regards
Rafał

[Plerry]

I am not the developer, but I'll give it a shot.
1) sync.ffs_db files are not 'differences files', but database files, primarily used for bi-directional syncs and contain information about the directory and file status as per the last FFS sync. Comparing the directory status per the last sync with the present status allows a straight forward way to detect not just added, modified or deleted files, but can also easily detect moved files. A simple re-use of something that is already available in FFS.
2) The file compare settings (F6) apply to files in their specific location in the (sub)folder tree of the left and right base location. Filecomparison is (presently) not designed to compare files in other locations than in their specific location in the (sub)folder tree. Using file comparison to detect moved files probably means making quite substantial changes to the code; something that does not seem necessary, as there is this other way (via sync.ffs_db files) to detect these.

By the way: like you indicated, the "detect moved files" feature has the scope of the left-right base folder pair it is active in. So, like per your example, you can better define a higher level left-right base folder pair, e.g. D:\FFS_Test\Internal\ and D:\FFS_Test\External\, and define the left and right root folders to be included via the Include filter.
Replace the Include filter * (include all) by
\folderToBackup1\*
\folderToBackup2\*
\folderToBackup3\*

File moves from (within) folderToBackup1 to within folderToBackup3 should now also be detected and executed as moves, as opposed to a delete in the old right location and a left-to-right copy in the new location.

If you have more (root) folders to backup than not to backup, you might consider to add folders to be excluded to the Exclude filter, instead of adding the ones to be included to the Include filter.

[Zenju]

Modification time and size alone are not enough to conclude that files are the same. So additionally the file ID is used to see if this file pair was in sync last time. Thous, it follows that we need a database file (for last sync state) and that move detection only kicks in during the second sync.

In your example FreeFileSync's detection of moved files indeed does not go beyond the scope of a folder pair. Adding support for that is probably not worth the extra complexity. As a workaround you could instead use a single folder pair and set include/exclude filters accordingly.

[rrhuffy]

Yes, one big D:\ -> E:\ synchronisation with include list is working, but...

I understand that file ID comparison (I assume it's like inode on ext filesystem) gives us sureness, but how about adding a checkbox "compare moved files by size and modified date instead of a database file (WARNING! Can rarely cause false positives)" below "detect moved files"?
If FreeFileSync is internally using _BY_HANDLE_FILE_INFORMATION structure, then ftLastWriteTime is a type of FILETIME: "Contains a 64-bit value representing the number of 100-nanosecond intervals since January 1, 1601 (UTC)."
I really believe that a risk of false positive when comparing that number (plus file size!) is really small. And it can be opt-in, so it won't affect users by default.

Then if a user will take a risk of possible false positives then we can detect movement of files without database file (thus it will work even on the first synchronisation)

[rinaty]

I have the same problem, but my folder on different HDD so I could not synchronize parent folder with filter.
I have:
D:\folder1\
E:\folder2\
And all of this folders are synchronizing to F:\

If I move file from D:\folder1\ to E:\folder2\ FreeFileSync does not detect move, and delete backup of file from D:\folder1 and copy from E:\folder2

I have try create junction link:
D:\folder_for_backup\folder1 with link to D:\folder1
D:\folder_for_backup\folder2 with link to E:\folder2
and synchronize D:\folder_for_backup with F:\
but it does not help FreeFileSync detect moves inside one drive (for example D:\folder_for_backup\folder1\a.txt to D:\folder_for_backup\folder1\1\a.txt) but does not detect between different drives (for example D:\folder_for_backup\folder1\a.txt to D:\folder_for_backup\folder2\a.txt)