Detection of moved and renamed files

[Zenju]

It's a TBD, status is attached to the corresponding feature request item.

[bright5]

Rename detection works only in automatic mode. This is VERY dangerous for
backups!
E.g. if someone deletes the file on the backup location (a shared drive), or
it dissapears from the backup because of corruption, malware etc - the file
WILL also be deleted from original location. It's a disaster.
So, one definitely needs a "mirror" mode for a backup of files, not "auto".

I have several terabytes of video files (my camera files) that I want to
backup. When I change the name of the folder (e.g. to matc the event that
video is about), it first deletes several terabytes from the backup and then
copies the same terabytes to the backup disk in the new-named folder.
You can image how long that takes - forever! It's practically unusable. I
can't change my videos folder name and group them according to event recorded.

Is the algorithm to detect rename difficult? Could be, but it is solvable:

A) If all files are common to two maps, it surely is a rename/move of a folder
(easiest case)
B) If no file is common to two maps, it is surely NOT a rename of a folder
C) if some (but not all) files are common we rename the map and copy/delete
differences. That is always cheaper (faster) than to apply the usual delete
all/copy all process as now it is.

In C, with rename you are saving copy/delete of common files, while non-common
file must be copied/deleted as usual - so it is definitely always faster than
to copy/delete all files.

But, If you implement just A, I am satisfied!

If I were you, algorithm would go something like this: generate list of FILES
on source and destination, and match tthem by FILE ID, regardless of the file
location. If the file location (path to the file) has changed - move/rename
it, do not delete+copy!
Everything else, delete/copy as usual.

[bkeadle]

Once again, I find the need to do a Mirror job, but without move detection
(like is available for Automatic), the job can be excessively large for no
good reason because folders have been moved. The job is syncing a music
library from a network source to a local drive. As a music library can be
quite large, when it is (re)organized it often means that folders are moved. I
can't use Automatic mode because of some files that become in conflict and I
have to manually intervene which I'm trying to avoid.

If including move detection for Mirror is as (relatively) trivial as you say,
why is there an apparent resistance to making it available?

[Zenju]

> why is there an apparent resistance to making it available?


There are "costs": Unless there is some other clever design, there needs to be
at least one new option in sync settings.

[Giangi]

Scenario: I have installed DropBox and BoxCryptor to protect a subfolder, this subfolder is then *exposed* as a virtual disk formerly using FAT32.
I use FFS to keep a local backup of DB and a local (unencrypted) of BC in two different folders. My local disk is NTFS.
Both the jobs are configured as Automatic but when I move a file from the BC virtual disk:

* FFS job for BC backup didn't *catch* it and always does a delete/copy new
* FFS job for DB backup didn't always *catch* it and so sometime does a delete/copy new

I'm pretty sure that's related to the *trick* for the DST change, how can I produce some kind of debug logs to see what is going on?

[Zenju]

Detection of moved files is not supported by all file systems. FAT has only very rudimentary support for file ids, this explains the problems you are seeing:
- if a file is moved from one folder to another on FAT, this will change the file's id! If a file is just renamed, this may also change the file id, if the new file name is "too long".

The consequences for a NTFS <-> FAT setups are:
1. if a file is moved/renamed on NTFS, this will be detected and propagated to FAT. However the move operation on FAT silently changed the file id, so that a subsequent renaming on NTFS will likely not be detected. At the time of the next comparison, the new file id on FAT will be read and stored after synchronizaton. Thous move/rename operations will be detected again for this file.

2. If a file is renamed/moved on FAT it will almost always not be detected.

[Giangi]

Uhm... my old tool, SyncToy by MS, was able to detect moves everywere... it has other major drawbacks (and that's way I moved to FFS :-) but the move detection was the only reason I have used for many years "closing an eyes" on its problems... I really hope you'll find a way to solve this... SyncToy uses an SQL compact database... actually very big but personally I do not care about the db size if the tool does its job in a good way! :-)

[Zenju]

I don't think there is a way to detect moved files without the help of file ids. Unless you're willing to play russian roulette with the user's data and just consider files with different names and paths as being identical by checking filesize and date only: Most of the time it will probably work correctly, but in a small(???) number of cases, it will rename the wrong files corrupting data silently... "but screw him, he'll never notice anyway, hehe"
This is clearly not good enough for FFS.

[Giangi]

I have used it for a long time sync'ing terabyte of data without a problem (regarding this "topic"), sometimes SyncToy too was doing a copy/delete instead of a move; so I guess it was able to handle what you are talking about.
I remember that where was some kind of sync source example somewhere on the MS site, maybe it's worth checking

[bkjhkjgjjkk4]

detecting renamed files was one of the features why I gave FFS a try - but I cannot ind this setting anywhere?!

[Zenju]

It's always active for (and only for) "automatic" mode, but it might get an option to enable it for mirror:
[404, Invalid URL: https://sourceforge.net/p/freefilesync/feature-requests/299/]

[bright5]

If you implement move detection just on the file systems that have FILE ID (e.g. NTFS), it is a huge gain!

For the systems that have no file id, you could:

a) do delete/copy as before (so what? it is already a lot better than before), or

b) transfer responsibility to a customer by providing a checkbox "for move detection use size, date and filename on systems with unreliable file id (FAT)" that is initially unchecked. That means for systems without FILEID (of fileid that is not reliable), file rename will not be detected, but directory rename and file moved to another directory (if filename stays the same) will be detected as "move"! And that covers vast majority of the cases and will dramatically improve performance of "mirror" option.

I'm just trying to help, I don't know if it is of any use.

[finisterre]

So to summarize:

Can FFS detect files or folders that have been renamed or moved?

What I have is folders on the source drive that have been renamed and some their contents (subfolders or files) have been moved around and renamed - and some new files added that aren't on the target (backup) drive.

This thread is long and complicated and has me confused. It looks like FFS can do this but not the first time you run a synchronization -?

Thanks. This looks like a great program.

[kdjfcsjflksj]

FFS Can detected files that habe been moved in sync mode after the first sync. however, this feature is not (yet) available in mirror mode

[gioita]

I read all the post and then try the last release, but I don't have the evidence of renamed or moved files. Renaming 1 file, the software say "delete 1 file and add 1 file", in all Synchronisation modes and also second times. Probably I miss something. Can someone explain how to have evidence of a renamed or moved files ?

[gioita]

Any news ?

[Zenju]

Implemented for 5.21: [404, Invalid URL: http://freefilesync.sourceforge.net/FreeFileSync_5.21_beta_Windows_Setup.exe]

[markling]

Will this change be making it through to Linux any time soon? I'm using FFS version 5.18. My system tells me this is the latest version.

I would be very helpful to know because I'm just at that crucial stage where I am coming to terms with the fact that I must manually deal with renamed files and folders across multiple copies of the same 60G set of data. After that, I have another, smaller but more complex set that needs syncing.

I had come to FFS for the same reason I imagine a lot of people do (aside from it seeming like the best tool available to me): multiple copies of data had got out of hand. One of the problem's main characteristics was files and folders that had been moved and renamed.

I would think it fairly easy for a tool to handle this, because I can see it easily enough when one element of data in two copies has been moved and renamed. It's just time consuming for me to do it over 60G.

FFS will at least make my job of manually syncing the data easier, becuase it pulls out the renamed files and folders and lists them among the other files that are genuinely different. This will save me time. But not as much time as I hoped.

FFS does nevertheless look brilliant. And I will be getting right on now to sync at least two sets of data with its help. It's ability to sync those files that are simply present in one set and not the other, and those files of the same name and location that have been updated in one and not the other, will save me a considerable amount of time.

[markling]

Ah well. That was painless. What a relief. There was one moved/renamed folder I had to handle manually. I think there may not have been much difference between these two. Or was FFS helping by identifying moved/renamed files and folders?

The real test will come when I get on the to the set of files where there are some real thorny differences.

[Giangi]

Implemented for 5.21: [404, Invalid URL: http://freefilesync.sourceforge.net/FreeFileSync_5.21_beta_Windows_Setup.exe]

Database support for detecting move/rename is still not available (checkbox cannot be ticked) for: Update & Custom!