Detection of moved and renamed files

[nick2011]

Version 4.4 has a new feature 'Detection of moved and renamed files' but when
I rename a file sincronize is done in 2 steps: delete the file on destination
and copy the file (like in previous versions).
In that case which is the difference ?

[Zenju]

The new feature requires the database to detect renaming and moving. So you'll
have to choose `<automatic>` variant.

[nick2011]

I need to change only right side so I have to choose Mirror.
Database works only with automatic variant ?

[Zenju]

Yes, database is created only for variant <automatic>.

[geoff987654]

I am also having problems with renaming folders and files.
I have the sycn varient set to Automatic (it appaers in grey above the
synchronise button).
when I run compaire the preview show the files being deleted from the old
folders and then copied to the new folder.
I stugle to understand quiet how this works.
Must I be using compaire contents?
Does it only work with files not folders ?
Is there a different icon for renamed files.

Thanks
Geoff

[Zenju]

Detection of renamed files works beginning with the *second* sync using
<automatic> mode with either "time + size" or "binary" comparison. The
algorithm applies to files, not folders and can be recognized by a distinct
icon in middle grid.

[geoff987654]

Thank you for the clarification.
I guess asking for the algorithm to work with folders as well is too much to
ask for.
I guess I will just have to be more careful in choosing the names for my
folders.

Geoff

[Zenju]

Folders are very problematic in the context of moving/renaming. In the most
trivial of cases it's possible, e.g. if only the folder is renamed, but none
of it's child-folders and files is changed. In all other cases, one basically
would have to determine the list of changes the user has applied to this
folder in the correct order. This can become very complex quickly. Then trying
to visualize these changes on GUI so that the user gets a preview is futile
... except for the trivial case.
An even bigger problem: Applying such a consecutive list of changes to a
folder is not transactional (every folder rename operation affects all
children!). If anything goes wrong, e.g. power-loss, etc. the user's data may
be doomed (= left behind in an inconsistent state).

However let's note that detection of moved files is a performance optimization
only. And for detection of moved folders one finds that usually there is no
benefit at all, except for extreme cases where the costs to rename all child-
elements is actually noticable.

[yppip]

After reading the previous posts about 'Detection of moved and renamed files'
resulting in the time saving 'move' process applied in the target folders, I
presumed that when moving/renaming folders, both the folders and the files
they contained would be deleted and recopied as in mirror mode.
However having experimented with ver.4.5 in Automatic mode, I was both
surprised and delighted to see that although the moved/renamed folders were
deleted and recreated, the files that they contained were still only moved, no
matter how complicated the folder structure. Not only saving time, but even
better, storage space, by only putting moved/renamed folders and not their
files in versioning folders.
Brilliant, the best syncing software just got even better. Thanks.

[Giangi]

Isn't it possibile to create/manage the database sync.ffs_db for each sync
mode? Me too I would like to have the move detection at least on the
mirror sync mode!

Anyway, mee too I would like to thank you for this great program!!!

[geoff987654]

Thank you for the detailed reply. Your customer support puts many the
commercial organisations to shame.
For your information to explain why moving folders, rather than deleting and
creating a new folder, would be beneficial.
I have following system
plain files folder <=>sync program<=>encf for Windows<=> trend micro safesync
folder
Any file written to the plain file folder ends up in my tend Micro sayfsync
account in encrypted form.
The system also works in reverse
Any file(pre-encrypted) written to the SafeSync account appears in my plain
file folder in an unencrypted form.
I did this beacuse the output of encf for window cannot be shared over the
network.
The safesync software is clever enough to reconize moved and renamed files,
saving a lot of tome re-uploading files.
If the sysn program does not also do this the chain is broken and moved files
have to be uploaded.
Historically so that programs could find current data all current year folders
had a standard name. At the end of the year the current folders were renamed
with apropriate year number.
We are moving away from this to cut down the amount of renaming and hence
reloading.

Geoff

[Zenju]

> create/manage the database sync.ffs_db for each sync mode
Maybe worth considering. I like the idea of not having an additional
configuration option to detect moves in <automatic> mode. But the "costs" are
design-only, implementation is trivial.

>I presumed that when moving/renaming folders, both the folders and the files
they contained would be deleted and recopied as in mirror mode.
I think this is a common misconception, so I want to make this clear: When I
say FreeFileSync does not detect moved folders, I mean the folder only, not
its child-elements. All files contained, no matter how deep in hierarchy, are
properly detected as a file move. In this context there is *no* difference
between a file renaming, or a move in hierarchy (or both).
In general, just trying this feature, may already answer most questions! ;)

[Giangi]

I may have found a bug into the move detection...

For semplicity (and having different base name) I have executed on all my
three computers the command subst R: %USERPROFILE% because SyncToy do not
allow using envars...

Anyway, if a Mirror job is using on both side the virtual drive R: it
will always delete&copy and never move!
I'm almost sure is related to the way the subst'ed drive is handled (I'm using
Seven).

I know that FFS do support envars, but I thought it was worth reporting
that behaviour! :-)

[Anonymous]

Hi

when I move some files from "folder A" to "folder B" and sync, FFS recopy the
whole things again instead of "moving" within the machine. I sync through
"internet" with VPN, this makes sync super slow. I just wonder if there
anything I can make it better?

thanks!

[Zenju]

> I may have found a bug into the move detection...
Indeed! This subst was able to out-smart FFS! :) Not anymore, fixed:
[404, Invalid URL: http://ifile.it/srn36fo/FreeFileSync_v4.6_beta_setup.exe]

> through "internet" with VPN
Unfortunately file ids, the technical vehicle that allows for detection of
moved files, is not supported by all file system implementations.
But in general: all locally mounted file systems should be fully supported, no
matter how complex the setup (symbolic link, to some "subst"-included
volume...etc.)

[Giangi]

> Indeed! This subst was able to out-smart FFS! :) Not anymore, fixed:
[404, Invalid URL: http://ifile.it/srn36fo/FreeFileSync_v4.6_beta_setup.exe]


You've been too quick! :-)

[Anonymous]

Is there a possibility that a detected moved/renamed pair of files are
displayed in the same line (left and right panels)? At present it is difficult
to find out what are corresponding files on both sides.

[Zenju]

I don't think it is considering all the design constraints. However FFS
compensates by showing both source and target in middle grid's tool tip.

[eugen7]

First, thanks for the product. its good...
I use it for backup just to copy left to right. Mirror Variant. v4.4 came up
with database to detect moved and rename. Awesome feature. So i change to
Automatic Variant but for a Mirror purpose. All i do is un-tick changes on the
left MANUALLY to achieve it.

Can you add an option to disable any changes to left side for Automatic
Variant ? Or maybe option to mass untick changes on left side ?

[Zenju]

> Or maybe option to mass untick changes on left side ?
You can CTRL+A and select all rows of a certain view category, then change
direction via ALT+<arrow. Or do the same with a mouse selection + right-click
context menu.

[profmayer]

Hi,
Thanks a lot for that great programme :) I'm using v5 and i would be happy to
have that detection feature on mirror sync too.
Would it be possible, like in GIT i think, to make a table of each file with
SHA1 or "size and modif time" signature. Using it instead of file ids would
allow to have the feature on any filesystem and on the first mirroring. Could
then also add an option to replace doubles by only one file and symlinks to it
for the other location.
example :
dir1 :
file1.txt 50657 Bytes Modified 02-03-2011 xxxxx SHA1 xxx
file2.7Z 23486509479 Bytes Modified 01-03-2011 yyyyy SHA1 yyy
subdir/file2.7Z 23486509479 Bytes Modified 01-03-2011 yyyyy SHA1 yyy

dir2:
file1.txt 50657 Bytes Modified 02-03-2011 xxxxx SHA1 xxx
newdir/file2.7Z 23486509479 Bytes Modified 01-03-2011 yyyyy SHA1 yyy

instead of del and copy file2.7z, detect dir2 has somewhere a file with the
same signature ("size and modif time" or SHA1) and does a move of the file
from dir1/newdir to /dir1 and a symlink to it in dir1/subdir.

In case of SHA1, the table could be saved where the user want to not redo the
digest again or allow integrity check.
Thanks again :)

[mark1990]

Hello, I was wondering whether FFS is able to recognize filemovements by a
third party on a network drive, when the computer that runs FFS is turned off.
Will it then still be able to see that a file has merely moved or will it
remove the file on the computer and then retransmit it from the network drive
to the computer? I'm guessing that FFS is able to determine
filemovements/renames by means of the service running in the background and
that when it is not running it will not be able to do so for a network drive.
But I hope I'm wrong.

I think I saw somewhere a request for somebody to write a FFS package for
Synology DiskStations...? :)

[Zenju]

FFS is able to detect any file movements, by using file ids. This is a file
system feature supported by all local Windows file systems and generally on
Unix. But it might not be available for some arbitrary mapped NAS volume. Just
try and see if it works. And since this requires the database file as
reference, this featuer needs "automatic" sync variant.

>Synology DiskStations
Never heard of those.

[mark1990]

Tried it and it works without problems! Only 'problem' is that rescanning all
files on the NAS takes long compared to scanning files on the computer which
is sped up by threading. But that ain't something FFS can help.

[bkeadle]

I would like to add to the chorus for adding move detection for Mirror Sync. I
have submitted this as a Feature Request on the tracker page.

[bkeadle]

In an attempt to accommodate not having move detection for the Mirror variant,
I thought I might be able to use environment variables in my gui/batch
settings to define the variant as either Automatic or Mirror - I found that I
cannot. Would it be possible to allow for a vairant variable?


<SyncConfig>
<Variant>%VARIANT%</Variant>



Because I want to enable move detection between a folder pair, I (currently)
need to use the AUTOMATIC variant. But if differences are detected in both
causing a conflict, I need to be able to switch to the MIRROR variant,
defining my left side as master. This would be a 2 step process: run Automatic
variant ignoring the conflicts to do a best attempt sync (including move
detection), then re-run the sync as Mirror variant to make the directories in
sync.

If move detection could be enabled for the Mirror variant, then I could always
define my left as Master, and Mirror sync to the right and still take
advantage of "move detection" - an AWESOME feature!

[Zenju]

If what you really want is a "mirror" but you use "automatic" mode instead in
order to have move detection, then this should only be used in GUI mode, where
you have have a preview. Else you won't notice if there were changes on target
side that are propagated to the source, with conflicts being only a special
case of this category.

[bkeadle]

??? I have a source I deem the master that I want to mirror to the target, but
the source would like have moved files. With mirror today, that would mean
deleting from the target, then doing a new copy of files that had already
existed on the target but in a different location, right?

[Zenju]

Right. But instead of using mirror you can "abuse" automatic mode and just
manually look that all directions are one-way. As long as you are only
modifying one side, but not the other, e.g. a backup medium, you have the
benefit of detection of moved files.

Conceptually: 1. Mirror models "owner -> observer" semantics 2. automatic
models "shared ownership". The right variant should be chosen according to
these models.

[bkeadle]

So that's a "No" for enabling move detection for Mirror?

[Zenju]

It's a TBD, status is attached to the corresponding feature request item.

[bright5]

Rename detection works only in automatic mode. This is VERY dangerous for
backups!
E.g. if someone deletes the file on the backup location (a shared drive), or
it dissapears from the backup because of corruption, malware etc - the file
WILL also be deleted from original location. It's a disaster.
So, one definitely needs a "mirror" mode for a backup of files, not "auto".

I have several terabytes of video files (my camera files) that I want to
backup. When I change the name of the folder (e.g. to matc the event that
video is about), it first deletes several terabytes from the backup and then
copies the same terabytes to the backup disk in the new-named folder.
You can image how long that takes - forever! It's practically unusable. I
can't change my videos folder name and group them according to event recorded.

Is the algorithm to detect rename difficult? Could be, but it is solvable:

A) If all files are common to two maps, it surely is a rename/move of a folder
(easiest case)
B) If no file is common to two maps, it is surely NOT a rename of a folder
C) if some (but not all) files are common we rename the map and copy/delete
differences. That is always cheaper (faster) than to apply the usual delete
all/copy all process as now it is.

In C, with rename you are saving copy/delete of common files, while non-common
file must be copied/deleted as usual - so it is definitely always faster than
to copy/delete all files.

But, If you implement just A, I am satisfied!

If I were you, algorithm would go something like this: generate list of FILES
on source and destination, and match tthem by FILE ID, regardless of the file
location. If the file location (path to the file) has changed - move/rename
it, do not delete+copy!
Everything else, delete/copy as usual.

[bkeadle]

Once again, I find the need to do a Mirror job, but without move detection
(like is available for Automatic), the job can be excessively large for no
good reason because folders have been moved. The job is syncing a music
library from a network source to a local drive. As a music library can be
quite large, when it is (re)organized it often means that folders are moved. I
can't use Automatic mode because of some files that become in conflict and I
have to manually intervene which I'm trying to avoid.

If including move detection for Mirror is as (relatively) trivial as you say,
why is there an apparent resistance to making it available?

[Zenju]

> why is there an apparent resistance to making it available?


There are "costs": Unless there is some other clever design, there needs to be
at least one new option in sync settings.

[Giangi]

Scenario: I have installed DropBox and BoxCryptor to protect a subfolder, this subfolder is then *exposed* as a virtual disk formerly using FAT32.
I use FFS to keep a local backup of DB and a local (unencrypted) of BC in two different folders. My local disk is NTFS.
Both the jobs are configured as Automatic but when I move a file from the BC virtual disk:

* FFS job for BC backup didn't *catch* it and always does a delete/copy new
* FFS job for DB backup didn't always *catch* it and so sometime does a delete/copy new

I'm pretty sure that's related to the *trick* for the DST change, how can I produce some kind of debug logs to see what is going on?

[Zenju]

Detection of moved files is not supported by all file systems. FAT has only very rudimentary support for file ids, this explains the problems you are seeing:
- if a file is moved from one folder to another on FAT, this will change the file's id! If a file is just renamed, this may also change the file id, if the new file name is "too long".

The consequences for a NTFS <-> FAT setups are:
1. if a file is moved/renamed on NTFS, this will be detected and propagated to FAT. However the move operation on FAT silently changed the file id, so that a subsequent renaming on NTFS will likely not be detected. At the time of the next comparison, the new file id on FAT will be read and stored after synchronizaton. Thous move/rename operations will be detected again for this file.

2. If a file is renamed/moved on FAT it will almost always not be detected.

[Giangi]

Uhm... my old tool, SyncToy by MS, was able to detect moves everywere... it has other major drawbacks (and that's way I moved to FFS :-) but the move detection was the only reason I have used for many years "closing an eyes" on its problems... I really hope you'll find a way to solve this... SyncToy uses an SQL compact database... actually very big but personally I do not care about the db size if the tool does its job in a good way! :-)

[Zenju]

I don't think there is a way to detect moved files without the help of file ids. Unless you're willing to play russian roulette with the user's data and just consider files with different names and paths as being identical by checking filesize and date only: Most of the time it will probably work correctly, but in a small(???) number of cases, it will rename the wrong files corrupting data silently... "but screw him, he'll never notice anyway, hehe"
This is clearly not good enough for FFS.

[Giangi]

I have used it for a long time sync'ing terabyte of data without a problem (regarding this "topic"), sometimes SyncToy too was doing a copy/delete instead of a move; so I guess it was able to handle what you are talking about.
I remember that where was some kind of sync source example somewhere on the MS site, maybe it's worth checking

[bkjhkjgjjkk4]

detecting renamed files was one of the features why I gave FFS a try - but I cannot ind this setting anywhere?!

[Zenju]

It's always active for (and only for) "automatic" mode, but it might get an option to enable it for mirror:
[404, Invalid URL: https://sourceforge.net/p/freefilesync/feature-requests/299/]

[bright5]

If you implement move detection just on the file systems that have FILE ID (e.g. NTFS), it is a huge gain!

For the systems that have no file id, you could:

a) do delete/copy as before (so what? it is already a lot better than before), or

b) transfer responsibility to a customer by providing a checkbox "for move detection use size, date and filename on systems with unreliable file id (FAT)" that is initially unchecked. That means for systems without FILEID (of fileid that is not reliable), file rename will not be detected, but directory rename and file moved to another directory (if filename stays the same) will be detected as "move"! And that covers vast majority of the cases and will dramatically improve performance of "mirror" option.

I'm just trying to help, I don't know if it is of any use.

[finisterre]

So to summarize:

Can FFS detect files or folders that have been renamed or moved?

What I have is folders on the source drive that have been renamed and some their contents (subfolders or files) have been moved around and renamed - and some new files added that aren't on the target (backup) drive.

This thread is long and complicated and has me confused. It looks like FFS can do this but not the first time you run a synchronization -?

Thanks. This looks like a great program.

[kdjfcsjflksj]

FFS Can detected files that habe been moved in sync mode after the first sync. however, this feature is not (yet) available in mirror mode

[gioita]

I read all the post and then try the last release, but I don't have the evidence of renamed or moved files. Renaming 1 file, the software say "delete 1 file and add 1 file", in all Synchronisation modes and also second times. Probably I miss something. Can someone explain how to have evidence of a renamed or moved files ?

[gioita]

Any news ?

[Zenju]

Implemented for 5.21: [404, Invalid URL: http://freefilesync.sourceforge.net/FreeFileSync_5.21_beta_Windows_Setup.exe]

[markling]

Will this change be making it through to Linux any time soon? I'm using FFS version 5.18. My system tells me this is the latest version.

I would be very helpful to know because I'm just at that crucial stage where I am coming to terms with the fact that I must manually deal with renamed files and folders across multiple copies of the same 60G set of data. After that, I have another, smaller but more complex set that needs syncing.

I had come to FFS for the same reason I imagine a lot of people do (aside from it seeming like the best tool available to me): multiple copies of data had got out of hand. One of the problem's main characteristics was files and folders that had been moved and renamed.

I would think it fairly easy for a tool to handle this, because I can see it easily enough when one element of data in two copies has been moved and renamed. It's just time consuming for me to do it over 60G.

FFS will at least make my job of manually syncing the data easier, becuase it pulls out the renamed files and folders and lists them among the other files that are genuinely different. This will save me time. But not as much time as I hoped.

FFS does nevertheless look brilliant. And I will be getting right on now to sync at least two sets of data with its help. It's ability to sync those files that are simply present in one set and not the other, and those files of the same name and location that have been updated in one and not the other, will save me a considerable amount of time.

[markling]

Ah well. That was painless. What a relief. There was one moved/renamed folder I had to handle manually. I think there may not have been much difference between these two. Or was FFS helping by identifying moved/renamed files and folders?

The real test will come when I get on the to the set of files where there are some real thorny differences.

[Giangi]

Implemented for 5.21: [404, Invalid URL: http://freefilesync.sourceforge.net/FreeFileSync_5.21_beta_Windows_Setup.exe]

Database support for detecting move/rename is still not available (checkbox cannot be ticked) for: Update & Custom!