Detection of moved and renamed files

[nick2011]

Version 4.4 has a new feature 'Detection of moved and renamed files' but when
I rename a file sincronize is done in 2 steps: delete the file on destination
and copy the file (like in previous versions).
In that case which is the difference ?

[Zenju]

The new feature requires the database to detect renaming and moving. So you'll
have to choose `<automatic>` variant.

[nick2011]

I need to change only right side so I have to choose Mirror.
Database works only with automatic variant ?

[Zenju]

Yes, database is created only for variant <automatic>.

[geoff987654]

I am also having problems with renaming folders and files.
I have the sycn varient set to Automatic (it appaers in grey above the
synchronise button).
when I run compaire the preview show the files being deleted from the old
folders and then copied to the new folder.
I stugle to understand quiet how this works.
Must I be using compaire contents?
Does it only work with files not folders ?
Is there a different icon for renamed files.

Thanks
Geoff

[Zenju]

Detection of renamed files works beginning with the *second* sync using
<automatic> mode with either "time + size" or "binary" comparison. The
algorithm applies to files, not folders and can be recognized by a distinct
icon in middle grid.

[geoff987654]

Thank you for the clarification.
I guess asking for the algorithm to work with folders as well is too much to
ask for.
I guess I will just have to be more careful in choosing the names for my
folders.

Geoff

[Zenju]

Folders are very problematic in the context of moving/renaming. In the most
trivial of cases it's possible, e.g. if only the folder is renamed, but none
of it's child-folders and files is changed. In all other cases, one basically
would have to determine the list of changes the user has applied to this
folder in the correct order. This can become very complex quickly. Then trying
to visualize these changes on GUI so that the user gets a preview is futile
... except for the trivial case.
An even bigger problem: Applying such a consecutive list of changes to a
folder is not transactional (every folder rename operation affects all
children!). If anything goes wrong, e.g. power-loss, etc. the user's data may
be doomed (= left behind in an inconsistent state).

However let's note that detection of moved files is a performance optimization
only. And for detection of moved folders one finds that usually there is no
benefit at all, except for extreme cases where the costs to rename all child-
elements is actually noticable.

[yppip]

After reading the previous posts about 'Detection of moved and renamed files'
resulting in the time saving 'move' process applied in the target folders, I
presumed that when moving/renaming folders, both the folders and the files
they contained would be deleted and recopied as in mirror mode.
However having experimented with ver.4.5 in Automatic mode, I was both
surprised and delighted to see that although the moved/renamed folders were
deleted and recreated, the files that they contained were still only moved, no
matter how complicated the folder structure. Not only saving time, but even
better, storage space, by only putting moved/renamed folders and not their
files in versioning folders.
Brilliant, the best syncing software just got even better. Thanks.

[Giangi]

Isn't it possibile to create/manage the database sync.ffs_db for each sync
mode? Me too I would like to have the move detection at least on the
mirror sync mode!

Anyway, mee too I would like to thank you for this great program!!!

[geoff987654]

Thank you for the detailed reply. Your customer support puts many the
commercial organisations to shame.
For your information to explain why moving folders, rather than deleting and
creating a new folder, would be beneficial.
I have following system
plain files folder <=>sync program<=>encf for Windows<=> trend micro safesync
folder
Any file written to the plain file folder ends up in my tend Micro sayfsync
account in encrypted form.
The system also works in reverse
Any file(pre-encrypted) written to the SafeSync account appears in my plain
file folder in an unencrypted form.
I did this beacuse the output of encf for window cannot be shared over the
network.
The safesync software is clever enough to reconize moved and renamed files,
saving a lot of tome re-uploading files.
If the sysn program does not also do this the chain is broken and moved files
have to be uploaded.
Historically so that programs could find current data all current year folders
had a standard name. At the end of the year the current folders were renamed
with apropriate year number.
We are moving away from this to cut down the amount of renaming and hence
reloading.

Geoff

[Zenju]

> create/manage the database sync.ffs_db for each sync mode
Maybe worth considering. I like the idea of not having an additional
configuration option to detect moves in <automatic> mode. But the "costs" are
design-only, implementation is trivial.

>I presumed that when moving/renaming folders, both the folders and the files
they contained would be deleted and recopied as in mirror mode.
I think this is a common misconception, so I want to make this clear: When I
say FreeFileSync does not detect moved folders, I mean the folder only, not
its child-elements. All files contained, no matter how deep in hierarchy, are
properly detected as a file move. In this context there is *no* difference
between a file renaming, or a move in hierarchy (or both).
In general, just trying this feature, may already answer most questions! ;)

[Giangi]

I may have found a bug into the move detection...

For semplicity (and having different base name) I have executed on all my
three computers the command subst R: %USERPROFILE% because SyncToy do not
allow using envars...

Anyway, if a Mirror job is using on both side the virtual drive R: it
will always delete&copy and never move!
I'm almost sure is related to the way the subst'ed drive is handled (I'm using
Seven).

I know that FFS do support envars, but I thought it was worth reporting
that behaviour! :-)

[Anonymous]

Hi

when I move some files from "folder A" to "folder B" and sync, FFS recopy the
whole things again instead of "moving" within the machine. I sync through
"internet" with VPN, this makes sync super slow. I just wonder if there
anything I can make it better?

thanks!

[Zenju]

> I may have found a bug into the move detection...
Indeed! This subst was able to out-smart FFS! :) Not anymore, fixed:
[404, Invalid URL: http://ifile.it/srn36fo/FreeFileSync_v4.6_beta_setup.exe]

> through "internet" with VPN
Unfortunately file ids, the technical vehicle that allows for detection of
moved files, is not supported by all file system implementations.
But in general: all locally mounted file systems should be fully supported, no
matter how complex the setup (symbolic link, to some "subst"-included
volume...etc.)

[Giangi]

> Indeed! This subst was able to out-smart FFS! :) Not anymore, fixed:
[404, Invalid URL: http://ifile.it/srn36fo/FreeFileSync_v4.6_beta_setup.exe]


You've been too quick! :-)

[Anonymous]

Is there a possibility that a detected moved/renamed pair of files are
displayed in the same line (left and right panels)? At present it is difficult
to find out what are corresponding files on both sides.

[Zenju]

I don't think it is considering all the design constraints. However FFS
compensates by showing both source and target in middle grid's tool tip.

[eugen7]

First, thanks for the product. its good...
I use it for backup just to copy left to right. Mirror Variant. v4.4 came up
with database to detect moved and rename. Awesome feature. So i change to
Automatic Variant but for a Mirror purpose. All i do is un-tick changes on the
left MANUALLY to achieve it.

Can you add an option to disable any changes to left side for Automatic
Variant ? Or maybe option to mass untick changes on left side ?

[Zenju]

> Or maybe option to mass untick changes on left side ?
You can CTRL+A and select all rows of a certain view category, then change
direction via ALT+<arrow. Or do the same with a mouse selection + right-click
context menu.

[profmayer]

Hi,
Thanks a lot for that great programme :) I'm using v5 and i would be happy to
have that detection feature on mirror sync too.
Would it be possible, like in GIT i think, to make a table of each file with
SHA1 or "size and modif time" signature. Using it instead of file ids would
allow to have the feature on any filesystem and on the first mirroring. Could
then also add an option to replace doubles by only one file and symlinks to it
for the other location.
example :
dir1 :
file1.txt 50657 Bytes Modified 02-03-2011 xxxxx SHA1 xxx
file2.7Z 23486509479 Bytes Modified 01-03-2011 yyyyy SHA1 yyy
subdir/file2.7Z 23486509479 Bytes Modified 01-03-2011 yyyyy SHA1 yyy

dir2:
file1.txt 50657 Bytes Modified 02-03-2011 xxxxx SHA1 xxx
newdir/file2.7Z 23486509479 Bytes Modified 01-03-2011 yyyyy SHA1 yyy

instead of del and copy file2.7z, detect dir2 has somewhere a file with the
same signature ("size and modif time" or SHA1) and does a move of the file
from dir1/newdir to /dir1 and a symlink to it in dir1/subdir.

In case of SHA1, the table could be saved where the user want to not redo the
digest again or allow integrity check.
Thanks again :)

[mark1990]

Hello, I was wondering whether FFS is able to recognize filemovements by a
third party on a network drive, when the computer that runs FFS is turned off.
Will it then still be able to see that a file has merely moved or will it
remove the file on the computer and then retransmit it from the network drive
to the computer? I'm guessing that FFS is able to determine
filemovements/renames by means of the service running in the background and
that when it is not running it will not be able to do so for a network drive.
But I hope I'm wrong.

I think I saw somewhere a request for somebody to write a FFS package for
Synology DiskStations...? :)

[Zenju]

FFS is able to detect any file movements, by using file ids. This is a file
system feature supported by all local Windows file systems and generally on
Unix. But it might not be available for some arbitrary mapped NAS volume. Just
try and see if it works. And since this requires the database file as
reference, this featuer needs "automatic" sync variant.

>Synology DiskStations
Never heard of those.

[mark1990]

Tried it and it works without problems! Only 'problem' is that rescanning all
files on the NAS takes long compared to scanning files on the computer which
is sped up by threading. But that ain't something FFS can help.

[bkeadle]

I would like to add to the chorus for adding move detection for Mirror Sync. I
have submitted this as a Feature Request on the tracker page.

[bkeadle]

In an attempt to accommodate not having move detection for the Mirror variant,
I thought I might be able to use environment variables in my gui/batch
settings to define the variant as either Automatic or Mirror - I found that I
cannot. Would it be possible to allow for a vairant variable?


<SyncConfig>
<Variant>%VARIANT%</Variant>



Because I want to enable move detection between a folder pair, I (currently)
need to use the AUTOMATIC variant. But if differences are detected in both
causing a conflict, I need to be able to switch to the MIRROR variant,
defining my left side as master. This would be a 2 step process: run Automatic
variant ignoring the conflicts to do a best attempt sync (including move
detection), then re-run the sync as Mirror variant to make the directories in
sync.

If move detection could be enabled for the Mirror variant, then I could always
define my left as Master, and Mirror sync to the right and still take
advantage of "move detection" - an AWESOME feature!

[Zenju]

If what you really want is a "mirror" but you use "automatic" mode instead in
order to have move detection, then this should only be used in GUI mode, where
you have have a preview. Else you won't notice if there were changes on target
side that are propagated to the source, with conflicts being only a special
case of this category.

[bkeadle]

??? I have a source I deem the master that I want to mirror to the target, but
the source would like have moved files. With mirror today, that would mean
deleting from the target, then doing a new copy of files that had already
existed on the target but in a different location, right?

[Zenju]

Right. But instead of using mirror you can "abuse" automatic mode and just
manually look that all directions are one-way. As long as you are only
modifying one side, but not the other, e.g. a backup medium, you have the
benefit of detection of moved files.

Conceptually: 1. Mirror models "owner -> observer" semantics 2. automatic
models "shared ownership". The right variant should be chosen according to
these models.

[bkeadle]

So that's a "No" for enabling move detection for Mirror?