Synchronize existing folders only

[diysw]

Hi,

when I search for "synchronize existing folders only" in this forum I do not find a lot, that's why I am raising this question here:

How can I setup FFS in a way, that only existing folders (written in bold) are compared and synchronized?
To save time, FFS should not search for files in the folders not written in bold, because they do not exist on the left side; how can this be achieved?

Thank you and kind regards
Thomas

In detail:
I use usually only a small subset of folders (including files) on a laptop.
The structure looks like this (e.g. as left side):
*\documents
*\documents\a
*\documents\b
*\documents\c
and has no further subfolders.

The rest of the entire structure is located on hard drive connected to my Fritzbox and can look like this (e.g. as right side):
*\documents
*\documents\a
*\documents\b
*\documents\b\a
*\documents\b\a\a
*\documents\b\a\a\a\...
*\documents\c
*\documents\c\a
*\documents\c\a
*\documents\c\d

[Plerry]

Simply use the FFS Filter.

Put
*\documents\a
*\documents\b
*\documents\c
In your Include Filter (replacing the default * (=everything)
Don't put the parent folder *\documents in your Include Filter

And because you do not want the content of any subfolders of a, b and c to be part of your sync, put
*\documents\*\
in your Exclude Filter

[Plerry]

I can not change my above reply, but I just noticed that
*\documents\a
*\documents\b
*\documents\c
must be folders, not files.
Therefore, the Exclude Filter rule
must not be *\documents\*\
but instead should be *\documents\*\*\

[diysw]

@Plerry
Thank you very much for your quick reply, but I think that this does not solve my "problem".

I do not want to adapt filters each time the structure of folders (or the scope I want to synchronize) on "the left" side changes.
I simply want to tell FFS in general:
- please synchronize only folders which do exist on the "left side".
- do not scan / compare the other one million files and folders which do additionally exist on the "right side".

Is there a way to do this using FFS?

Tanks and kind regards
Thomas

[Plerry]

> - please synchronize only folders which do exist on the "left side".
Possible by using the Update sync variant (leaves items that exist right-side only intact)
or by creating a Custom sync variant matching your use case.

> - do not scan / compare the other one million files and folders which do additionally exist on the "right side".
Not possible without using the FFS Filter. But apart from the time there should not be a difference in the sync outcome.

[diysw]

@Plerry
Thanks again für quick reply!

The point is: I want to save time and do not want FFS to compare some huge amount of folders and files which are out of scope on the "right side" located on a slow hard drive connected to my Fritz Box.

And "out of scope" can be clearly defined as follows:
Any subfolder on the right side, which has no "relative partner" on the left side, shall be skipped from Compare (and from Synchronize).

Is there any way to do this (without setting up any complicated filters)?
May be by adding such an option to a future version of FFS? 😃

Kind regards again
Thomas

[xCSxXenon]

Any subfolder on the right side, which has no "relative partner" on the left side, shall be skipped from Compare (and from Synchronize).

Is there any way to do this (without setting up any complicated filters)?
May be by adding such an option to a future version of FFS? 😃

Kind regards again
Thomas

The only way FFS can know if an item exists only on the right side is to scan everything on the right side during the comparison. It is impossible to find a "subfolder on the right side, which has no "relative partner" on the left side" without knowing what is there. You can exclude these items from a sync though. Set up a custom configuration so that items that exist on the right side only are set to "do nothing."

[diysw]

@xCSxXenon

The only way FFS can know if an item exists only on the right side is to scan everything on the right side during the comparison.

I don't think so:
For this use case FFS shall not know what is there everything is on the right side but only is there something corresponding to the left side.

Example:
Left: Main folder to be compared / synced:= LAPTOP
C:\NASCopy\documents
The following subfolders exist locally
C:\NASCopy\documents\a
C:\NASCopy\documents\z

Right: Main folder to be compared / synced: =NAS
\\192.x.y.z\NAS\documents
The following subfolders exist on NAS
\\192.x.y.z\NAS\documents\a
\\192.x.y.z\NAS\documents\b
\\192.x.y.z\NAS\documents\c
\\192.x.y.z\NAS\documents\d
\\192.x.y.z\NAS\documents\e
\\192.x.y.z\NAS\documents\f
\\192.x.y.z\NAS\documents\g
\\192.x.y.z\NAS\documents\h
...
\\192.x.y.z\NAS\documents\z

While FFS compares and detects that folder \\192.x.y.z\NAS\documents\b has no "relative partner" on the left side (because C:\NASCopy\documents\b does not exist on the left side), it shall skip folder \\192.x.y.z\NAS\documents\b completely (and all of its subfolders and files (because this takes a lot of time on a slow drive)).

That's not complicated!

And because the structure e.g. on NAS can change (e.g when I add \\192.x.y.z\NAS\documents\ee which may be still out of scope for the left side) I do not want to use filters but a simple algorithm described above.

What's the problem?

Kind regards
Thomas

[xCSxXenon]

Set up a custom configuration so that items that exist on the right side only are set to "do nothing."

[diysw]

@xCSxXenon
How can I setup such a configuration?
Can you give me an example?

[diysw]

@xCSxXenon

You mean something like this?

[xCSxXenon]

Looks about right, that looks like an older version of FFS compared to mine.
Certainly test it though, don't ever blindly trust anything lol

[diysw]

@xCSxXenon
Yes it's quite old on this machine...

And to be honest I do not trust what you recommend to me 😉, because I don't think you understood my problem properly (maybe I haven't explained it properly either).

The configuration shown above does not prevent FFS from searching (comparing) eternally long folders that should not be searched at all...

I will perhaps make a feature request about this some time.

Kind regards
Thomas

[xCSxXenon]

Maybe your reading skills are lacking

The only way FFS can know if an item exists only on the right side is to scan everything on the right side during the comparison. It is impossible to find a "subfolder on the right side, which has no "relative partner" on the left side" without knowing what is there.

[xCSxXenon]

I think I figured out what you are after after some pondering what you could mean. I feel like this is already the current behavior but let me perform some tests

[xCSxXenon]

OMG I can't believe FFS is doing this. I think I 100% understand what you are saying now!
To test, my setup is as follows:
- Folder "Desktop" containing a single subfolder "Tools". "Tools" is ~500MB and has a thousand files.
- Folder "Storage" containing a subfolder "Tools", along with ~20 other subfolders with hundreds of thousands of nested subfolders and millions of files within. "Tools" is ~500MB since it is a copy of the folder inside "Desktop" and the other subfolders are >1TB in total.
- Left side is "Desktop" and right side is "Storage"
- Custom sync - If file only exists on right side, do nothing

I think the expectation you have, which I also had, is that running a comparison would see that the "Tools" subfolder is the only folder that doesn't "exist only on the right side". This, ideally, would mean only the "Tools" subfolder on the right side gets scanned! I could have sworn this was how it's always worked, but I'm guessing I never had so much "extra" data on the right side that caused a noticeable amount of overhead.
I agree this would certainly be infinitely simpler than exclusion/inclusion filters. I'll try to consider the possible dangers of this behavior, but it would only really apply with a custom sync selection, so the risk seems low.
@Zenju - Is this something that has been considered and cut, or could it be looked into?

[diysw]

I think the expectation you have, which I also had, is that running a comparison would see that the "Tools" subfolder is the only folder that doesn't "exist only on the right side". This, ideally, would mean only the "Tools" subfolder on the right side gets scanned! I could have sworn this was how it's always worked, but I'm guessing I never had so much "extra" data on the right side that caused a noticeable amount of overhead.
I agree this would certainly be infinitely simpler than exclusion/inclusion filters.

Now you got it! 👍
And I am the first to have discovered this ❓
Now I'm waiting for further reactions... and thank for testing it!
😃

[Plerry]

> And I am the first to have discovered this ❓

Certainly not, but you are the first one with apparently such a dynamic right-side content that you don't consider using the In- and/or Exclude Filter to be practical, and is not patient enough to wait until full Comparison will end.

> Now I'm waiting for further reactions...

1) The Comparison FFS executes is universal for all sync variants (Two-way, Mirror, Update and Custom).
You can even change the sync variant after the Comparison has run, and the proposed Actions per Compared item change according to the modified sync variant.
Therefore FFS always does, and must, perform a full Comparison, because when you select or change into a Two-way sync, and in some cases a Custom sync, the full Comparison information is required.
It would be a drastic change (and loss) to the software to make it have a dedicated Compare per sync variant, and loose the option to change sync variant after running a Comparison.
Likely a change the author is not prepared to make.

2) If none of the proposed solutions above suit you, consider to create a dedicated folder on your NAS that holds your laptop information.
E.g. create a folder \\192.x.y.z\NAS\documents\LAPTOP,
and then Mirror sync
C:\NASCopy\documents => \\192.x.y.z\NAS\documents\LAPTOP
No time "wasted" any more in Comparing any other folders than the LAPTOP folder (and its subfolders) in \\192.x.y.z\NAS\documents

[diysw]

@Plerry
I don't agree with you at all.

I agree with the visions mentioned here: https://freefilesync.org/vision.php

This means that its main purpose is to speed up backup operations by examining the differences between source and target folders, and then only copy what is really needed rather than copy everything each time like disk cloning/imaging tools do.
Since synchronization is usually very fast and can be automated by creating batch jobs, it is easy to always have your important files backed up in a second location — without needing any cloud services at all.

Having worked professionally mainly in the area of software performance optimization, another goal for FreeFileSync was to have excellent performance. In other words, if I were to find a file synchronization tool that was faster than FreeFileSync I would take it as a challenge and not stop until FreeFileSync is at least equally fast.

I think The use case I have described is quite normal, but FFS does not currently fulfil the above requirements, which surprises me greatly.

I am still waiting for feedback from users who share my opinion and for feedback from the author of this software (@Zenju).
I can do without other feedback.

Kind regards
Thomas

[Zenju]

1) The Comparison FFS executes is universal for all sync variants (Two-way, Mirror, Update and Custom).
You can even change the sync variant after the Comparison has run, and the proposed Actions per Compared item change according to the modified sync variant.
Therefore FFS always does, and must, perform a full Comparison, because when you select or change into a Two-way sync, and in some cases a Custom sync, the full Comparison information is required.
It would be a drastic change (and loss) to the software to make it have a dedicated Compare per sync variant, and loose the option to change sync variant after running a Comparison.

This.

[diysw]

Ok,

The Comparison FFS executes is universal for all sync variants (Two-way, Mirror, Update and Custom) ...

... and does not fulfill the vision to have excellent performance for this common use case.

What a pity!

But fortunately Total Commander https://www.ghisler.com/index.htm can also synchronize directories and has an simple option named Only selected (in main window) ...

[xCSxXenon]

That certainly is a pitfall I didn't think of. Being able to switch between sync methods on-the-fly is definitely a handy feature. It likely doesn't get used all that often, as it's probably pretty rare for an environment to change often enough to need something like that. But it is arguably less rare than the situation described in this thread. This means that, logically, the current behavior is better.

[Plerry]

There must be pretty strong arguments to change current behavior, particularly if that means loosing features (or the unintended loss of data).

There seem to be several pretty straight forward, partially already described ways in FFS to deal with the use case at hand.
1) Be patient and deal with the longer Comparison time. Quite commonly, syncs run in the background, while the user is doing other things anyway.
2) Organize your data such that you can organize your syncs such that you avoid the potential problem
3) Use the Include and/or Exclude Filter
And/or
4) Use multiple left-right folder pairs

But the topic starter seems to be uncompromising, and not open to any of the above.
Fortunately for diysw, he/she found another solution that (better) fits his/her use case.

[diysw]

@All
I always thought that FFS is actually a great tool for hobby users and not a tool for professional synchronization of large amounts of data, which can, for example, run in the background and in which the file structure does not really matter (because it is mirrored 1:1 anyway).

@Plerry
There must be pretty strong arguments to change current behavior, ...

No, you don't have to change the current behavior but only add a new "handy feature".
At the end it is nothing else than a filter, which is set / fed by an option named "synchronize existing folders only (e.g. on the left side)".
I will help you with beta testing, because I like synchronizing files with FFS more than doing it with TC.
For this reason, I donated to FFS, even though I had been using TC for a very long time before

1) Be patient and deal with the longer Comparison time. Quite commonly, syncs run in the background, while the user is doing other things anyway.

No, I will not, because when I want to synchronize I want to synchronize and not wasting my time with a tools which does not cover a common use case using a "handy feature".
Without that feature FFS the overhead for useless compare is way to high.

2) Organize your data such that you can organize your syncs such that you avoid the potential problem.

The effort for this organization is also far too high compared to using an additional function "handy feature".

@xCSxXenon
Thank you for your support.