RTS Constant Syncing to NAS

[schmidtp]

Hi,

I'm having an issue setting up RTS to sync files to a NAS. I'm sure years ago I had it working correctly, but there was a change at some point and I stopped using it. After the initial sync, RTS idles during my windows session (as it should). However after putting the laptop to sleep and restarting, RTS starts FFS comparing everything again (which takes hours). I'm sure when I used it years ago, it would only sync changes and not compare everything constantly (much like what FFS GUI does). I'm just wondering whether I'm doing something wrong?

Here's how I set it up:
RTS shortcut is in the windows startup directory (and starts in the taskbar)

RTS.jpg (40.99 KiB) Viewed 1177 times

I had to add the "& exit 0" at the end as I was getting an error message with the recycle bin on X:

RTS setting were created with FFS and saved as a batch file

FFS batch.jpg (78.17 KiB) Viewed 1177 times

The batch job was setup to monitor D:\Shared\ and sync to X:\schmidtp\Files (mirror). X: is a mapped network drive

explorers.jpg (65.12 KiB) Viewed 1177 times

The Shared directory contains a lot of files (Approx 600GB), so a full sync takes anywhere from 1-2hrs to complete. After a computer restart or waking from sleep, RTS will restart and starts scanning all of the directories again under \Shared when nothing has changed between the two (after the initial 120sec wait and then taking hours). Its like theres no comparison file like there is in FFS to only scan the changed file(s).

Is it the way I've set it up to monitor?
Is there another setting I'm missing?
Am I interpreting how RTS works?
Should I be making an entry for each subfolder instead in the batch file instead of monitoring a single folder ( \Shared) with many subfolders?

Thanks for any help

Pete

[Plerry]

> I'm having an issue setting up RTS to sync files to a NAS.

For all clarity:
RealTimeSync (RTS) monitors for changes, and upon detecting any changes, will launch FreeFileSync (FFS) to perform the sync. RTS does not perform any sync itself.
This has not changed in the past years.

When launched (e.g. by RTS), FFS performed a full Compare on all folders and files in the scope of the sync configuration. That has always been the case, and also this has not changed in the past years.

> when I used it years ago, it would only sync changes and not compare everything constantly (much like what FFS GUI does)

This is not true!
As stated above, RTS does not Compare nor Sync, just launches FFS.
And FFS has always performed a full Compare before running its sync.

What may have changed, is that your amount of data in the scope of the sync has substantially increased/grown over time, and consequently, where an FFS Compare previously only took seconds or tens of seconds, now takes considerably more time.
Furthermore, you previously perhaps Compared by file Time+Size, but your picture shows that (at least now) you have FFS set to Compare by file Content. Comparing by Content obviously takes considerably more time that comparing by Time+Size; even more so if you would have many large files.
And yet even more so, because your X-drive is a remote drive, and all X-drive data (not just file Time+Size) needs to be downloaded over your network connection to your machine running FFS.

[schmidtp]

Thanks for the quick reply

>When launched (e.g. by RTS), FFS performed a full Compare on all folders and files in the scope of the sync configuration. That has always been the case, and also this has not changed in the past years.

Thanks for clarifying. Its probably been at least 7-8 years since I last used RTS and swapped to just using FFS manually. My memory might be a bit fuzzy, but I could have sworn it wasn't doing full compares when I first started using it (as it seemed almost instant like it was only updating one file). Then there was an update and I noticed it doing full compares. Maybe the full compare wasn't showing up in the task bar at the time in an older version so I didn't notice it?

>What may have changed, is that your amount of data in the scope of the sync has substantially increased/grown over time, and consequently, where an FFS Compare previously only took seconds or tens of seconds, now takes considerably more time.

Maybe slightly. But the drives that were in the old laptop were only about 1TB as well. I generally don't have huge drives in my laptops. Once they start getting too full, I offload and cleanup regularly. So they regularly sit about 500GB with files to sync.

>And yet even more so, because your X-drive is a remote drive, and all X-drive data (not just file Time+Size) needs to be downloaded over your network connection to your machine running FFS

This has always been the case, except I swapped to a newer NAS a few years back. Both were using gigabit connections so will regularly transfer at about 100MB/s with little fluctuation in speed.

>Furthermore, you previously perhaps Compared by file Time+Size, but your picture shows that (at least now) you have FFS set to Compare by file Content.

Maybe. I just used whatever the default was. I didn't even notice this until you pointed it out! This might be a plausible answer then in that case. Did this change as a default then years ago?

Is there any advantage of using mapped drives vs ftp (other than the harder setup for FTP)? I've had to use FTP on the Linux version of FFS, but Windows is easy to just map a drive to the NAS and work that way. SAMBA and FTP both transfer at about the same speed on my network (100MB/s).

Cheers Pete

[Plerry]

If you need to Compare by Content 500GB over a 100MB/s connection, you should expect the Compare to last at least 5000 seconds (plus a notable overhead in case of many small files). So that seems to fit your observation.
If comparing by Time+Size function-wise fits your needs, simply change the Compare setting and see how notably much faster it is.

I am not aware that the default Compare method would have changed.
You can try your default by clicking New in the top-left Configuration window in the FFS GUI, and then check which Compare method is indicated.
However, if you open and modify an existing *.ffs_gui or *.ffs_batch configuration, the Compare method of that configuration will be retained, unless you modify it.

FTP is a very fast protocol; a bit faster than the SMB/SAMBA protocol you (seem to) use for your mapped drive. But, like you indicate yourself, those differences should not be dramatic.
However, I avoid using FTP wherever/whenever possible for syncing, because (almost?) all FTP servers are not able to retain file-dates when copying a file. So the file gets the timestamp of the time of copying over; not the timestamp of the original file.
For my use, I find that highly objectional.
It makes the combination of a Compare by Time+Size with a Mirror Sync variant very impractical.
Assuming your right-side is an FTP location, all right-side files would be "newer" than their left-side counterpart (if that still exists) and would be copied over left-to-right (every sync) again.
You could overcome that by defining a suitable Custom sync variant, but still I find the difference in timestamp objectional.

[schmidtp]

Thanks again for the reply

I ended up changing from "file content" to "file time and size" which made a huge difference. Instead of +90min to do a full comparison, it only took a couple of seconds to detect the changes and then about 5 minutes to sync about 20GB of data. That must have been what it was set to when I first started using it. Although I can't remember changing any defaults on it.

Afterwards, I read up on FTP and SFTP suggestions. But when I tried to run RTS, it gave me an error message "Directory Monitoring wasn't support in FTP and SFTP". Although I though I was only monitoring D:\Shared, but apparently it was monitoring both. So I removed the Remote entry from RTS before saving it as a ffs_real file. I also set SFTP with 8 channels (although the server supports 10).

So for those playing at home, with a 100K files or so in the directory to be checked and changing 1x4GB file:
A mapped X: drive scan took about 25sec and transferred at 105MB/s
SFTP scan was under 10s and transferred at about 80MB/s
FTP scan took a whopping 4m10s and transferred at about 110MB/s

FTP was definitely the loser in all of this as the initial scan took 4min when started, another 4min when something was deleted, then 4 minutes again to scan when something was added, plus another 60s of so to transfer the actual file.

>However, I avoid using FTP wherever/whenever possible for syncing, because (almost?) all FTP servers are not able to retain file-dates when copying a file. So the file gets the timestamp of the time of copying over; not the timestamp of the original file.

I checked the time stamps after using FTP and for me at least it retained the time and date stamps?

All up, either SFTP or SMB seem to work well. Although I was getting errors occasionally using any FTP setup. But I can deal with any odd error and just hit retry if need be. At least its not taking hours to sync anymore!

[Zenju]

A mapped X: drive scan took about 25sec and transferred at 105MB/s
SFTP scan was under 10s and transferred at about 80MB/s
FTP scan took a whopping 4m10s and transferred at about 110MB/s

This doesn't sound right. SFTP should almost always be slower than FTP(*). In any case, both FTP and SFTP only give decent comparison performance when used with multiple threads: https://freefilesync.org/manual.php?topic=performance

*) Perhaps it's this libcurl bug:

FreeFileSync 12.1 [2023-02-20]
------------------------------
Fixed slow FTP comparison performance due to libcurl regression

[schmidtp]

This doesn't sound right. SFTP should almost always be slower than FTP(*). In any case, both FTP and SFTP only give decent comparison performance when used with multiple threads: https://freefilesync.org/manual.php?topic=performance

SFTP is usually 20% slower transferring files on my server for whatever reason, regardless of whether I'm using Windows or Linux using a filemanager (Explorer or Dolphin). The transfer speeds I was getting confirms this (80MB/s vs 100MB/s). I just put the comparison speed down to the fact I was using 8 channels instead of one as in FTP. But they were all still only using one connection. I remember reading something on the site/forum that FFS only used the channels when comparing, not when transferring files to sync.

*) Perhaps it's this libcurl bug:

FreeFileSync 12.1 [2023-02-20]
------------------------------
Fixed slow FTP comparison performance due to libcurl regression

Maybe? Although I'm using the latest version 12.3 ATM. If there's any info you need (log files etc), let me know.

And BTW - This morning RTS is sitting there peaceful after waking from sleep, just waiting for changes like it should.

[Plerry]

> SFTP is usually 20% slower transferring files on my server for whatever reason ...

That reason is clear; stated oversimplified:
SFTP is FTP, but adds encryption at the sending side and decryption at the receiving side.
This encryption/decryption takes extra time.

[Zenju]

I just put the comparison speed down to the fact I was using 8 channels instead of one as in FTP. But they were all still only using one connection.

That was not a fair comparison: SFTP in this case had 8-time parallelism vs FTP which had only one. A meaningful test would have been e.g.
FTP: 8 threads
SFTP: 8 threads, 1 channel or 2 threads, 4 channels

That reason is clear; stated oversimplified:
SFTP is FTP, but adds encryption at the sending side and decryption at the receiving side.
This encryption/decryption takes extra time.

That's not it, the bottleneck for both protocols is network latency, not CPU time. Getting the folder contents for FTP (either with or without encryption) takes one network round trip, while for SFTP it takes at least three (!) (open directory handle, read list of files, close directory handle).

[schmidtp]

That was not a fair comparison: SFTP in this case had 8-time parallelism vs FTP which had only one. A meaningful test would have been e.g.
FTP: 8 threads
SFTP: 8 threads, 1 channel or 2 threads, 4 channels

Well it is when FFS only has options for 1 connection and channel with FTP and SFTP has the option of 8 channels with its 1 connection. At the end of the day, I just wanted to know what the faster option was to use when sync'ing.

That's not it, the bottleneck for both protocols is network latency, not CPU time. Getting the folder contents for FTP (either with or without encryption) takes one network round trip, while for SFTP it takes at least three (!) (open directory handle, read list of files, close directory handle).

So if SFTP is supposed to be slightly slower by it's nature, even with 8 threads going why is FTP performing so badly against it? I mean by my time measurements of about 4 min per transaction, SFTP would be 4min div 8 (which would be about 30sec). Its doing compares even faster than that?

[Plerry]

> That's not it ...
I stand corrected!

[xCSxXenon]

Just chiming in on the original issue here lol
When your laptop wakes up from sleep, the connection to the mapped drive is usually going to take a little bit to reestablish. RTS sees this as a 'missing' drive and triggers an 'initial' sync once it is available again

[schmidtp]

When your laptop wakes up from sleep, the connection to the mapped drive is usually going to take a little bit to reestablish. RTS sees this as a 'missing' drive and triggers an 'initial' sync once it is available again

True, although I would think that the 10 sec delay would fix that? Even if you set it to 20-30 sec maybe better?

Once I had it running correctly again, I'd planned on making the gap even longer as I don't need it to sync every 10 sec. That could blow out to 5-10 minutes.

[xCSxXenon]

The idle time is irrelevant to that issue. The idle time is simply the amount of time that needs to pass since the last change before the command is triggered. An inaccessible location becoming accessible is regarded as a change, so the command is triggered. This is a feature that enables automatic syncing upon insertion of a location, like a USB drive