Given I've had a bit of time on my hands I've retagged my music collection and so I now need to perform a full backup from my Windows machine to a Linux server (both in the house), which is about 550GB.
Using v10.19 the transfer rate was only ~20 MB/s, I tried upgrading to v10.23 but the issue persisted.
I ran iperf3 tcp tests between the Windows and Linux server and got about 85MB/s, so I thought I'd try an old version of freefilesync I had (v9.6) and it jumped to 53MB/s!
I have the free version, so if this is the restriction then I'm happy to donate as it's software I use from time to time, but the v9.6 connection settings appeared to be aligned with the v10 versions (1 for reading, 10 per connection) so I'm unsure what it could be.
Slow synchronisation from Win to Ubuntu
- Posts: 17
- Joined: 4 May 2020
- Posts: 17
- Joined: 4 May 2020
Just to add that I've gone back through the archives and have narrowed it down to a change in 10.17, if that helps?
9.6 53MB/s
10 56MB/s
10.10 56MB/s
10.15 55MB/s
10.16 56MB/s
10.17 21MB/s
10.19 20MB/s
10.23 21MB/s
If there's anything I can do this end to debug please let me know.
FreeFileSync 10.17 Oct 17, 2019
Support PuTTY private key files for SFTP login
Enable zlib compression for SFTP servers if supported
Update last sync time despite differences if nothing to do
Reduce graph total time update interval
Remember folder history not just for first folder pair
Allow unprivileged symlink creation in Windows Developer Mode
Integrate latest libcurl FTP bug fixes
Detect common invalid SFTP key file formats
Fixed startup crash caused by corrupted HDD properties
Allow SFTP access via Ed25519 key in PKIX format
9.6 53MB/s
10 56MB/s
10.10 56MB/s
10.15 55MB/s
10.16 56MB/s
10.17 21MB/s
10.19 20MB/s
10.23 21MB/s
If there's anything I can do this end to debug please let me know.
FreeFileSync 10.17 Oct 17, 2019
Support PuTTY private key files for SFTP login
Enable zlib compression for SFTP servers if supported
Update last sync time despite differences if nothing to do
Reduce graph total time update interval
Remember folder history not just for first folder pair
Allow unprivileged symlink creation in Windows Developer Mode
Integrate latest libcurl FTP bug fixes
Detect common invalid SFTP key file formats
Fixed startup crash caused by corrupted HDD properties
Allow SFTP access via Ed25519 key in PKIX format
- Posts: 17
- Joined: 4 May 2020
I've just found and tested with the 10.24 beta and the speed is back, but only if I change the number of parallel file operations.
With v10.16, 1 parallel file operation and 10 SFTP channels I get 56MB/s
With v10.24, 1 parallel file operation and 10 SFTP channels I get 21MB/s
With v10.24, 4 parallel file operation (sweet spot for my setup) and 10 SFTP channels I get 60MB/s
Is the required increase in file IO channels in 10.24 with respect to 10.16 expected?
With v10.16, 1 parallel file operation and 10 SFTP channels I get 56MB/s
With v10.24, 1 parallel file operation and 10 SFTP channels I get 21MB/s
With v10.24, 4 parallel file operation (sweet spot for my setup) and 10 SFTP channels I get 60MB/s
Is the required increase in file IO channels in 10.24 with respect to 10.16 expected?
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
Great analysis, simbun! There might indeed be an issue (maybe an update of libssh2?) that is causing a perf regression! The latest beta 10.24 "fixes" it by increasing the buffer sizes, but that is probably not tackling the underlying cause. I'll have a closer look at the 10.17 changes and report back.
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
I've found the issue after numerous tests: With FFS 10.17 FFS enabled zlib compression! Turning zlib off, fixes the speed issue!
The buffer size increase in the other topic is unrelated however: It's a different scenario where the bottleneck is latency. Toggling zlib does nothing in this case, and I confirm there hasn't been a regression for such cases in all tested FFS versions since 9.4. So the buffer size adaption in the upcoming 10.24 is still the right thing to do.
I'll now dig deeper to see what can be done about zlib slowing things down when SFTPing in the local network.
The buffer size increase in the other topic is unrelated however: It's a different scenario where the bottleneck is latency. Toggling zlib does nothing in this case, and I confirm there hasn't been a regression for such cases in all tested FFS versions since 9.4. So the buffer size adaption in the upcoming 10.24 is still the right thing to do.
I'll now dig deeper to see what can be done about zlib slowing things down when SFTPing in the local network.
- Posts: 17
- Joined: 4 May 2020
I did look at disabling compression on the server to test that, but ssh is my only access (short of carrying a monitor downstairs) and I didn't want to lock myself out :-)
- Posts: 17
- Joined: 4 May 2020
I've just tried, and turning compression support off at the server end brings the speeds back, so at least it can be controlled that way (assuming the user has that access).
Having looked around, the advice is that it probably will slow things down unless you're on a very slow network, so if you did want to offer compression, it looks like it might be worth having it off by default, and offering it in the sftp connection settings like Mobaxterm and WinSCP does.
Thanks for your support.
Having looked around, the advice is that it probably will slow things down unless you're on a very slow network, so if you did want to offer compression, it looks like it might be worth having it off by default, and offering it in the sftp connection settings like Mobaxterm and WinSCP does.
Thanks for your support.
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
The next FFS version will have an option to enable zlib (off by default):
https://www.mediafire.com/file/jloe77823qu16r4/FreeFileSync_10.24_%5BBeta%5D_Windows_Setup.exe
https://www.mediafire.com/file/jloe77823qu16r4/FreeFileSync_10.24_%5BBeta%5D_Windows_Setup.exe
- Posts: 17
- Joined: 4 May 2020
I've just tested it and the results look good! With compression off and 3 threads I'm now close to my Gigabit maximum.
60MB/s 1 parallel operation
110MB/s 3 parallel operation
To be honest I only use it once every few months and never looked into increasing the speed as it was always good enough; maybe you should somehow show what speed could be achieved with the donation edition, which I'll be getting once 10.24 is out.
Thanks again
60MB/s 1 parallel operation
110MB/s 3 parallel operation
To be honest I only use it once every few months and never looked into increasing the speed as it was always good enough; maybe you should somehow show what speed could be achieved with the donation edition, which I'll be getting once 10.24 is out.
Thanks again
