File compare extremely slow using 4G connection with high/variable latency

[fappp]

Hello

My use case for FFS is
* Synchronisation of 50'000 local files with a server. Usually, only a few files have changed.
* Local: Windows 10 64 bit. Server: Samba
* Connection through a VPN (OpenVPN, UDP mode) or SFTP (SFTP Net Drive).
* Connection over WLAN and 4G-modem. Throughput usually about 20Mbit/s up and down. Latency (measured using PING) in the order of 40ms, about 25% of the packets need several 100ms, i guess 5% need so long, that PING says "timeout".

The problem is that the "comparison" phase lasts for ages. It is maybe one order of magnitude longer than using a VDSL or fibre link with the same throughput.. This makes FFS almost useless over the 4G connection (which is the only one I can use).

I guess that the problem is the high (or variable) latency of the link. I don't know how FFS makes the comparison, but I can imagine that a high latency can be compensated by sending many requests to the server without waiting for an immediate answer (large "window").

Is there any way to optimize the behaviour of FFS in this regard, e.g. by changing a parameter? Or would it need a modification of the software? Or is it impossible?

I cannot use FFS to access the server directly using SFTP as the server accepts only connections with a private key (not just a password) and FFS doesn't offer this funktionality.

Thanks and regards
Patrick

[Zenju]

You can parallelize the comparison phase by spliting the job into multiple folder pairs. Each pair is processed in a separate thread. This should reduce the latency, provided that Net Drive supports parallel access.

[fappp]

You can parallelize the comparison phase by spliting the job into multiple folder pairs. Each pair is processed in a separate thread. This should reduce the latency, provided that Net Drive supports parallel access.

Thank you for the answer.

Do you mean that the job should contain several folder pairs, and that they would then be processed in parallel by FFS? OpenVPN is able to do that, I suppose that SFTPNetDrive too. I tested it and it works.

The problem is that my folder structure is very complicated (not just 10 folders at top level with all about the same number of files)... so it's not easy at all to achieve, it would need to happen at several sub-folders levels, it would need exclusions and so on.

==> QUESTION, if this worked it would be good:
I create a job, containing 10 times the same folder. In each folder I exclude part of the folder and of the sub-folders, so that there is no overlapping between the 10 sub-jobs.
Will that work? Will FFS be able to manage the 10 sub-synchronisations correctly or will it try to write all 10 in the same "sync.ffs_db" file in the top folder, creating havoc and synchronisation errors?

It would be great if FFS could automatize parallelization, or let the user choose how many requests can be pending at the same time!

Anyways, thank you for this great software!
Patrick