FreeFileSync performs the scan of folder pairs in parallel, when sources / targets reside on separate physical units.
It it possible to implement this for the actual sync procedure as well? Say I have the following folder pairs set up, to sync my local drive to three separate targets:
Local Drive -> USB Drive
Local Drive -> LAN Server
Local Drive -> Internet Server
At the moment they are processed sequentially, which is kind to the source drive and RAM buffers, but wastes enormous amounts of time. I would love for it to push data towards all targets simultaneously!
Either let the source "thrash" to keep each outgoing link saturated - minimizing the total transfer speed (preferable! esp. since thrashing isn't an issue for SSDs). Or make reads sequential but writes parallel: upload one file to all targets, when all are done, move on to the next file. This would limit each target to the slowest destination, but it would still be faster than sequential uploads.
Does this seem like a reasonable idea?
Parallel uploads for folder pairs? (feature request)
- Posts: 16
- Joined: 19 Jul 2017
- Posts: 16
- Joined: 19 Jul 2017
Bump!
My wish for 2018: parallel - simultaneous - uploads to each target. Please!
My wish for 2018: parallel - simultaneous - uploads to each target. Please!
-
- Site Admin
- Posts: 7212
- Joined: 9 Dec 2007
Great feature, it's on the todo list!
- Posts: 2
- Joined: 4 Jan 2018
Oh wow, perfect timing, I was just thinking of this!
I mirror sync two drives on two local computers in a similar manner
PC1/drive1 -> PC2/drive1
PC1/drive2 -> PC2/drive2
And I've found it's faster to run two instances of FFS and sync each drive in a separate job simultaneously. FFS allows two instances to run when they are not accessing the same drive at the same time and I can maximize my transfer rate this way. The downside is that I have to update exclusions and rules in two separate sync configurations. It would be amazing if I could run one job and do it all together (with the performance benefits).
Glad to hear it's on the todo list!
I mirror sync two drives on two local computers in a similar manner
PC1/drive1 -> PC2/drive1
PC1/drive2 -> PC2/drive2
And I've found it's faster to run two instances of FFS and sync each drive in a separate job simultaneously. FFS allows two instances to run when they are not accessing the same drive at the same time and I can maximize my transfer rate this way. The downside is that I have to update exclusions and rules in two separate sync configurations. It would be amazing if I could run one job and do it all together (with the performance benefits).
Glad to hear it's on the todo list!
- Posts: 16
- Joined: 19 Jul 2017
Bumping this again. :)
-
- Site Admin
- Posts: 7212
- Joined: 9 Dec 2007
Done for FFS 10!!! :) Release in the next few days...
-
- Site Admin
- Posts: 7212
- Joined: 9 Dec 2007
Upsie... Didn't read correctly. FFS 10 *will* have parallel file I/O at the device level for use with comparison *and* sync. Parallel processing of multiple folder pairs as described here will be relatively easy to implement with the new FFS 10 infrastructure (both comp and sync has been completely revamped to handle parallel I/O!). But what 10 has is parallel I/O when syncing single folder pairs (e.g. 10 parallel FTP connections)Done for FFS 10!!! :) Release in the next few days... Zenju, 24 Apr 2018, 16:04
- Posts: 16
- Joined: 19 Jul 2017
That is great news! Looking forward to the update! I'll make sure to put a few more bucks in the hat too.
- Posts: 2
- Joined: 4 Jan 2018
Amazing!Done for FFS 10!!! :) Release in the next few days... Zenju, 24 Apr 2018, 16:04
