Compare slows down as threads drop....

[jmonejmone]

Great Prog - Dumped SyncToy for this and donation provided - 5HN72358P91843842

I see that when the initial compare is done it flys along with 6 threads but it progressivly drops threads (and hence speed) till it crawls along. I'm syncing around 500K items with the first 100K taking only 30sec to compare but the last 100K takes minutes. Have I got a config wrong?

[Zenju]

This will be fixed in the next release (or is already in the beta):
viewtopic.php?t=1097

Thanks for the support!

[jmonejmone]

Hi - I tried the 5.11 beta and it was the same result for me. Started with 6 Threads and
- first 100K took only 12 Seconds to compare,
- second 100K was down to 4 threads but took 16sec,
- third 100K was down to 2 threads and took 30 sec.
- fourth 100K was down to 2 threads but took 2:45 min
- fifth 100K was down to 1 thread and took around 2:45 again

Towards the end you can even see the Items Found counter pause occasionally.

[jmonejmone]

FYI - Tried 5.8 and it was the same

[Zenju]

> it progressivly drops threads (and hence speed)

Sorry, this is completely unrelated to the link I posted an actually just a misconception: There is one thread per folder pair, so if you see less threads FFS is actually faster because it has processed everything already except for the threads shown. Generally it is expected to have the thread count drop more rapidly for repeated syncs because the OS buffers file accesses.

[jmonejmone]

Thanks - please ignore the bit on threads - it just seemed to be a coincidence. Is it expected for the Compare throughput to drop like it does from the first 100,000 compares being only a dozen seconds to almost 3 minutes for the last 100,000?

[Zenju]

The simple explanation is that the slowest thread, i.e. the folder which is on the slowest hard disk/network takes the longest time and the initially available parallelism that FFS could take advantage of, at this point, has been reduced to a single volume (there is nothing left to run in parallel).

[jmonejmone]

Thanks for the explanation....