Parallel file operations

[hapEcat]

What is the Max number of Parallel file operations, before reaching a point of diminishing returns?
And is there any difference whether you are using Linux or W10?

[xCSxXenon]

That's a very theoretical question and depends on what your hardware is. For most people on here, a single thread will be just as fast as any other amount of threads, if not faster.

Linux has less overhead, but you aren't likely to see any noticeable changes. Maybe if you are running Windows on a really slow hard drive while also syncing to/from that same drive.

[Zenju]

https://freefilesync.org/manual.php?topic=performance

[hapEcat]

That's a very theoretical question and depends on what your hardware is. For most people on here, a single thread will be just as fast as any other amount of threads, if not faster.

While certainly not cutting edge, it's no slouch either:

HP 15-f271wm Laptop
OSs: x64 dual boot (W10/Mint21.0 Cinnamon/kernel: 5.15.0-47-generic)
Shared NTFS DATA partition
CPU: 2.16GHz Intel Pentium N3540 64-bit Quad Core
RAM: 8GB DDR3L SDRAM (1 DIMM)
GPU: Intel HD graphics (Bay Trail)
SSD: ADATA SU760 (512GB)


Using Linux, I have run a few timing tests at, 4 parallels vs 8 parallels, and the 8 appears to compare ~5 seconds quicker. The write speed does not appear to be affected by the number of parallels, if any, but more by the hardware involved… In other words, a 1/2" hose, can flow only so much water, or you need to get a bigger hose.
Hence, my question, would more parallels be beneficial, or after 8, are the gains so miniscule that it's of no real benefit.

[xCSxXenon]

You nailed it with the hose analogy. It could be improved by asking if splitting the spicket and adding a second hose from your house to the pool will fill the pool faster than the single hose. The answer is "not really".

You'll have to run more tests to know, but sounds like even 4 -> 8 threads doesn't add any benefit either

[hapEcat]

You'll have to run more tests to know, but sounds like even 4 -> 8 threads doesn't add any benefit either

I guess more "trial & error" testing is in order, but was hoping that had already been tested, and that there was a suggested number of parallel ops.

[xCSxXenon]

Since the optimal number of threads is entirely dependent on hardware/connection, it is impossible to provide any suggestions. Again, most people don't get any benefit from changing it from a single thread in most cases, so leaving it as is probably the best umbrella suggestion.

[hapEcat]

Thanks for all the input.

[cableghost]

I find a positive difference in using Parallel threads. For one such Config, I went from 4.5 hrs (using 1 thread) to 1 hr 40 mins in using multiple threads. My system below.

My question is... I started with, and the process was using 10 Parallel threads. By the time there was like 20 mins remaining, the number of threads used was dropped to 2 threads, without any change to system processing requirements. Why might this be? Pardon if I missed this in the instructions.

System:
Win 11 Pro 22H2
i7-8700
64 RAM
NVMe M.2 SSD (main)

Config: syncing HDD 7,200 <> USB 3.1-connected portable HDD

[hc530]

I wonder how high is optimal for ssd to ssd backup vs ssd to harddrive. I'm currently trying 8 as it seems safe.

[hc530]

Ended upping it to 16 threads, 10Gbps usb limiting the backup destination of qlc pny 2241 drives, pegged at 100% drive activity in taskmanager performance tab.

Good enough.

[fileinator]

Hey xCSxXenon,

First of all: this software is AMAZING, regardless of what some here say! I recently changed my version to the donation version to support all your efforts in providing a FREE sync tool that's better than anything else I came across in over 15 yrs!

With the donation edition I now have access to multi-threaded syncs, and ran into the same questions othere here did: "Am I using the optimal number of parallel operations for my source/destination? How do I know what IS the best setting?"

What I came up with was running sync tests with a folder containing multiple files sized <100kB up to 1GB. I kept changing the threads for source/destination and times how long it took for the sync to finish, making sure to delete the sync.ffs_db between tests.

I then picked the # of threads that gave me the best timing. (NO, I'm not bored, but I had to sync multiple old hard drives to my NAS and an external hard drive. I couldn't wait weeks for that to finish, and activating multi threads reduced the time it took to sync by about 75%)

After performing this manual "what's-my-best-thread-count" benchmark so often, I was wondering if there is ANY way you would consider creating such a "ideal # of threads test" as an option for us donation edition owners?

Here is what I think this could look like:
- You create an option in the "Performance improvement" section called "perform thread test". Activating that will download a folder containing many (however many you think it needs to run a test) various sized files. You could "warn" the user that this would take up however many gigabytes you need for testing.
- First Filesync would see if there is enough room on both source and destination to perform the test. If not, tell the user to free space, or run the test manually himself like I did ;)

- The sync will automatically sync that folder to the destination, starting with a thread count of 1 for source/destination. It will remember the time it took and the max. throughput to perform the sync.

- Filesync then would start "playing" with the values for the threads for both source and destination depending on how YOU would change the settings based on your knowledge about how multi-threading works. I'm no programmer, so I don't know it it's best to start increasing the source/destination by +1, or if it's better to increase them individually until a good mix has been found. Only you would know the algorithm that should be used. I completely trust your judgment on this ;) FFS obviously needs to delete its database every time so it performs a "fresh" test.

- After some time FFS should have found the "optimal" number of threads that gave the biggest performance boost.

Is something like that an option you'd consider? It would help SO much, and could push some folks off the fence to finally "upgrade" to the donation edition to have this feature ;)

Either way - keep up the great work!

[xCSxXenon]

I am not related to the development of FFS/RTS or this site, just a user who spends some time here.

With that said, this idea isn't feasible. There are so many optimizations done by the OS and storage devices that would result in unreliable results. Write-caching is a very common feature and write caches on drives are another. The specs of these items are basically never going to be the same for two people, so a universal test can't work. Self-tests are the only real way to know. Your test results are almost certainly skewed by these technologies also, not to mention the experimental file set is not perfectly representative of real-world sets ¯\_(ツ)_/¯

The rule of thumb is that multiple threads are only useful for artificially limited bandwidths.

[fileinator]

THANK YOU,

And sorry for confusing you with the creator of this software. I came across your name in this forum a lot. You've always been so helpful and knowledgeable that I assumed you are the developer.

Thanks again for your input.

[hc530]

I don't think that kind of write caching is common, the kind where you have an actual separate ssd to buffer writes, but even then you could just measure how busy the drive is as stated it shows in task manager % busy and response time. This isn't impossible technology.