Parallel file operations

Discuss new features and functions
Posts: 25
Joined: 29 Mar 2022

hapEcat

What is the Max number of Parallel file operations, before reaching a point of diminishing returns?
And is there any difference whether you are using Linux or W10?
User avatar
Posts: 3551
Joined: 11 Jun 2019

xCSxXenon

That's a very theoretical question and depends on what your hardware is. For most people on here, a single thread will be just as fast as any other amount of threads, if not faster.

Linux has less overhead, but you aren't likely to see any noticeable changes. Maybe if you are running Windows on a really slow hard drive while also syncing to/from that same drive.
Posts: 25
Joined: 29 Mar 2022

hapEcat

That's a very theoretical question and depends on what your hardware is. For most people on here, a single thread will be just as fast as any other amount of threads, if not faster.xCSxXenon, 03 Sep 2022, 14:09
While certainly not cutting edge, it's no slouch either:
HP 15-f271wm Laptop
OSs: x64 dual boot (W10/Mint21.0 Cinnamon/kernel: 5.15.0-47-generic)
Shared NTFS DATA partition
CPU: 2.16GHz Intel Pentium N3540 64-bit Quad Core
RAM: 8GB DDR3L SDRAM (1 DIMM)
GPU: Intel HD graphics (Bay Trail)
SSD: ADATA SU760 (512GB)
Using Linux, I have run a few timing tests at, 4 parallels vs 8 parallels, and the 8 appears to compare ~5 seconds quicker. The write speed does not appear to be affected by the number of parallels, if any, but more by the hardware involved… In other words, a 1/2" hose, can flow only so much water, or you need to get a bigger hose.
Hence, my question, would more parallels be beneficial, or after 8, are the gains so miniscule that it's of no real benefit.
User avatar
Posts: 3551
Joined: 11 Jun 2019

xCSxXenon

You nailed it with the hose analogy. It could be improved by asking if splitting the spicket and adding a second hose from your house to the pool will fill the pool faster than the single hose. The answer is "not really".

You'll have to run more tests to know, but sounds like even 4 -> 8 threads doesn't add any benefit either
Posts: 25
Joined: 29 Mar 2022

hapEcat

You'll have to run more tests to know, but sounds like even 4 -> 8 threads doesn't add any benefit either xCSxXenon, 04 Sep 2022, 14:04
I guess more "trial & error" testing is in order, but was hoping that had already been tested, and that there was a suggested number of parallel ops.
User avatar
Posts: 3551
Joined: 11 Jun 2019

xCSxXenon

Since the optimal number of threads is entirely dependent on hardware/connection, it is impossible to provide any suggestions. Again, most people don't get any benefit from changing it from a single thread in most cases, so leaving it as is probably the best umbrella suggestion.
Last edited by xCSxXenon on 06 Sep 2022, 14:16, edited 1 time in total.
Posts: 25
Joined: 29 Mar 2022

hapEcat

Thanks for all the input.
Posts: 1
Joined: 29 Jan 2023

cableghost

I find a positive difference in using Parallel threads. For one such Config, I went from 4.5 hrs (using 1 thread) to 1 hr 40 mins in using multiple threads. My system below.

My question is... I started with, and the process was using 10 Parallel threads. By the time there was like 20 mins remaining, the number of threads used was dropped to 2 threads, without any change to system processing requirements. Why might this be? Pardon if I missed this in the instructions.

System:
Win 11 Pro 22H2
i7-8700
64 RAM
NVMe M.2 SSD (main)

Config: syncing HDD 7,200 <> USB 3.1-connected portable HDD