Understanding the file compare process (re: speed)

[Garbonzo17]

Hi,
I have used freefilesync off and on for small tasks over the last decade (since a v2.x iirc) but never as part of a routine workflow. And about 2 years ago I made a mistake while syncing my media library and lost almost 1/3 of it... my fault, a f/u of an oversight that I am only now making whole..

Now, new setup... moved to unraid, but still use a windows drivepool for my backup.
- first drive sync happens at a pretty consistent speed...
- second sync (weekly) from the saved session compares like 282,000-ish files in like 10 seconds, then goes through the other 500,000 over the next 15-20 mins...
I can only guess this increase speed has to do with the db files used, but need to look into that more... not sure if I need anything increased (memory, threads, etc) to optimize this some, but it seems like it always bottlenecks at around 282k, no matter if I put the TV folder or the Movie folder first...
the content is only media files and metadata/images/subs for the media.. so it's all pretty similar stuff.. and it was all just recently moved to the unraid box so that end is pretty sequential, but I suppose it could be getting hung up on the windows drivepool side...
So far, this has been nearly identical on a standalone windows 10 pc, and with the usb controller passed through to a windows 2022 server vm with the same drivepool setup... So as I am typing this I am thinking that could be the common denominator...

SO, I guess TLDR; would be:
Before I blow away this drivepool (using stablebit drivepool) and either remake from scratch, or with windows native drivepool, or use mergefs or similar under linux, is there some reason as to why this should behave this way that I can change/tinker with to see what makes it better or worse? (fast compare for 1/4million then SLOW for another 1/2 mil)

TIA for any help, I realize that was probably alot, and I should do more research, but before I experiment with other filesystems that may not make a difference, I thought I would throw this question out and see if someone can help... my googling hasn't yeilded much help so far... but just typing this has helped me, as I am realizing that I have gotten similar results over 1gb, 10gb ethernet and USB3, so it has got to be something with the data-set used for comparison. I just don't know how to adjust...

-G

[xCSxXenon]

It's probably filling up a buffer and then slowing down. How much RAM do you have?
https://freefilesync.org/faq.php#limitations

[Garbonzo17]

It's probably filling up a buffer and then slowing down. How much RAM do you have?
https://freefilesync.org/faq.php#limitations

I have as much as needed (its currently on a unraid box with 72gb, so I can throw more at it but doesn't seem necessary), its in a VM with only 8gb but during the compare process my memory never never really goes above 35% and it idles around 30... and the CPU never passes 20% (in taskman) so it seems like I'm hitting a ceiling in the software.. I just don't know if there is a dial somewhere that will help... but I will read your link now, and thanks for input.. I'd like to make this work more efficient, but it is getting done.. Just always bottlenecks at 288k...

[xCSxXenon]

Maybe it's filling up a drive cache?

[xCSxXenon]

I ran a test with 1,000,000 empty txt files and noticed a significant decrease after about 330,000 files. It went from 1,200 items/s to <500. For me, the active time for the drive in Task manager was 100%. This was folder to folder on a local NVMe drive. You are likely filling up a cache.