Optimizing the synchronization of a large quantity of files

Get help for specific problems
Posts: 3
Joined: 27 Dec 2020

maketakelake

Hello!
I need to sync local folder with ~2 millions files with 40 network fileshares.
New files (small amount) appear on these fileshares and are copied or replaced to the local folder.
Because of the large number of files in the local folder the comparison process is very long and my RAM and swap are completely used up (32GB RAM).
Sometimes it makes the local folder (it is also a network folder) inaccessible and this is very bad.
How can I optimize this process?
I have only one idea - synchronize all network shares with an empty folder on the local computer, and already synchronize it with the final folder locally.
Posts: 3
Joined: 27 Dec 2020

maketakelake

And after synchronization my RAM is not cleared until I close the program. And closing the program in this case is very long (5-10 minutes).
User avatar
Posts: 4867
Joined: 11 Jun 2019

xCSxXenon

You could run a batch script that will run each pair independently and one-by-one. This will do a compare on one pair, only filling up a small amount of RAM, run the sync, then close freeing the RAM up.
Posts: 3
Joined: 27 Dec 2020

maketakelake

Thanks, I did so.
First I sync network fileshares with empty folder and then copy\replace files to original folder.