Hello!
I need to sync local folder with ~2 millions files with 40 network fileshares.
New files (small amount) appear on these fileshares and are copied or replaced to the local folder.
Because of the large number of files in the local folder the comparison process is very long and my RAM and swap are completely used up (32GB RAM).
Sometimes it makes the local folder (it is also a network folder) inaccessible and this is very bad.
How can I optimize this process?
I have only one idea - synchronize all network shares with an empty folder on the local computer, and already synchronize it with the final folder locally.
Optimizing the synchronization of a large quantity of files
- Posts: 3
- Joined: 27 Dec 2020
- Posts: 3
- Joined: 27 Dec 2020
And after synchronization my RAM is not cleared until I close the program. And closing the program in this case is very long (5-10 minutes).
-
- Posts: 4867
- Joined: 11 Jun 2019
You could run a batch script that will run each pair independently and one-by-one. This will do a compare on one pair, only filling up a small amount of RAM, run the sync, then close freeing the RAM up.
- Posts: 3
- Joined: 27 Dec 2020
Thanks, I did so.
First I sync network fileshares with empty folder and then copy\replace files to original folder.
First I sync network fileshares with empty folder and then copy\replace files to original folder.