Hi!
I have to synchronize some really huge directories (total file count > 12 mio
files).
I already divided it into subdirectories with 1 mio files, but freefilesync
crashes during the load of the directory tree at about 500000 files.
Any idea?
System: Win XP 32bit 4 GB RAM effectively free 3.3 GB)
sync some HUGE directories
- Posts: 2
- Joined: 13 Jan 2003
-
- Site Admin
- Posts: 7279
- Joined: 9 Dec 2007
What do you mean with "crash". Is it an out of memory?
- Posts: 2
- Joined: 13 Jan 2003
I did not make a screenshot, sorry.
As far as I remember, a standard Windows alert (no custom freefilesync
message) came up stating that the application cannot continue and has been
aborted.
Reproducing it is possible but quite expensive - the PC would need many hours
and would consume most of network bandwidth to the share during this time. If
you think the exact error message is of great help to you, I could run it
again next week.
The system has 3.3 GB RAM usable in XP, OS and other applications (only
explorer) consume about 0.5 GB, so about 2 GB should have been available to
freefilesync - if it demands maximal memory. Assumed that the directory tries
are on the heap and only restricted by available memory, out-of-memory with 1
mio files at source and target would mean the memory footprint of one file of
the directory trees within freefilesync is at least 10 KB - which is
unexpected high. I would expect 20-100 bytes with usual file names (in my
case, all below 20 char-s long), including object management overhead.
As far as I remember, a standard Windows alert (no custom freefilesync
message) came up stating that the application cannot continue and has been
aborted.
Reproducing it is possible but quite expensive - the PC would need many hours
and would consume most of network bandwidth to the share during this time. If
you think the exact error message is of great help to you, I could run it
again next week.
The system has 3.3 GB RAM usable in XP, OS and other applications (only
explorer) consume about 0.5 GB, so about 2 GB should have been available to
freefilesync - if it demands maximal memory. Assumed that the directory tries
are on the heap and only restricted by available memory, out-of-memory with 1
mio files at source and target would mean the memory footprint of one file of
the directory trees within freefilesync is at least 10 KB - which is
unexpected high. I would expect 20-100 bytes with usual file names (in my
case, all below 20 char-s long), including object management overhead.
-
- Site Admin
- Posts: 7279
- Joined: 9 Dec 2007
It's almost certainly an out of memory problem (for an old discussion see [404, Invalid URL: https://sourceforge.net/tracker/?func=detail&atid=1093080&aid=2927120&group_id=234430]
I made a quick test, setting source and target directory to my hard drive
"C:\" containing 420.000 files and folders:
The 32 bit FreeFileSync release required a 174 MB memory peak during
comparison.
The 64 bit FreeFileSync release required a 278 MB.
Since you're using 32 bit, assuming source and target both having 12 million
files, which match exactly, extrapolating to 12 million files and folders
gives a peak memory consumption of around 4,6 GB.
PS: for checking peak memory consumption I suggest using Sysinternals Process
Explorer showing Peak Private Bytes. This is the "virtual" memory that
application requests from the operating system explicitly.
I made a quick test, setting source and target directory to my hard drive
"C:\" containing 420.000 files and folders:
The 32 bit FreeFileSync release required a 174 MB memory peak during
comparison.
The 64 bit FreeFileSync release required a 278 MB.
Since you're using 32 bit, assuming source and target both having 12 million
files, which match exactly, extrapolating to 12 million files and folders
gives a peak memory consumption of around 4,6 GB.
PS: for checking peak memory consumption I suggest using Sysinternals Process
Explorer showing Peak Private Bytes. This is the "virtual" memory that
application requests from the operating system explicitly.
-
- Site Admin
- Posts: 7279
- Joined: 9 Dec 2007
I've done a detailed analysis of peak memory consumption and added the result
to the old tracker, to keep the topic at one place:
[404, Invalid URL: https://sourceforge.net/tracker/index.php?func=detail&aid=2927120&group_id=234430&atid=1093080]
to the old tracker, to keep the topic at one place:
[404, Invalid URL: https://sourceforge.net/tracker/index.php?func=detail&aid=2927120&group_id=234430&atid=1093080]
