Below are the testresults I made to compare FreeFileSync to other software.
The software had to copy 5,088,304 bytes (2553 files & 69 folders) from my
laptop to a sdhc memory card and as you can see FreeFileSync is a lot slower
than the others. Anybody noticed this too?
DirSyncPro 15:26 15:11 15:16
FreeFileSync 19:02 18:51 19:08
Win XP Explorer 15:17 15:15 15:30
TeraCopy 13:54 13:51 13:51
FreeFileSync slow comparing to other software
- Posts: 2
- Joined: 8 Jul 2008
-
- Site Admin
- Posts: 7058
- Joined: 9 Dec 2007
These performance numbers test a very specific synchronization scenario,
namely "copying a lot of small files to a slow target medium with buffering
disabled. Consequently the times will be dominated by number of accesses per
file rather than transfer speed. (latter will be pretty much the same for all
four tools, when large files are copied).
For reference, here are my number on Windows 7 x64 SDD -> FAT32 usb memory
stick:
(153 directories, 3077 files, 10,0 MB)
DirSyncPro 2:25 2:15
FreeFileSync 2:45 2:42
Explorer 2:03 1:59
TeraCopy 1:58 1:56
FreeFileSync (minimal) 2:00
Explanation for FreeFileSync is quite simple: FFS does more, basically three
extra steps.
1. copy files as temp file, then renames. By doing so a unique temp name is searched for, all in all results in a number of extra accesses. Reason is to prevent risk of data loss that occurs if sync process is stopped unexpectedly, leaving behind a corrupted file that has a more recent date than its source.
2. FFS copies all files times, not only modification, but also file creation time.
3. FFS solves DST +-1h issue automatically on FAT drives by storing extra information
A new performance test with steps 1, 2, 3 disabled gets FFS a total time of
2:00. So this explains the numbers for this special scenario.
-Zenju
namely "copying a lot of small files to a slow target medium with buffering
disabled. Consequently the times will be dominated by number of accesses per
file rather than transfer speed. (latter will be pretty much the same for all
four tools, when large files are copied).
For reference, here are my number on Windows 7 x64 SDD -> FAT32 usb memory
stick:
(153 directories, 3077 files, 10,0 MB)
DirSyncPro 2:25 2:15
FreeFileSync 2:45 2:42
Explorer 2:03 1:59
TeraCopy 1:58 1:56
FreeFileSync (minimal) 2:00
Explanation for FreeFileSync is quite simple: FFS does more, basically three
extra steps.
1. copy files as temp file, then renames. By doing so a unique temp name is searched for, all in all results in a number of extra accesses. Reason is to prevent risk of data loss that occurs if sync process is stopped unexpectedly, leaving behind a corrupted file that has a more recent date than its source.
2. FFS copies all files times, not only modification, but also file creation time.
3. FFS solves DST +-1h issue automatically on FAT drives by storing extra information
A new performance test with steps 1, 2, 3 disabled gets FFS a total time of
2:00. So this explains the numbers for this special scenario.
-Zenju
-
- Site Admin
- Posts: 7058
- Joined: 9 Dec 2007
Further analysis:
Testcase from above, using FFS v3.14:
-> 2:47
Applying an "optimistic strategy" I got rid of all file accesses except
I) rename temp file
II) copy file time
-> v3.15 beta: 2:42
Isolated performance penalty of I) is 36 sec (32%), penalty of II) is 16 sec
(14%).
New test candidate with features I) and II) disabled:
-> 1:52!
Furter test using simple stream copy instead of WinAPI file copy:
-> 1:47!
Result: Leaving out features I) and II) there is potential to even beat tools
like TeraCopy which claim fastest file copy speed. TBD: speed vs safety
switch?
Testcase from above, using FFS v3.14:
-> 2:47
Applying an "optimistic strategy" I got rid of all file accesses except
I) rename temp file
II) copy file time
-> v3.15 beta: 2:42
Isolated performance penalty of I) is 36 sec (32%), penalty of II) is 16 sec
(14%).
New test candidate with features I) and II) disabled:
-> 1:52!
Furter test using simple stream copy instead of WinAPI file copy:
-> 1:47!
Result: Leaving out features I) and II) there is potential to even beat tools
like TeraCopy which claim fastest file copy speed. TBD: speed vs safety
switch?
-
- Site Admin
- Posts: 7058
- Joined: 9 Dec 2007
Update:
Using an optimized stream file copy version instead of ::CopyFileEx
effectively gets rid of overhead II), time down to:
new time v3.15 beta: 2:28 This change will most likely make it into the next
version.
Left for optimizing: I) Transactional file copy:
None of the contestants provide this feature, that is they leave garbage
behind in case of network drops or power loss, but more critical this garbage
file has a more recent date than its sane counterpart. This effectively leads
to data loss if the file is synchronized with other backups.
Possible target performance, still 1:52, but with II) included. Trading speed
vs safety/reliability, this step will be kept included until further notice.
Using an optimized stream file copy version instead of ::CopyFileEx
effectively gets rid of overhead II), time down to:
new time v3.15 beta: 2:28 This change will most likely make it into the next
version.
Left for optimizing: I) Transactional file copy:
None of the contestants provide this feature, that is they leave garbage
behind in case of network drops or power loss, but more critical this garbage
file has a more recent date than its sane counterpart. This effectively leads
to data loss if the file is synchronized with other backups.
Possible target performance, still 1:52, but with II) included. Trading speed
vs safety/reliability, this step will be kept included until further notice.
- Posts: 2
- Joined: 8 Jul 2008
Thanks Zenju to put so much effort in speeding up the copy process, I'm really
impressed! And I like the idea of a safety versus speed switch cause in my
case safety is not an issue. I'm just copying OpenStreetMap tiles and if one
is missing or corrupted it doesn't matter. Speed is what I need :) The 5MB
example I showed you was just a tiny part of a huge amount of small files
(500MB-1GB) which I have to copy. So I like the switch option but if you
manage to speed up the process to 1:52 (your testfile) with all safety options
included then that would be really great. Nicest GUI and fastest performance
:)
Cheers!
impressed! And I like the idea of a safety versus speed switch cause in my
case safety is not an issue. I'm just copying OpenStreetMap tiles and if one
is missing or corrupted it doesn't matter. Speed is what I need :) The 5MB
example I showed you was just a tiny part of a huge amount of small files
(500MB-1GB) which I have to copy. So I like the switch option but if you
manage to speed up the process to 1:52 (your testfile) with all safety options
included then that would be really great. Nicest GUI and fastest performance
:)
Cheers!
-
- Site Admin
- Posts: 7058
- Joined: 9 Dec 2007
>speed up the process to 1:52 (your testfile) with all safety options included
Surprisingly it is possible to get disk accesses down to the bone by using
optimistic strategies, that is, get all the safety checking, but don't pay for
them (in terms of performance). While at it, I've implemented copying NTFS
extended attributes such as compression, encryption, sparse state and
alternate data streams for v3.15 without any negative performance impact on
the standard file copy scenario.
Now while I don't expect any significant performance improvement for buffered
file systems with latest optimizations, it's interesting how large the impact
of an innocuous looking rename operation can be on a FAT memory stick: 35%
perf decrease! However it seems that this step cannot be logically avoided
without giving up FFS's high consistency guarantees (transactional behavior is
inherent in every part of FFS's design and overall reliability is defined by
the weakest link). On the other hand it seems not fair having other apps brag
with their transfer speeds while silently stepping back on data integrity
promises ;) Maybe this alone justifies a switch.
Surprisingly it is possible to get disk accesses down to the bone by using
optimistic strategies, that is, get all the safety checking, but don't pay for
them (in terms of performance). While at it, I've implemented copying NTFS
extended attributes such as compression, encryption, sparse state and
alternate data streams for v3.15 without any negative performance impact on
the standard file copy scenario.
Now while I don't expect any significant performance improvement for buffered
file systems with latest optimizations, it's interesting how large the impact
of an innocuous looking rename operation can be on a FAT memory stick: 35%
perf decrease! However it seems that this step cannot be logically avoided
without giving up FFS's high consistency guarantees (transactional behavior is
inherent in every part of FFS's design and overall reliability is defined by
the weakest link). On the other hand it seems not fair having other apps brag
with their transfer speeds while silently stepping back on data integrity
promises ;) Maybe this alone justifies a switch.
-
- Site Admin
- Posts: 7058
- Joined: 9 Dec 2007
With v3.21 there's a new switch to turn off "transactional file copy", which
will favor maximum speed vs transactional file copy behavior:
(Note: the difference will be significant only for unbuffered(!) volumes like
FAT-formatted USB sticks!)
Here are the numbers:
New testcase: copy 2.000 files and folders, 173 MB from NTFS ssd, to FAT32 usb
stick
1. Transactional file copy enabled:
157 sec (two-run average)
2. Transactional file copy disabled:
132 sec (two-run average)
Now I don't have time to run another full testcase against all the other
tools, but I expect FFS to be among the fastest with transactional file copy
disabled. On to the next challenge.
will favor maximum speed vs transactional file copy behavior:
(Note: the difference will be significant only for unbuffered(!) volumes like
FAT-formatted USB sticks!)
Here are the numbers:
New testcase: copy 2.000 files and folders, 173 MB from NTFS ssd, to FAT32 usb
stick
1. Transactional file copy enabled:
157 sec (two-run average)
2. Transactional file copy disabled:
132 sec (two-run average)
Now I don't have time to run another full testcase against all the other
tools, but I expect FFS to be among the fastest with transactional file copy
disabled. On to the next challenge.
- Posts: 1
- Joined: 20 Jun 2016
NTFS compression is a good option if the receiving end has decent hardware and transfer rate. Any way to turn it off? Nothing makes a large sync grind to halt like compression does.
