Feature request: Unbuffered copy

[filamento]

Sir,

I'm having lots of problems when I try to sync huge files (outlook pst files)
from my local hard drive to a network shared folder.

The huge files consume the RAM of the server and make it go extremely slowly.

Would it be possible to add an option to make unbuffered copies?

Here's the problem explained by Microsoft employees:

http://blogs.technet.com/b/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx

Could you please implement this feature?

Thanks a million!!

[Zenju]

Hi,

the article sounds reasonable, however I know a lot of Win32 functions have
problems with unbuffered file access and explicitly prohibit such flags in
their documentation. Having said that, CopyFileEx offers
"COPY_FILE_NO_BUFFERING". I don't know its implementation but I wouldn't be
surprised if using this flag entails various regression when copying from and
to "non-standard" file containers (network, samba, seldom-used protocols,
etc.). Therefore I'm reluctant making this the new default. Anyway let's test
it first, here's the updated version:

[404, Invalid URL: http://www.2shared.com/file/G_Sfz8gg/FreeFileSync_v53_beta_setup.html]

[filamento]

Thank you very much!

Unfortunately, I cannot test the beta right now, because the computers where
I'm having problems use Windows XP, and the server uses Windows 2003 R2 :(

Microsoft says it here:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363852%28v=vs.85%29.aspx
COPY_FILE_NO_BUFFERING
The copy operation is performed using unbuffered I/O, bypassing system I/O
cache resources. Recommended for very large file transfers.
Windows Server 2003 and Windows XP: This value is not supported.

I guess I'll have to make a script and use Microsoft's Exchange Eseutil.exe
tool to transfer those files separately :(

God, I really hate those huge PST files!!

[Zenju]

From my tests so far the flag seems to perform well: 1. no perf difference for
FAT USB stick -> NTFS SSD and converse, 2. 20% faster for NTFS -> NTFS SSD. If
I don't find any more problems during testing or researching the net, I'll
want to give this setting a try.
You're right, XP and 2003 are not supported. Unfortunately the alternative (=
rolling a custom copy function as suggested by the article) doesn't look very
promising since there is tremendous support and maintenance effort included in
CopyFileEx by Microsoft supporting all sorts of obscure devices.

[filamento]

I understand your concern. It seems that using unbuffered copies under Windows
XP risks creating compatibility problems.

I'll simply exclude those PST files from FreeFileSync and will transfer them
using Eseutil.exe in a different batch file. That's probably going to be my
final solution.

By the way, if the "COPY_FILE_NO_BUFFERING" flag does work well for Windows 7
and doesn't create problems, what will you do? Will it be a configuration
option, or will it be the default copy method?

Kind regards.

[Zenju]

If there aren't any compatibility problems (= MS did their job) then I'll set
it as the new default beginning with Vista. Currently 2/3 of FFS's users are
already on Windows 7, so it will be a big plus to have this setting for most
users! It's somewhat ironic that this fix doesn't work for you since you got
this issue to my attention. Thanks a lot for providing me with detailed
technical information in first place, rather than just saying FFS was slow
during copy :>

[Zenju]

First collateral damage:
viewtopic.php?t=772

[Zenju]

Second collateral damage:
[404, Invalid URL: https://sourceforge.net/tracker/?func=detail&atid=1093080&aid=3529683&group_id=234430]

[filamento]

Oops, that's a pity :(

Would it be possible to make the "NO BUFFERING" flag an optional (non default)
setting? Perhaps hidden in the registry (in order to not clutter the interface
too much) ??