Feature request: Unbuffered copy

[filamento]

Sir,

I'm having lots of problems when I try to sync huge files (outlook pst files)
from my local hard drive to a network shared folder.

The huge files consume the RAM of the server and make it go extremely slowly.

Would it be possible to add an option to make unbuffered copies?

Here's the problem explained by Microsoft employees:

http://blogs.technet.com/b/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx

Could you please implement this feature?

Thanks a million!!

[Zenju]

Hi,

the article sounds reasonable, however I know a lot of Win32 functions have
problems with unbuffered file access and explicitly prohibit such flags in
their documentation. Having said that, CopyFileEx offers
"COPY_FILE_NO_BUFFERING". I don't know its implementation but I wouldn't be
surprised if using this flag entails various regression when copying from and
to "non-standard" file containers (network, samba, seldom-used protocols,
etc.). Therefore I'm reluctant making this the new default. Anyway let's test
it first, here's the updated version:

[404, Invalid URL: http://www.2shared.com/file/G_Sfz8gg/FreeFileSync_v53_beta_setup.html]

[filamento]

Thank you very much!

Unfortunately, I cannot test the beta right now, because the computers where
I'm having problems use Windows XP, and the server uses Windows 2003 R2 :(

Microsoft says it here:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363852%28v=vs.85%29.aspx
COPY_FILE_NO_BUFFERING
The copy operation is performed using unbuffered I/O, bypassing system I/O
cache resources. Recommended for very large file transfers.
Windows Server 2003 and Windows XP: This value is not supported.

I guess I'll have to make a script and use Microsoft's Exchange Eseutil.exe
tool to transfer those files separately :(

God, I really hate those huge PST files!!

[Zenju]

From my tests so far the flag seems to perform well: 1. no perf difference for
FAT USB stick -> NTFS SSD and converse, 2. 20% faster for NTFS -> NTFS SSD. If
I don't find any more problems during testing or researching the net, I'll
want to give this setting a try.
You're right, XP and 2003 are not supported. Unfortunately the alternative (=
rolling a custom copy function as suggested by the article) doesn't look very
promising since there is tremendous support and maintenance effort included in
CopyFileEx by Microsoft supporting all sorts of obscure devices.

[filamento]

I understand your concern. It seems that using unbuffered copies under Windows
XP risks creating compatibility problems.

I'll simply exclude those PST files from FreeFileSync and will transfer them
using Eseutil.exe in a different batch file. That's probably going to be my
final solution.

By the way, if the "COPY_FILE_NO_BUFFERING" flag does work well for Windows 7
and doesn't create problems, what will you do? Will it be a configuration
option, or will it be the default copy method?

Kind regards.

[Zenju]

If there aren't any compatibility problems (= MS did their job) then I'll set
it as the new default beginning with Vista. Currently 2/3 of FFS's users are
already on Windows 7, so it will be a big plus to have this setting for most
users! It's somewhat ironic that this fix doesn't work for you since you got
this issue to my attention. Thanks a lot for providing me with detailed
technical information in first place, rather than just saying FFS was slow
during copy :>

[Zenju]

First collateral damage:
viewtopic.php?t=772

[Zenju]

Second collateral damage:
[404, Invalid URL: https://sourceforge.net/tracker/?func=detail&atid=1093080&aid=3529683&group_id=234430]

[filamento]

Oops, that's a pity :(

Would it be possible to make the "NO BUFFERING" flag an optional (non default)
setting? Perhaps hidden in the registry (in order to not clutter the interface
too much) ??

[Zenju]

Sine the sourceforge link above is dead, but might be interesting, here's the content:

Bugs item #3529683, was opened at 2012-05-25 00:57
Message generated for change (Comment added) made by zhnmju123
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=1093080&aid=3529683&group_id=234430

Submitted By: Marcelo Castellani (mcfly-br)
Assigned to: Nobody/Anonymous (nobody)
Summary: Corrupting files on copy

Initial Comment:
Version 5.3 is having problems to copy files to a network drive. It's crazy, I know. It seems that the problem is in my network drive, but I assure you it's not only the same drive since last year, when I used Versions 4, 5.0 and 5.2, and it never happened before.
Just to make me extra-crazy, I have 2 computers using FreeFilesync and only 1 suffer this problem. The other makes copies normally. 🙂
I tried uninstall and install again, but it did not work.
Thank you in advance!

----------------------------------------------------------------------

Comment By: ZenJu (zhnmju123)
Date: 2012-05-25 03:46

Message:
What operating systems are you using on your network shares an your local
computer running FFS?

There has been a little tweak in v5.3 by switching to unbuffered copy.
Nothing that could cause any problems except for faulty network drivers -
which are a annoying reality.
So let's to a test: Here are two versions of the recent release. What are
your results?

http://www.2shared.com/file/zrKF_hQ_/FreeFileSync_54_setup_-_TEST1.html
http://www.2shared.com/file/GLXGuB60/FreeFileSync_54_setup_-_TEST2.html

----------------------------------------------------------------------

Comment By: Marcelo Castellani (mcfly-br)
Date: 2012-05-25 21:20

Message:
Hi!
First I want to apologize myself for being so naive. It's lack of practice.
My computers are running Windows 7. The one working is in test mode because
I have 1 unsigned driver for my webcam.

The host system for the share is a router Netgear that uses some sort of
Linux with samba, I guess.

The files are being corrupted by adding ZEROES to the end of file. The
copies work almost perfectly apart of the fact that if is a txt file I get
extra "spaces" at the end. In zipped files, I see no difference in use
because trailing zeroes (0) are cutted off anyway.

Libreoffice also seems to ignore the fact that a file has extra 0s after
"the end".

Saying the same thing again, the copy is a few bytes longer and the "new"
file is pretty much the same as original but appended with 0s.

I got 5.4-Test1 and this problem has GONE. I still have broken files just
to illustrate, but I believe they are unnecessary, as I already described
the problem. Anyway, if you believe that it can help anyway, I can send to
you, just tell me if you want.

I thank you again for helping me solving and, much more important, for
creating a such good program. 🙂
Sincerely yours,
McFly

----------------------------------------------------------------------

Comment By: ZenJu (zhnmju123)
Date: 2012-05-26 04:47

Message:
Related?

By: smith1212

Hi, when I use version 5.3 to sync my PC to my Dlink Sharecenter 320, files
with size over 100mb will not copy correctly to the NAS. The copied file on
the NAS has a size of 0kb after the process. With Version 5.2 it works flawless,
everything copies fine.

http://sourceforge.net/projects/freefilesync/forums/forum/847543/topic/5256646

----------------------------------------------------------------------

Comment By: Marcelo Castellani (mcfly-br)
Date: 2012-05-26 10:15

Message:
Hi!
I forgot to change things up here... Noob makes too much mistakes. Sorry!
The problem is different than that because in my case it's not related with
size.
The reason can be the same, any piece of software wrong that makes these 2
errors, but I got different sizes in a file with only 6 bytes. (not Kbyte,
bytes - a text file containing "TEST" plus 0D 0A of EOF). The result copy
was a file with about 50 bytes containing "TEST" 0D 0A ... and 00 00 00 00
a dozen times.
As I said, this problem does not exist anymore in version 5.4. 🙂
If you need, I can install again the old version and make some guided tests
to provide data for analysis, it's up to you. 🙂 Just let me know.
Thank you one more time.
Mc

----------------------------------------------------------------------

Comment By: ZenJu (zhnmju123)
Date: 2012-05-26 11:49

Message:
> I got 5.4-Test1 and this problem has GONE.
This was in fact the "old" way with buffered file copy (= without
COPY_FILE_NO_BUFFERING)
Both test versions are identical except for this single flag.

> As I said, this problem does not exist anymore in version 5.4. 🙂
This has been merely a test version and is not incorporated in the the
release build yet.

> The files are being corrupted by adding ZEROES to the end of file.
This is really bad. Even if this is the "best possible way of file
corruption", FFS cannot allow any leeway in this regard.

File copying without buffering requires extreme care and adherence to the
documented constraints from the developer. Therefore this is often gotten
wrong, just like in your case. I cannot believe MS screwed this one up, as
it's them who advertize this "COPY_FILE_NO_BUFFERING" flag. So it's
probably (an old?) version of SAMBA which is to blame.

No matter how unfortuante this is (and it's very unfortunate:
http://blog.mischel.com/2011/06/07/windows-file-copy-bug-revisited/)
I will have to remove this flag and "pessimize" the general usage, even if
it "works most of the time". But this is not enough.

> Anyway, if you believe that it can help anyway, I can send to
> you, just tell me if you want.
Yes there is one thing: It's a shot out of the blue, but I'd like to
analyze what's happening during a corrupted file copy. I don't think I will
be able to fix it, since copying is internal to CopyFileEx, but maybe I
"see" something.

Can you send me a Process Monitor log
http://technet.microsoft.com/de-de/sysinternals/bb896645.aspx
of v5.3? Start monitoring before syncing a single file and stop it
afterwards. Of couse I need the pathological case, i.e. a corruption
happened.

----------------------------------------------------------------------

Comment By: ZenJu (zhnmju123)
Date: 2012-05-27 10:02

Message:
Removed COPY_FILE_NO_BUFFERING from v5.4.

----------------------------------------------------------------------

> Comment By: ZenJu (zhnmju123)
Date: 2012-05-28 07:10

Message:
Analysis attached. All looks well, except that it doesn't work:

ReadFile D:\TEMP\FFS-Test\Test.txt SUCCESS Read Offset: 0, Length: 4, I/O
Flags: Non-cached, Priority: Normal
-> fine read 4 bytes

WriteFile \\readyshare\usb_storage\BKP\Marcelo\ffs-Test\Test.txt.ffs_tmp SUCCESS Write Offset:
0, Length: 512, I/O Flags: Non-cached, Priority: Normal

-> write 512 bytes! This is due to unbuffered copy. Buffered copy would
write 4 bytes at this place!

CopyFileEx: end of copy
SetEndOfFileInformationFile \\readyshare\usb_storage\BKP\Marcelo\ffs-Test\Test.txt.ffs_tmp SUCCESS Write
Metadata EndOfFile: 4

-> Here things probably go wrong. Although "SetEndOfFileInformationFile"
reports success this is obviously a lie.

Theory:
Samba just ignores "SetEndOfFileInformationFile" but simply sets file size
to what is given on last "WriteFile".