Incredibly slow syncing 2 million small files

Get help for specific problems
Posts: 2
Joined: 7 Jan 2013

camiloeid

Hello,

I'm syncing 2 million small files from a Windows 2008 server to a USB external drive with the Update method (FFS 5.10). The destination folder was empty, this is the first run and it took around 17 minutes to generate the file list. The file copy process is taking 4 days now and is still going.

Files are images of around 60KB plus thumbnails.

When the sync started, the transfer rate was 1MB/s and slowly degraded to 150 KB/s.

Any ideas ?

Thank you
Posts: 141
Joined: 10 Sep 2009

srjones67

Try V5.11. I think it resolves that issue.
User avatar
Site Admin
Posts: 7312
Joined: 9 Dec 2007

Zenju

USB storage devices, often FAT formatted, have very poor speed on random access, so this performance is probably "normal".
Posts: 2
Joined: 7 Jan 2013

camiloeid

Thanks for answering.

The USB drive is a 2TB LaCie formatted as NTFS. The server has 12GB RAM and 2 Xeon 6 Cores Processors.

The sync speed has worsen even more, its around 15 KB/s now. The FFS process is using 2GB Ram, is that normal ? The "Copy NTFS permissions" setting is enabled.

We are going to abort the sync, install FFS 5.11 and try syncing to a folder on the same server, just to rule out any problem with the USB drive. If that works fine then the next step is to do a standard file copy to the USB drive to see how long it takes, it's 88GB in total.

Best regards,
User avatar
Site Admin
Posts: 7312
Joined: 9 Dec 2007

Zenju

srjones67 is right, it's probably:
viewtopic.php?t=1097