Slow Transfer with many small files?

Posts: 15 · rednroll 21 Jul 2021, 03:52

Hi, I was just curious to know if my experience is to be expected.

I have a bunch of Midi (*.mid) files that I'm doing a folder transfer (mirror folder pair) from a SSD to a HDD. The entire content of the folder is only 168MB in size, but there are a total of 147,340 files in this folder with an average size of each file around 200 Bytes each. It seems to be taking forever to complete this transfer. Attached is a screen capture to give you an idea. FFS is showing an estimate of 6.5 hours to complete this transfer. Is this just a Windows thing and to be expected or is this a problem with FFS or something else?

: Processing.jpg (39.64 KiB) Viewed 3533 times

Posts: 4055 · xCSxXenon 21 Jul 2021, 14:49

9 items/s is pretty normal for a mechanical hard drive. If you did the same transfer via File Explorer, it would take the same time, most likely. In every previous case of slow transfers, it has always been something externally causing the issue. Try disabling your Antivirus as well. Most of them scan data during the transfer, especially Microsoft Security.

Posts: 1038 · therube 21 Jul 2021, 16:09

I'd say something is amiss.
43K small items, 18MB.
From SSD to NAS (over 1 GB network).
Win7, no A/V.

(My favorite midi was [I'm pretty sure], baris_manco_gulpembe.mid
https://www.youtube.com/watch?v=HMakfcor4y4).

Posts: 15 · rednroll 21 Jul 2021, 16:28

Thanks for the responses. I will try to disable anti-virus next time to see if that has any improvement. I don't believe it's the drive speed, it's a USB3.0 5400RPM, 5TB Seagate Portable Backup Plus drive, 25% full, which isn't fast by any means but when I transfer larger files of say 10GB in size to this drive, that transfer takes well under 5 minutes. This transfer of 168MB with many small files took well over 4 hours to complete. I believe it's similar if I had done the transfer strictly through Windows file explorer. Even after I completed the transfer, I went in Windows File explorer and deleted the original folder with all the files on the SSD and when I went to empty the recycle bin, it took Windows over 10 minutes just to complete emptying the recycle bin. Was really hoping there was someway to speed this transfer up....over 4 hours for 168MB seemed like I was transferring to a floppy disk, hoping turning off MS real-time antivirus helps. :D

Posts: 4055 · xCSxXenon 21 Jul 2021, 16:44

That's exactly what I would expect from a mechanical drive, especially a 5TB portable external drive, especially from Seagate.

Posts: 1038 · therube 27 Jul 2021, 16:55

25% full

That crossed my mind.

Other thought is (file) fragmentation, but with only 25% full, that too would seem unlikely.
Can't hurt to check file fragmentation.

Cluster size?
Maybe Seagate is using some funky cluster size that simply doesn't play well with small files?

As it is, & 150K isn't that many, each of your small files will occupy 1 cluster (with a lot of slack space), but again with only 25% used, again thinking that shouldn't be an issue.

5400 is "slow". Even so, that does not account for what you're seeing.

It is typical that transferring lots of small files is less efficient then single large files, but not to the extent that you are seeing.

Posts: 15 · rednroll 27 Jul 2021, 18:53

I ran a Chkdsk on the drive from Windows command prompt, results posted below. It's formatted as NTFS 4096 block size. It's now closer to 50% full, since I had transferred a bunch of other audio files to it after this transfer that seemed to take forever to complete. The 1st thing I usually do is reformat drives right after purchase. Don't trust the formats from the factory or the junk backup software/adware that comes on many drives, that's what I use FFS for. :D

Anways, here are the results from CHDSK. Nothing odd jumping out at me about this drive in these results.

Posts: 15 · rednroll 27 Jul 2021, 19:06

Crystal Disk Info

Posts: 4055 · xCSxXenon 27 Jul 2021, 19:26

100 items/s is about exactly what I expect a high density portable drive to handle.

Posts: 15 · rednroll 27 Jul 2021, 19:28

Crystal Disk Read/Write test results.

Anything look amiss?

I dunno, without having done further testing but for now chalking it up to the high quantity of small file sizes seems to be pretty taxing when copying to another drive in windows. I suppose the windows anti-virus being turned on as well could have some impact if it's scanning each file in the transfer process. However, I've now copied 2TB of data over to this drive at this point with anti virus enabled. None of the other close to 2TB of data took as long to transfer through windows file explorer as this job of transferring 168MB of many small files taking 4 hours to complete.

Posts: 15 · rednroll 27 Jul 2021, 19:31

Other things I considered. The folders these midi files are in, include an "@" sign in their name. Unsure if that causes any issues.

Posts: 15 · rednroll 27 Jul 2021, 19:36

These are some of the typical files contained in these folders. I observe they typically range in size around 50-200 Bytes.

Posts: 15 · rednroll 27 Jul 2021, 19:53

Here is the top layer folder info that took 4 hours to complete the transfer.

Posts: 15 · rednroll 27 Jul 2021, 20:10

100 items/s is about exactly what I expect a high density portable drive to handle. xCSxXenon, 27 Jul 2021, 19:26

Great to know.

However, sanity math check would seem to indicate differently. :D

147,340 Files/100 Files per Second= 1,473.4 seconds
1,473.4 seconds/60 Seconds per Minute= 24.5 minutes.

At least I think my math is correct in that calculation. :D

In your previous post you mentioned 9 items/Sec was pretty typical for mechanical HDD. Am I misunderstanding?

147, 340 Files/9 Files per Second= 16,371.1 Seconds
16,371.1/60 seconds per minute= 272.85 minutes
272.35 minutes/60 Minutes per Hour= 4.55 Hours

Now that 2nd calculation is more on par with what I actually experienced.

Posts: 4055 · xCSxXenon 27 Jul 2021, 20:33

Well I was basing it off the other factors like transfer speed and estimated file sizes.
The first picture was running 10 times slower

Posts: 15 · rednroll 27 Jul 2021, 20:37

Well I was basing it off the other factors like transfer speed and estimated file sizes.
The first picture was running 10 times slower xCSxXenon, 27 Jul 2021, 20:33

I guess that's where I'm confused. A 24 minute total transfer I wouldn't even have questioned but it's that 10 times slower of over 4 hours it actually took to complete the transfer makes me question if that this is to be expected or not?

Posts: 1038 · therube 27 Jul 2021, 20:45

Size on disk: is interesting - 0 bytes?
.

: Midi with 0 byte size on disk.png (8.7 KiB) Viewed 3445 times

Posts: 15 · rednroll 27 Jul 2021, 21:10

Size on disk: is interesting - 0 bytes?
.
Midi with 0 byte size on disk.png therube, 27 Jul 2021, 20:45

That's a great observation. Got me wondering if these files got corrupted somewhere along the way. So I just opened up that Midi file I posted above in the Reaper DAW software and it had no gripes opening it, displayed a single midi note inside of it, and that note triggered a sound of a virtual synth during playback.

Everything seems ok there but that is odd seeing the size on disk displaying as 0 bytes.

Posts: 4055 · xCSxXenon 27 Jul 2021, 21:16

um
I see size on disk as 4,096

Posts: 15 · rednroll 27 Jul 2021, 21:25

Here's that particular file if anyone would like to try further inspection.
https://drive.google.com/file/d/1DOZCua5XsFL28_kKGeuFO56jCyPmCbWl/view?usp=sharing

Posts: 15 · rednroll 27 Jul 2021, 21:30

um
I see size on disk as 4,096 xCSxXenon, 27 Jul 2021, 21:16

That's a similar Midi file that Therube posted for comparison in regards to file properties which is from his PC.

He was pointing out what I had posted before that.

See here:

Posts: 4055 · xCSxXenon 27 Jul 2021, 21:35

Ah I see!
That is odd. What is your external formatted as?

Posts: 15 · rednroll 27 Jul 2021, 21:56

NTFS, 4096 bytes block size.

Posts: 1038 · therube 28 Jul 2021, 15:20

Ah.
How can a file's size on disk be 0 bytes when there's data in it?
(Without reading fully, on my end, Win7, your file shows 'Size on disk:' as 4096 bytes.)

Posts: 15 · rednroll 28 Jul 2021, 21:54

Good find. Differences between Windows 7 on your side and Windows 10 on my PC of the handling of the NTFS format and small files.

Well we got that problem solved. I still need to figure out why this transfer takes so long. It will likely take me some running of multiple test comparisons doing comparison transfers using Windows file explorer and FFS with and without anti-virus turned on to see if there any significant differences in transfer times. At least to be able to figure out if there's something odd going on here when using FFS.

The 500GB SSD I originally had these files on had gotten full due to other music instrument library audio files I added, where I had to transfer this folder along with those audio files to a larger capacity disc since the program I'm using requires the library audio files and their associated midi files be placed under the same folder on a drive. I don't anticipate having to do this transfer again, so not a burning issue was just curious if that was expected since copying 170MB worth of files and it taking over 4 hours to complete seemed a bit excessive, no matter how slow the disk they're being copied to. I now can copy them back to the SSD they were originally on and see how long that transfer takes.

Posts: 1038 · therube 29 Jul 2021, 15:55

(Obviously you could zip [7-zip, is probably better] up the directory, & transfer that single zip file...)

Posts: 1038 · therube 29 Jul 2021, 15:58

And I was thinking about this more, & thought... maybe... Size:0 & Win10, maybe there's something with that.

But that didn't pan out.

I took 100K files - mostly 0-bytes (18K of the 100K were small, >0 bytes), threw them onto a Win10 system, & then copied the same across a 100 MB LAN to a different Win10 system, in (relatively) short order.

: FFS 100K files Win10 size 0.png (13.24 KiB) Viewed 3368 times

(The Win10 systems, I suppose, are running Windows Defender.)

(100k sample files, 130 KB, if anyone is inclined.)