Slow Transfer with many small files?

Discuss new features and functions
Posts: 15
Joined: 21 Jul 2021

rednroll

Hi, I was just curious to know if my experience is to be expected.

I have a bunch of Midi (*.mid) files that I'm doing a folder transfer (mirror folder pair) from a SSD to a HDD. The entire content of the folder is only 168MB in size, but there are a total of 147,340 files in this folder with an average size of each file around 200 Bytes each. It seems to be taking forever to complete this transfer. Attached is a screen capture to give you an idea. FFS is showing an estimate of 6.5 hours to complete this transfer. Is this just a Windows thing and to be expected or is this a problem with FFS or something else?
Processing.jpg
Processing.jpg (39.64 KiB) Viewed 2801 times
User avatar
Posts: 3551
Joined: 11 Jun 2019

xCSxXenon

9 items/s is pretty normal for a mechanical hard drive. If you did the same transfer via File Explorer, it would take the same time, most likely. In every previous case of slow transfers, it has always been something externally causing the issue. Try disabling your Antivirus as well. Most of them scan data during the transfer, especially Microsoft Security.
Posts: 936
Joined: 8 May 2006

therube

I'd say something is amiss.
43K small items, 18MB.
From SSD to NAS (over 1 GB network).
Win7, no A/V.

(My favorite midi was [I'm pretty sure], baris_manco_gulpembe.mid
https://www.youtube.com/watch?v=HMakfcor4y4).
Attachments
FFS Copy 43K small items from SSD to NAS.png
FFS Copy 43K small items from SSD to NAS.png (102.24 KiB) Viewed 2787 times
Posts: 15
Joined: 21 Jul 2021

rednroll

Thanks for the responses. I will try to disable anti-virus next time to see if that has any improvement. I don't believe it's the drive speed, it's a USB3.0 5400RPM, 5TB Seagate Portable Backup Plus drive, 25% full, which isn't fast by any means but when I transfer larger files of say 10GB in size to this drive, that transfer takes well under 5 minutes. This transfer of 168MB with many small files took well over 4 hours to complete. I believe it's similar if I had done the transfer strictly through Windows file explorer. Even after I completed the transfer, I went in Windows File explorer and deleted the original folder with all the files on the SSD and when I went to empty the recycle bin, it took Windows over 10 minutes just to complete emptying the recycle bin. Was really hoping there was someway to speed this transfer up....over 4 hours for 168MB seemed like I was transferring to a floppy disk, hoping turning off MS real-time antivirus helps. :D
User avatar
Posts: 3551
Joined: 11 Jun 2019

xCSxXenon

That's exactly what I would expect from a mechanical drive, especially a 5TB portable external drive, especially from Seagate.
Posts: 936
Joined: 8 May 2006

therube

25% full
That crossed my mind.

Other thought is (file) fragmentation, but with only 25% full, that too would seem unlikely.
Can't hurt to check file fragmentation.

Cluster size?
Maybe Seagate is using some funky cluster size that simply doesn't play well with small files?

As it is, & 150K isn't that many, each of your small files will occupy 1 cluster (with a lot of slack space), but again with only 25% used, again thinking that shouldn't be an issue.

5400 is "slow". Even so, that does not account for what you're seeing.

It is typical that transferring lots of small files is less efficient then single large files, but not to the extent that you are seeing.
Posts: 15
Joined: 21 Jul 2021

rednroll

I ran a Chkdsk on the drive from Windows command prompt, results posted below. It's formatted as NTFS 4096 block size. It's now closer to 50% full, since I had transferred a bunch of other audio files to it after this transfer that seemed to take forever to complete. The 1st thing I usually do is reformat drives right after purchase. Don't trust the formats from the factory or the junk backup software/adware that comes on many drives, that's what I use FFS for. :D

Anways, here are the results from CHDSK. Nothing odd jumping out at me about this drive in these results.
Image
Posts: 15
Joined: 21 Jul 2021

rednroll

Crystal Disk Info
Image
User avatar
Posts: 3551
Joined: 11 Jun 2019

xCSxXenon

100 items/s is about exactly what I expect a high density portable drive to handle.
Posts: 15
Joined: 21 Jul 2021

rednroll

Crystal Disk Read/Write test results.
Image

Anything look amiss?

I dunno, without having done further testing but for now chalking it up to the high quantity of small file sizes seems to be pretty taxing when copying to another drive in windows. I suppose the windows anti-virus being turned on as well could have some impact if it's scanning each file in the transfer process. However, I've now copied 2TB of data over to this drive at this point with anti virus enabled. None of the other close to 2TB of data took as long to transfer through windows file explorer as this job of transferring 168MB of many small files taking 4 hours to complete.
Last edited by rednroll on 27 Jul 2021, 19:39, edited 1 time in total.
Posts: 15
Joined: 21 Jul 2021

rednroll

Other things I considered. The folders these midi files are in, include an "@" sign in their name. Unsure if that causes any issues.
Image
Posts: 15
Joined: 21 Jul 2021

rednroll

These are some of the typical files contained in these folders. I observe they typically range in size around 50-200 Bytes.
Image
Posts: 15
Joined: 21 Jul 2021

rednroll

Here is the top layer folder info that took 4 hours to complete the transfer.
Image
Posts: 15
Joined: 21 Jul 2021

rednroll

100 items/s is about exactly what I expect a high density portable drive to handle. xCSxXenon, 27 Jul 2021, 19:26
Great to know.

However, sanity math check would seem to indicate differently. :D

147,340 Files/100 Files per Second= 1,473.4 seconds
1,473.4 seconds/60 Seconds per Minute= 24.5 minutes.

At least I think my math is correct in that calculation. :D

In your previous post you mentioned 9 items/Sec was pretty typical for mechanical HDD. Am I misunderstanding?

147, 340 Files/9 Files per Second= 16,371.1 Seconds
16,371.1/60 seconds per minute= 272.85 minutes
272.35 minutes/60 Minutes per Hour= 4.55 Hours

Now that 2nd calculation is more on par with what I actually experienced.
Last edited by rednroll on 27 Jul 2021, 20:34, edited 1 time in total.
User avatar
Posts: 3551
Joined: 11 Jun 2019

xCSxXenon

Well I was basing it off the other factors like transfer speed and estimated file sizes.
The first picture was running 10 times slower
Posts: 15
Joined: 21 Jul 2021

rednroll

Well I was basing it off the other factors like transfer speed and estimated file sizes.
The first picture was running 10 times slower xCSxXenon, 27 Jul 2021, 20:33
I guess that's where I'm confused. A 24 minute total transfer I wouldn't even have questioned but it's that 10 times slower of over 4 hours it actually took to complete the transfer makes me question if that this is to be expected or not?
Posts: 936
Joined: 8 May 2006

therube

Size on disk: is interesting - 0 bytes?
.
Midi with 0 byte size on disk.png
Midi with 0 byte size on disk.png (8.7 KiB) Viewed 2713 times
Posts: 15
Joined: 21 Jul 2021

rednroll

Size on disk: is interesting - 0 bytes?
.
Midi with 0 byte size on disk.png therube, 27 Jul 2021, 20:45
That's a great observation. Got me wondering if these files got corrupted somewhere along the way. So I just opened up that Midi file I posted above in the Reaper DAW software and it had no gripes opening it, displayed a single midi note inside of it, and that note triggered a sound of a virtual synth during playback.

Everything seems ok there but that is odd seeing the size on disk displaying as 0 bytes.

Image
User avatar
Posts: 3551
Joined: 11 Jun 2019

xCSxXenon

um
I see size on disk as 4,096
Posts: 15
Joined: 21 Jul 2021

rednroll

Here's that particular file if anyone would like to try further inspection.
https://drive.google.com/file/d/1DOZCua5XsFL28_kKGeuFO56jCyPmCbWl/view?usp=sharing
Posts: 15
Joined: 21 Jul 2021

rednroll

um
I see size on disk as 4,096 xCSxXenon, 27 Jul 2021, 21:16
That's a similar Midi file that Therube posted for comparison in regards to file properties which is from his PC.

He was pointing out what I had posted before that.

See here:
Image
User avatar
Posts: 3551
Joined: 11 Jun 2019

xCSxXenon

Ah I see!
That is odd. What is your external formatted as?
Posts: 15
Joined: 21 Jul 2021

rednroll

NTFS, 4096 bytes block size.
Posts: 936
Joined: 8 May 2006

therube

Ah.
How can a file's size on disk be 0 bytes when there's data in it?
(Without reading fully, on my end, Win7, your file shows 'Size on disk:' as 4096 bytes.)
Posts: 15
Joined: 21 Jul 2021

rednroll

Good find. Differences between Windows 7 on your side and Windows 10 on my PC of the handling of the NTFS format and small files.

Well we got that problem solved. I still need to figure out why this transfer takes so long. It will likely take me some running of multiple test comparisons doing comparison transfers using Windows file explorer and FFS with and without anti-virus turned on to see if there any significant differences in transfer times. At least to be able to figure out if there's something odd going on here when using FFS.

The 500GB SSD I originally had these files on had gotten full due to other music instrument library audio files I added, where I had to transfer this folder along with those audio files to a larger capacity disc since the program I'm using requires the library audio files and their associated midi files be placed under the same folder on a drive. I don't anticipate having to do this transfer again, so not a burning issue was just curious if that was expected since copying 170MB worth of files and it taking over 4 hours to complete seemed a bit excessive, no matter how slow the disk they're being copied to. I now can copy them back to the SSD they were originally on and see how long that transfer takes.
Posts: 936
Joined: 8 May 2006

therube

(Obviously you could zip [7-zip, is probably better] up the directory, & transfer that single zip file...)
Posts: 936
Joined: 8 May 2006

therube

And I was thinking about this more, & thought... maybe... Size:0 & Win10, maybe there's something with that.

But that didn't pan out.

I took 100K files - mostly 0-bytes (18K of the 100K were small, >0 bytes), threw them onto a Win10 system, & then copied the same across a 100 MB LAN to a different Win10 system, in (relatively) short order.
FFS 100K files Win10 size 0.png
FFS 100K files Win10 size 0.png (13.24 KiB) Viewed 2636 times

(The Win10 systems, I suppose, are running Windows Defender.)

(100k sample files, 130 KB, if anyone is inclined.)
Attachments
100K_files.7z
(129.87 KiB) Downloaded 87 times