Extermly slow synchronization with my new flash drive

[tanin]

Hello,
I recently bought a usb 3.1 flash drive and created a new sync for it with FFS.
When I try to sync it is extremely, see picture.

The sync is of Mirror type, so before syncing for the first time I copied all the files in Windows Explorer and it went fine. I also tested the usb's write speed with an external program for an average of 34MB/s.

I previously had version 11.x and now upgraded to 12.1.
All my other sync's copy fine.

Any ideas?

[Plerry]

> ... so before syncing for the first time I copied all the files in Windows Explorer ...

As FFS copied 206MB and still intends to copy 12.9GB, it seems that, unless the changes to your left-side location are quite substantial since your copying in Windows Explorer, FFS intends to re-copy the entire contents, rather than just the modifications.
Are you sure the left- and right-side locations in FFS match with your Explorer copy?
Run a Compare in FFS and check if the proposed sync actions match your expectations.
And only then run your sync.

[tanin]

The first copy was through Windows Explorer and is about 300gb. FFS is copying avout 13gb, so it does only includes the changes (which I also checked manually).
I also see that the copying is slow by looking at the file being copied. For example a 2gb file takes 2-3 minutes.

> ... so before syncing for the first time I copied all the files in Windows Explorer ...

As FFS copied 206MB and still intends to copy 12.9GB, it seems that, unless the changes to your left-side location are quite substantial since your copying in Windows Explorer, FFS intends to re-copy the entire contents, rather than just the modifications.
Are you sure the left- and right-side locations in FFS match with your Explorer copy?
Run a Compare in FFS and check if the proposed sync actions match your expectations.
And only then run your sync.

[therube]

I recently bought a usb 3.1 flash drive

Which?

in Windows Explorer and it went fine. I also tested the usb's write speed with an external program for an average of 34MB/s

USB 3 with a USB 3 port, you would expect more, just saying.

All my other sync's copy fine.

Are they also to this same drive or elsewhere?

The theoretical transfer speed of USB 3.0 is 4.8 Gbit/s (600MBps) vs. 480 Mbit/s (60MBps) which
is a 10X improvement. Sustained transfer speeds (real life) for external hard drives are about
85MBps for USB 3.0 and about 22MBps for USB 2.0, so about a 5X improvement but still a
significant advancement in transfer speed.

https://www.verbatim.com/UserFiles/File/USB%203%20FAQs.pdf

I get 30MB/s on my Kingston USB 3.x flash drive - with USB 2 ports.

Prior, I had a Team 3.x flash drive, which was a drive from hell.
It simply did not work, from 1 second to another. Simply not reliable at all.

[tanin]

Which?

PNY Attache 4 512GB USB 3.1 Flash Drive

USB 3 with a USB 3 port, you would expect more, just saying.

USB 3 port indeed, how much more? From what I saw online, 30MB + is expected.

Are they also to this same drive or elsewhere?

Other drives.

Note that copying directly to the drive is fine.

[xCSxXenon]

According to PNY's website and specs: https://www.pny.com/en-eu/attache-4-3-1-usb-flash-drive?sku=FD512ATT431KK-EF
The max write speed is 20MB/s for the 512GB version.
Sounds like a cheap flash drive performing like a cheap flash drive. It probably has a cheap controller, cheap NAND, and bad thermal management. This is not to say PNY is bad or it's a bad flash drive, but it looks like your needs require something more advanced than what you have.

Get one of these SanDisks for the best flash drive on the market:
- amazon.com/dp/B01N7QDO7M -
Get this SSK for a cheaper option
- amazon.com/dp/B09HK6M8CC -

The SanDisk will write almost 400 MB/s forever and ever, it is my daily driver. The SSK is about 350 MB/s for about 80GB (for the 256GB version), then the cache fills and it goes down to 80 MB/S. It is a lot cheaper than the SanDisk for that reason. Read speeds on both are similar, ~350 - 400 MB/s

[xCSxXenon]

viewtopic.php?t=9681#p35687

[tanin]

Sounds like a cheap flash drive performing like a cheap flash drive.

I see.
But the drive performs enough for my needs (backup) under Windows Explorer. Copying a 2GB file directly doesn't take 2-3 minutes like it does when using FFS.

[xCSxXenon]

FFS probably transfers that same 2GB file just as fast, but your initial picture is not showing 2GB files, it is showing 156 files around 84MB each. Are you sure FFS is slower when you are testing the exact same file(s)?

[tanin]

FFS probably transfers that same 2GB file just as fast, but your initial picture is not showing 2GB files, it is showing 156 files around 84MB each. Are you sure FFS is slower when you are testing the exact same file(s)?

Pretty sure. When I took the screenshot I missed the part with the file name. The current file that was being copied was 2gb large and I saw that after 2+ minutes it was still copying the same file.

[xCSxXenon]

Test some more, 'pretty sure' isn't meaningful unfortunately

[tanin]

Test some more, 'pretty sure' isn't meaningful unfortunately

100% sure. "Pretty sure" was just a manner of speaking.

Just checked again just in case, a 300mb file took about a minute. Copying 103 files, ~2gb took 9:46 minutes.

[xCSxXenon]

Ok, that's 5 MB/s and ~3.33 MB/s respectively
I'm assuming that's with FFS?
How long do those same transfers take with File Explorer instead?
If there is still a significant difference, revert to your previous FFS version and see what happens

[tanin]

I upgraded to 12.1 from 11.x since I thought this will solve my problem.
Copying a 2.5gb file from Windows Explorer took 2:05 minutes.

[xCSxXenon]

Your tests are useless.
Stop changing what data you are transferring and timing

[tanin]

Your tests are useless.
Stop changing what data you are transferring and timing

What do you mean?
Copying the same file through FFS and Windows Explorer is not good comparison?

[xCSxXenon]

a 300mb file took about a minute.

Copying 103 files ~2gb took 9:46 minutes.

Copying a 2.5gb file from Windows Explorer took 2:05 minutes.

Three completely different datasets

[tanin]

a 300mb file took about a minute.

Copying 103 files ~2gb took 9:46 minutes.

Copying a 2.5gb file from Windows Explorer took 2:05 minutes.

Three completely different datasets

The first two are estimates, afterwards I started using a stopwatch.

But there's still a pretty huge gap between direct copying and using FFS with this USB.

[azure55]

Copying multiple small files are always much slower than copying 1 large file, even if they have the same total, due to various reasons:

https://superuser.com/questions/808279/why-does-copying-individual-files-take-so-much-longer-than-one-large-file

That's why they are not comparable, especially since 103 is a pretty substantial difference over 1. Instead, try copying the exact same file/s and see if there are differences.

[tanin]

There are.
Just tried syncing one 2gb file with FFS, it reached the 20% mark at about 4 minutes, so about 20m for a 2gb file.

Copying multiple small files are always much slower than copying 1 large file, even if they have the same total, due to various reasons:

https://superuser.com/questions/808279/why-does-copying-individual-files-take-so-much-longer-than-one-large-file

That's why they are not comparable, especially since 103 is a pretty substantial difference over 1. Instead, try copying the exact same file/s and see if there are differences.

[xCSxXenon]

There are.
Just tried syncing one 2gb file with FFS, it reached the 20% mark at about 4 minutes, so about 20m for a 2gb file.

That's NOT a test.
The only test you have performed is

Copying a 2.5gb file from Windows Explorer took 2:05 minutes.

because you actually timed the whole transfer. It averaged 20 MB/s. WHAT A SURPRISE

According to PNY's website and specs: https://www.pny.com/en-eu/attache-4-3-1-usb-flash-drive?sku=FD512ATT431KK-EF
The max write speed is 20MB/s for the 512GB version.

[xCSxXenon]

tanin: Hey guys, these results seem weird, what's up?
Everyone: Run consistent tests and measure the performance that way
tanin: no

[tanin]

tanin: Hey guys, these results seem weird, what's up?
Everyone: Run consistent tests and measure the performance that way
tanin: no

What is a consistent test?
I though that copying the same file through FFS and through Windows explorer, measuring times is enough of a test.

[therube]

I get 30MB/s on my Kingston USB 3.x flash drive

And it just bricked itself, sigh.
(Is there no more quality left in this world?)

The media is write protected.

It has put itself in some sort of Read-only mode, & frankly, there simply isn't a way out of it.

Seemingly it does this when it realizes there are issues with itself, no longer allowing Writes, but it does allow Reads - to give you a chance to backup data from the drive (before RMA'ing it).

(And that said, some of the files on the drive, I'm thinking, though not certain, some of the most recent files, are in fact, corrupt.

In the beginning, beginning being 2-1-2023, I did constantly verify that my files were correct [given my experiences with Team], but more recently, I slowed up on verification [by means other then FFS] - assuming that all was [still] OK. [And you know what happens when you assume...]

The drive was going along just fine... until, heh. MUCH better then the Team ever did.)

(My prior drive, Corsair, lasted over 5 years - never corrupted, until the day it died altogether.)

[therube]

Copy speeds.
I would expect similar copy speeds regardless of the means of the copy.

Some programs may attempt to verify a copy, or may retry reads reads if they are having an issue on the reading (of the source file). Other programs, "dumb programs" (& I'd have to think that Windows Explorer is dumb) & may not give a rats ass as to correct or not, may simply say, "done".

All my other sync's copy fine.

Does that mean that only with FFS's Mirror copy is where you are running into slowness?

As in, if instead of Mirror, you did an Update sync, the Update's copy speeds are on par with Windows Explorer?

[therube]

What I would do...

Take a handful of files, say big files, the bigger the better, take 10 10-GB files, 100 GB in total, & copy them to the drive - using Windows Explorer.

Then do the same with FFS - in Update mode (so a simply copy, not a Mirror or anything else) - into a different directory.

Time each.
Actually time them, don't rely upon any projected times that the programs may provide.
Maybe note CPU & RAM during each? Maybe note disk I/O during each?

Caching may come into play... depending.
A flash drive can be set up for "Quick Removal" or "Better Performance".
If "Better Performance" a dialog may "go away" sooner, but the actual copy is apt to take the same amount of time (only the interface to the user is hiding that fact).

So now you have 100 GB in one directory & 100 GB in another directory.

Then verify both sets of data against your source (by hash or similar means).
Time that also. (Likewise you again might note CPU & RAM.)

Now take another approach. Take a 100 GB directory tree of multi-sized files, big, small & everything in between, & do the same. Copy it into a 3rd. directory, with Windows Explorer. Copy it into a 4th. directory, with FFS. Again noting the time of each. Then verify the correctness of the copies...

At that point, is there substantial differences in the copy performance between W/E & FFS?


(Never trust your data solely to a single backup source. And it can never hurt to actually verify correctness of your backups ;-).)


Drives themselves may (or may not) have a cache. So (at least a) part of a copy operations (appears) to be running very speedily. But once that (on-drive) cache is filled, it "slows down". So a single file, or a few files, that do not fill that cache may appear to copy very fast, but a larger data set will give a better view of what is actually happening.

[Chrashla]

I have the same problem.
However, I use 2 internal hard drives.
I have the same data on both hard drives. Mostly photos and videos. Total data is 302GB
I then rearranged the pictures on hard drive 1 and also deleted some of them etc..
FFS takes 3-4 hours to sync with disk 2. This can not be.
It would actually be faster to delete the files on disk 2 and simply copy them from disk 1 using Explorer.

Does anyone have an idea?

Thanks in advance

[tanin]

All my other sync's copy fine.

Does that mean that only with FFS's Mirror copy is where you are running into slowness?

Only with FFS's sync I have for this (new flash) drive.
I have two other syncs, one between an external HD and my local HD and one between two external drives. Both with "correct" speeds.