Backup to USB is slow
- Posts: 3
- Joined: 10 Oct 2022
I'm a new user so maybe I'm missing something. I'm making a simple backup of my home directory in Linux Mint to a USB stick. The total is 2gb and FFS is telling me it's transferring @ 50kb/sec and will take about 6 hours. PLEASE tell me there is a way to solve this. It's a 3.0 USB that I've used to copy files etc at average 200mb/sec this has to be an issue with FFS.
-
- Posts: 4056
- Joined: 11 Jun 2019
Source type; HDD, USB, SSD?
USB port specs?
Without unplugging the destination, are you able to copy files to it at full speed manually? The transfer speed may be low, but what about items/s?
USB port specs?
Without unplugging the destination, are you able to copy files to it at full speed manually? The transfer speed may be low, but what about items/s?
- Posts: 3
- Joined: 10 Oct 2022
It's a new laptop with SSD. I've copied files from the SSD to this particular USB several times with really fast speeds. I'm sure there's a lot of files that FFS has to process honestly I didn't look at that only that the total size was 2gb. I can't imagine this is normal. I'm just a home user what do people do when they're backing up multiple tb.
-
- Posts: 4056
- Joined: 11 Jun 2019
Well, multiple TB of data is usually large files, so it transfers faster. Also, people with that much data typically use waaaay better storage media. USB drives don't really have a standard and as such can be really slow. As a techie, I am even disappointed in my Samsung flash drives write speeds / cache. Big data people use portable SSDs or some are using V90 SD cards which have guaranteed write speeds. I am personally looking at getting a SanDisk Extreme PRO because is is a "solid state" flash drive, which blows any USB flash drive out of the water
- Posts: 3
- Joined: 10 Oct 2022
Fair points. I guess I'm just behind the times. Looks like I'll have to make an investment in an external SSD. Thanks for taking the time to enlighten me.
- Posts: 1037
- Joined: 8 May 2006
The way I look at things...
One would expect one copy program to be relatively as efficient as another.
So if you
cp /2gb dir/* /dev/fd0
or you
FFS -> /2gb dir/ -> /dev/fd0/2gb dir/
you would expect to see at least similar throughput.
Did you actually let the FFS copy copy?
Or at least let it run long enough to have a good feel that it would in fact take much longer then a cp (or whatever other) method?
---
I'm in need of a new USB flash drive (64-128 GB range, I'm thinking).
I only have USB 2.0 on my end.
And in any case, speed is not particularly an issue for me.
That said, if you read the reviews, & it mainly appears when users are copy single large files, a few GB, that throughput - sucks. That it will be going along, then simply stops, for a time, before picking back up.
And they'll mention that even though it is "3.0", they don't get 3.0 speeds... (And over time, manufactures change the "make-up" of a device - leaving the name the same, so what was, at some point, may not be, anymore.)
(Granted, I'm talking "el-cheapo" drives here, but still. Also, what may be "slow" or even extremely slow to one person, [speed] may not be an issue to another - to an extent.)
---
So device, ports, (OS, if it happens to be Win10 or 11, as they seem to be a crapshoot) & who knows whatnot could be figuring in.
---
So I would test, running cp (or whatever OS utility you're using), & see what kind of throughput (& total time) you're getting against the same data set with FFS.
If FFS is substantially slower, then the question is, why?
One would expect one copy program to be relatively as efficient as another.
So if you
cp /2gb dir/* /dev/fd0
or you
FFS -> /2gb dir/ -> /dev/fd0/2gb dir/
you would expect to see at least similar throughput.
Did you actually let the FFS copy copy?
Or at least let it run long enough to have a good feel that it would in fact take much longer then a cp (or whatever other) method?
---
I'm in need of a new USB flash drive (64-128 GB range, I'm thinking).
I only have USB 2.0 on my end.
And in any case, speed is not particularly an issue for me.
That said, if you read the reviews, & it mainly appears when users are copy single large files, a few GB, that throughput - sucks. That it will be going along, then simply stops, for a time, before picking back up.
And they'll mention that even though it is "3.0", they don't get 3.0 speeds... (And over time, manufactures change the "make-up" of a device - leaving the name the same, so what was, at some point, may not be, anymore.)
(Granted, I'm talking "el-cheapo" drives here, but still. Also, what may be "slow" or even extremely slow to one person, [speed] may not be an issue to another - to an extent.)
---
So device, ports, (OS, if it happens to be Win10 or 11, as they seem to be a crapshoot) & who knows whatnot could be figuring in.
---
So I would test, running cp (or whatever OS utility you're using), & see what kind of throughput (& total time) you're getting against the same data set with FFS.
If FFS is substantially slower, then the question is, why?
-
- Posts: 4056
- Joined: 11 Jun 2019
The solid state flash drive (SSFD) I ordered will be here today. I will run a test between my Samsung USB, which is "top of the line" and the new SSFD. Most people online just run the default 1GB CrystalDiskInfo, but that never fills up the cache! I'll run that test, because the 4k RAND tests are valuable still, but I also want to see the speed over time when writing 100s of GBs alsoFair points. I guess I'm just behind the times. Looks like I'll have to make an investment in an external SSD. Thanks for taking the time to enlighten me. katclix, 10 Oct 2022, 21:46
- Posts: 1037
- Joined: 8 May 2006
TEAM 128GB C175 USB 3.2 Gen 1 Flash Drive
(with this drive):
exFAT vs NTFS makes a HUGE difference in how a copy operation to the drive may impact overall system responsiveness.
exFAT is certainly the way to go.
NTFS certainly looks to have a huge negative effect in almost all respects.
(Not knowing exFAT, first thing I did to the drive was to reformat it NTFS ;-).)
With NTFS, file deletions were abysmal.
(I'm talking about deleting a directory tree kind of thing.)
With NTFS copy operations to the drive were fraught with large, long expanses of "nothingness" (period of time, where seemingly nothing was happening).
(I'm not certain of actual "speed", throughput, of exFAT vs NTFS - yet.)
\LIB, 25.2 GB, 13,629 objects (12685 files, 944 directories)
NTFS (17th) vs exFAT (19th) [& with 128 KB block size]
Lets call it 4+ minutes faster copy time with exFAT.
(More "accurate" today, & set FFS to exit on completion.)
(with this drive):
exFAT vs NTFS makes a HUGE difference in how a copy operation to the drive may impact overall system responsiveness.
exFAT is certainly the way to go.
NTFS certainly looks to have a huge negative effect in almost all respects.
(Not knowing exFAT, first thing I did to the drive was to reformat it NTFS ;-).)
With NTFS, file deletions were abysmal.
(I'm talking about deleting a directory tree kind of thing.)
With NTFS copy operations to the drive were fraught with large, long expanses of "nothingness" (period of time, where seemingly nothing was happening).
(I'm not certain of actual "speed", throughput, of exFAT vs NTFS - yet.)
\LIB, 25.2 GB, 13,629 objects (12685 files, 944 directories)
NTFS (17th) vs exFAT (19th) [& with 128 KB block size]
Lets call it 4+ minutes faster copy time with exFAT.
(More "accurate" today, & set FFS to exit on completion.)
Microsoft Windows [Version 6.1.7601]
C:\DEV\BACKUP\FreeFileSync>timethis freefilesync.exe -dirpair C:\LIB K:\FFS
TimeThis : Command Line : freefilesync.exe -dirpair C:\LIB K:\FFS
TimeThis : Start Time : Mon Oct 17 13:22:34 2022
TimeThis : End Time : Mon Oct 17 14:00:18 2022
TimeThis : Elapsed Time : 00:37:43.479
C:\DEV\BACKUP\FreeFileSync>timethis freefilesync.exe -dirpair C:\LIB K:\FFS
TimeThis : Command Line : freefilesync.exe -dirpair C:\LIB K:\FFS
TimeThis : Start Time : Wed Oct 19 14:20:47 2022
TimeThis : End Time : Wed Oct 19 14:53:50 2022
TimeThis : Elapsed Time : 00:33:03.392
C:\DEV\BACKUP\FreeFileSync>
-
- Posts: 4056
- Joined: 11 Jun 2019
I have been using the new flash drive for a week. I am thrilled with it! It will write at 300MB/s +, which writing is my absolute highest priority. When I ran a sync of 150GB+, the average speed over the whole transfer was 135MB/s. There was a mix of large files and small files. When transferring only larger files, it is consistently over 300MB/s. It is about 3x the size of the Samsung Bar I was using, but I will never ever go back daily-driving a regular flash drive.
I will attach some results of CDM tests using the default 1GB size and 16GB size, as well as using a hub vs direct connection. I did not test the Samsung Bar at 16GB test sizes because it was so much slower that I didn't care to let it finish. Both are formatted exFAT.
The highlights are:
- No significant difference in read/write speed when using the SanDisk on a hub versus direct connection. The Samsung Bar drops 100MB/s in the Sequential tests when using the hub.
- The write speed of the SanDisk is 436% faster than the Samsung Bar for the 1GB test.
- The write speed of the SanDisk in the 16GB test, after the cache fills, is 334% faster than the Samsung Bar in the 1GB test, which is under the cache size for my 256GB version.
- The random 4k tests @ 32 tasks in the queue is handled three times as fast when reading from the SanDisk.
- Random 4k tests @ 1 task in the queue was basically a wash except when the Samsung was in the hub, I don't know what that is all about.
- The max size, 64GB, on the SanDisk produced identical results at the 16GB test, within margin of error.
In real use, the SanDisk is lightyears ahead of everything else I have used and will pay for itself if you use it for work.
I will attach some results of CDM tests using the default 1GB size and 16GB size, as well as using a hub vs direct connection. I did not test the Samsung Bar at 16GB test sizes because it was so much slower that I didn't care to let it finish. Both are formatted exFAT.
- Overall.png (584.96 KiB) Viewed 1987 times
- No significant difference in read/write speed when using the SanDisk on a hub versus direct connection. The Samsung Bar drops 100MB/s in the Sequential tests when using the hub.
- The write speed of the SanDisk is 436% faster than the Samsung Bar for the 1GB test.
- The write speed of the SanDisk in the 16GB test, after the cache fills, is 334% faster than the Samsung Bar in the 1GB test, which is under the cache size for my 256GB version.
- The random 4k tests @ 32 tasks in the queue is handled three times as fast when reading from the SanDisk.
- Random 4k tests @ 1 task in the queue was basically a wash except when the Samsung was in the hub, I don't know what that is all about.
- The max size, 64GB, on the SanDisk produced identical results at the 16GB test, within margin of error.
In real use, the SanDisk is lightyears ahead of everything else I have used and will pay for itself if you use it for work.
- Posts: 1037
- Joined: 8 May 2006
(
So anything I might have observed at the time was flawed due to utter inconsistencies with the drive.
My replacement drive, a Kingston, I formatted NTFS [& I've forgotten now, but there was a reason I went that way over exFAT].
)
You have to take my whole post above with a grain of salt - because the drive turned out to be utter crap.exFAT vs NTFS makes a HUGE difference in how a copy operation to the drive may impact overall system responsiveness
So anything I might have observed at the time was flawed due to utter inconsistencies with the drive.
My replacement drive, a Kingston, I formatted NTFS [& I've forgotten now, but there was a reason I went that way over exFAT].
)
