Waaaay Too Slow 5.12 doesn't fix

Get help for specific problems
Posts: 16
Joined: 19 Jul 2012

remmian

W764 & FFS 5.12

I used to love this program but some months back after an update I guess, it started slowing to a crawl. Took 2 min to transfer 1.8 megabytes of data! Another thread says 5.11 fixed the problem, but I updated to 5.12 and it's still the same thing. :(

UPDATE: For people who don't have time to read this thread, it was never a FFS problem I had, but a USB stick problem. A PNY stick with exFAT that has extremely poopy performance issues. FFS works great and I still love this program and it was premature of me to blame FFS.
User avatar
Site Admin
Posts: 7280
Joined: 9 Dec 2007

Zenju

You really need to give me something to "work with"...
Posts: 141
Joined: 10 Sep 2009

srjones67

PC issue? I transfer 11.2 GB from my laptop to my USB 3.0 hard drive in 2 minutes 56 seconds.
Posts: 141
Joined: 10 Sep 2009

srjones67

I am using W764 & FFS 5.12
Posts: 16
Joined: 19 Jul 2012

remmian

You really need to give me something to "work with"...Zenju
Sorry... I wrote a long detailed post 5 times and every time I hit POST REPLY I got thrown back to the login page and it lost everything I wrote. Finally saw the problem was NoScript even though it was set to allow SF... had to allow globally. Anyhow...

Am mirroring 1 small folder from C:\ (less than 1MB), then the entire D & F partitions (one nearly empty, the other a few GBs of text and doc files) to a USB stick as a nightly backup to augment monthly full backups. So any changed data files I create during the day will be in one of the FFS-mirrored directories, and usually only amounts to a few MB or less.

FFS used to sail through this process lightening fast. And I used the same version of FFS for a long time, then finally manually updated it a few months back. Seems ever since I switched from the original version I was using to the newer ones, I have had this problem. HOWEVER I also changed hardware a few months back from the internal WD320GB to WD 1.5TB, and the external USB memory stick was a Lexar 4GB, now a PNY 32GB. (Drive is regularly maintained with CCleaner and Auslogics defrag, plus real-time & boot AV scans with AVAST. It is also a fully patched system, which might be a bad thing since Windows releases some funky patches too.)

Updating to 5.12 got rid of some of the problem as it is analyzing faster again, whereas that process was hanging before ("not responding" errors and "FFS is locking this folder" hangups). However, when I click to do the synch it takes way too long for the couple of MB it's transferring.

I noticed you told someone else in another thread to use NTFS so I am currently reformatting the USB stick to NTFS and will try again. If I run into the same problem, I would like to help, but will need a little instruction for things like "do a memory dump" -- I know what that is but not how to do one.

I really love your program, so thank you so much for writing and maintaining it...

Will post back after reformatting and sync is done with results... of course the first sync will take awhile b/c it will be a few GB, but after that I will test just a few MB which is my norm.

Thanks again...
Posts: 16
Joined: 19 Jul 2012

remmian

@srjones67

If it IS a PC issue, I don't know what but am willing to find it. As I stated above, my system is fully patched, defragged, cleaned and no AVs. I'm pretty OC about it. I also ran Western Digital's diagnostic tool on my HD two days ago -- the extended test that takes several hours -- and it came back with zero errors. Additionally I have run sfc /scannow not long ago and it came back with no problems found. Finally, no other sftw is acting up and I am not experiencing any other problems...
Posts: 141
Joined: 10 Sep 2009

srjones67

Try a different memory stick or external hard drive
Posts: 16
Joined: 19 Jul 2012

remmian

Yes, great idea and was about to do just that but got interrupted and now have to get to an appt so will be back later. Thanks for helping. Am looking forward to resolving this and will report back in a few hours.
User avatar
Site Admin
Posts: 7280
Joined: 9 Dec 2007

Zenju

> wrote a long detailed post 5 times

The Allura platform here on Sourceforge is brand new, so there are likely a lot of smaller things that can be improved. If you think there's some nuissance without a good reason, let the sourceforge crew know on their official bug trackers.

> FFS used to sail through this process lightening fast.

The most obvious question is: Are you sure this slow speed is related to FFS or do you get the same speed when using other tools, even if Windows Explorer?

There are a few performance-critical settings in FFS that you should check:

1. what is your "deletion handling": "versioning" can be very slow if the target is not the same volume. Also recycler has issues in some cases (as you already seem to be aware of) when it is full and the drive is slow. The fastest setting is "permanent" which doesn't preserve old files.

2. global settings: "fail-safe file copy" will write to temporary files first. The performance penality is generally insignificant, except for slow and unbuffered drives (most usb sticks) when syncing large numbers of small files. In this case turning it off might be worth a test.

3. The locking file is active by default in GlobalSettings.xml: "LockDirectoriesDuringSync". It's updated only every 5 seconds, but this might still have some impact on overall performance on slow devices due to buffer invalidation on each write.

> use NTFS

FAT(FAT32) are legacy file systems, so using a newer one like NTFS is usually an improvement. Just be aware of the safety-concerns: FAT-formatted USB sticks are generally configured to be unbuffered, so that you can just pull the stick out after copy. If you do the same with a buffered NTFS volume without prior unmounting you may lose or corrupt your data!

If you can confirm that the speed problem is only in FFS and fiddling with the options above does not help, you can check what the last FFS version was that showed faster copy speeds.
Posts: 16
Joined: 19 Jul 2012

remmian

ARGH..... I just wrote another extremely long, detailed reply and again when I hit POST REPLY it got lost... this time with a 403 error.

So I am giving the short version: it was the memory stick. Something is funky with it. FFS works fine with the previous memory stick, and with an external drive.

So FFS is vindicated and I apologize for blaming it for the woes. :) I am so happy it's working fast again!!!

Thanks for your help, Zenju and for writing such a great program. And thanks to srjones67 for his help too. I appreciated it.
Posts: 16
Joined: 19 Jul 2012

remmian

Never mind... still having problems... even using the Lexar mem stick am now getting the "Wait while we lock this folder" msg...

When I did the previous tests I was just coping over one partition with 2.68GB... the F drive. Now I added my other stuff... two small folders from the Desktop (less than 1MB ea) and that caused this particular problem it seems, which WAS one of the original problems.
User avatar
Site Admin
Posts: 7280
Joined: 9 Dec 2007

Zenju

> REPLY it got lost... this time with a 403 error.

Welcome to my world... For these reasons I always "Copy + C" *every* text before posting.

> apologize for blaming it for the woes

Also here: welcome to my world :D

> Wait while we lock this folder

This message comes up when FFS finds a sync.ffs_lock file in one of the to be scanned folders and cannot quickly detect that it is obsolete (e.g. if the lock file is from the same user, same machine, but the associated process is no longer existing, FFS can deduce that the lock is abandoned and can just delete the file). Otherwise, there is indeed some other FFS process currently running holding this lock, or the lock file is corrupted so that the short-cut handling above doesn't grip. In this case FFS has no other choice than to monitor the file for activity. If there is no activity after 30 seconds, only then FFS may safely consider the lock abandoned and delete it. Depending on what you are doing you can now draw your own conclusion with respect to the behavior outlined above.

> caused this particular problem it seems

Binary tests like these (can reproduce/cannot reproduce) are the best way to track down the exact circumstances that lead to the unexpected slowdown.
Posts: 16
Joined: 19 Jul 2012

remmian

Hello Zenju....

Btw when I said "never mind" in my last post, I didn't mean that your program wasn't great... I meant never mind in relation to my problems being resolved. I still think your program is great and most problems are by and large user error or due to other things which is the case with just about any software ...but M$ where maybe it's more 50/50. :-P

Anywho...

From your description above I determined the last scenario was the culprit, as it was counting down 30 secs at each folder before proceeding. I enabled hidden system files and deleted all FFS-created lock files, then re-ran the program with the Lexar stick and it ran flawlessly transferring the few MB that is my normal daily.

To add some details left off of my earlier post that went to Puragtory instead of making it here, (back when I was transferring 2.68GB and you gave me a list of FFS settings to check)... The Lexar transferred 2.6 GB in 20 min. I feel like that's really slow but it's a stick limitation as FFS ran the same exact job to an external drive in 3 or 4 minutes.

My deletion handling has always been set to Recycle Bin but I changed it to Permanent. Then, aanting to run the test you suggested, I tried disabling fail-safe copy and had FFS run the same exact job (2.6 GB) and it finished in 18min, vs 20min with fail-safe enabled. So it made a small difference for large jobs, but again, the stick is just really slow. There is a budget Lexar 3.0 stick that is cheaper b/c it's only 4x faster than 2.0, but it seems to achieve that rate on a 2.0 port, according to one buyer who placed his own benchmarks on an Amazon review, so I might try that.

As for the PNY stick... when I tried to reformat to NTFS the system hung, then it showed up fine in My Computer, but not trusting it, I used chkdsk and it came back with an error that the drive was formatted RAW! So used EASEUS to delete its partition/recreate/reformat then did a surface scan test which came back with 0 problems. But it's still screwed up somehow b/c it still causes read/write problems when using. Food for the trashcan.

Thanks again Zenju... (and I, too, often copy my posts to the clipboard before hitting Send...a practice that will be instituted without fail at SF from now on!) ;)
Posts: 16
Joined: 19 Jul 2012

remmian

PS Can I somehow edit the title of this thread to add "RESOLVED" to it? Or don't they do that here? Clicking EDIT by the OP did not allow access to the title.
User avatar
Site Admin
Posts: 7280
Joined: 9 Dec 2007

Zenju

> I didn't mean that your program wasn't great

You got it. On this forum there cannot be such things as complaints of what I define to be flawless! :D

> deleted all FFS-created lock

At this point I would ask myself two things: 1. how come these files were not deleted by the FFS process that created them. 2. Why did the "quick detection of abandoned locks" handling not kick in. I don't think there is a bug in FFS in this regard, but there may be problems on the drive that is holding these abandoned lock files.

> Can I somehow edit the title

It looks like this is not (yet?) possible on the new Sourceforge platform. I would send them a feature request - but they have bigger issues to solve for me right now...
Posts: 16
Joined: 19 Jul 2012

remmian

Hello Zunju...

As to the questions regarding the previous FFS mirror screw-up on the PNY memory stick that is not an FFS bug (no, not our flawless FFS!! ;) ) ...but due to the stick, I learned why!

It was bugging me why this brand new PNY 32GB stick should be giving me grief and I was ready to trash it, but wanted to get to the bottom of it... why when I formatted it into NTFS would I get problems with chkdsk saying it was RAW? And why would My Computer show the drive letter but no icon for the drive itself with continuing read/write problems when it should be working flawlessly?

...So I went to Amazon to edit my original review about it but while there read another review (first) where the poster who was using XP was lamenting that no one mentioned the stick is formatted in exFAT, not supported by XP without a MS patch.

AH-HA! Now we're getting somewhere!! I had never heard of exFAT (was released with VISTA SP1 for memory stick formatting, not hard disks, to overcome the limitations of FAT32 regarding sheer numbers of files it can handle and large files in excess of 4GB)... and am sure that's at the basis of all the problems. I am on W7 (supports exFAT) but I reformatted it to NTFS and would bet you can't just reformat exFAT to NTFS without jumping through some hoop.

So I used Win Admin Tools to reformat the PNY to exFAT and it formatted the stick fine. My Computer can also now see it and it's behaving much better. However it still has "jerky" read/write behavior.

For example I wanted to see how it would behave with FFS now, so I plugged it in and did a full FFS mirror of my nightly backup folders. This time I did not get any "not responding" errors and FFS finished the job eventually. HOWEVER (and again I know this is the drive and not FFS) the program continually halted throughout, intermittently, to where I expected to see a "not responding" any sec, but then it'd start up again. And it didn't just 'freeze' on large files where you think "well this one's just taking longer" ... it froze on small files too, so it wasn't the load. It's the drive.

Meanwhile, an FFS comparison of using the 4GB FAT32 Lexar stick, and the 32GB exFAT PNY:

Lexar 4GB FAT32: FFS transferred 2.6GB in 20 minutes.
PNY 32GB exFAT : FFS transferred 3.2GB in 29 minutes.

Bottom line is, the 3.2GB transfer should have taken ~24 min to match the Lexar time. But it took 5 minutes longer than that because of all that halting. And again, no multitasking or background services were going on.

Anyway... FWIW!
User avatar
Site Admin
Posts: 7280
Joined: 9 Dec 2007

Zenju

The halting is actually where the usb stick writes the data. While FFS is responsive and *seems* to copy data it merely is filling the USB stick's buffers. From a usability perspective this is not optimal, but the behavior is harmless. The solution for the not-responding phases is to run the copy tasks asynchronously. But since this is only every noticeable for slow volumes it may not be worth the trouble to implement.
Posts: 16
Joined: 19 Jul 2012

remmian

Thanks, but it doesn't behave that way with the Lexar (FAT32). There is definitely something up with the PNY stick. Even when transferring a 480MB file from hard disk to PNY (exFAT), the TeraCopy progress bar does the same thing... it stops and starts in fits. If it's taking so freaking long to write, why not to the Lexar too? Both FFS and straight copy services work smoothly for Lexar with progress going along at an even pace. No fits and starts... it might 'pause' during long transfers as a larger file takes longer to complete, but the DTR remains steady.

480 MB file to PNY took 2:29
same file to Lexar took 1:58

Also, even mounting the PNY stick Windows burps and farts and hangs like it's not sure who's joining the party or if they should be let in. I think this is a defective stick, even though all tests come back as it being fine, it isn't behaving that way.

But IAC it is most definitely not a FFS problem, so this is not the venue for me to complain about it further. :) Just wanted to mention it here in case other ppl have FFS problems with slow transfers to USB sticks ... they should check to see if their stick is exFAT... I'm not saying every exFAT formatted stick will be a problem, but of those with problems I would wager the vast majority will be exFAT sticks... and I base that on the fact that it's a MS proprietary format, and fairly new, and we know MS's track record on... well, everything. :)

Thanks again for your support and time Zenju.
User avatar
Site Admin
Posts: 7280
Joined: 9 Dec 2007

Zenju

Did you try to disable the lock file? As mentioned above writing this file, if only every few seconds, can invalidate write buffers, so this can have a real impact on performance.

> exFAT

Of all the legacy issues that FAT/FAT32 had, the only problem exFAT seems to solve is large file sizes. It still does not properly support file IDs and still does not save file times as UTC. From a sync software's perspective it is a total disappointment given how long MS needed to finally come up with this format.
Posts: 16
Joined: 19 Jul 2012

remmian

I did try that previously (disabling the lock file) but might have been with the Lexar before I knew it was the PNY at fault. I will try it with the PNY but it is currently being reformatted to FAT32 using the long method rather than Quick Format, to see if it helps the 'jerky' slow performance. (I decided to chuck exFAT so I can use the stick with Linux, for the little I will use it until I throw it out!) But I Googled "PNY 32GB USB memory stick slow performance" and suffice it to say I will not be buying another PNY. Methinks this is unfortunately normal for this brand, regardless of format choice.

I am not surprised that MS's exFAT is a disappointment. In the late 80s I thought MS was the greatest thing ever (3.x days), but over the years the shine really wore off that bloom. :)

Will report back after I run the lock test once the PNY is formatted, but it's not going to be done for a couple hours, doesn't look like... using the slow format method is my last hail mary pass for this stick, barring something I haven't thought of yet.

"Meanwhile, FFS sure runs flawlessly!" :-D
Posts: 141
Joined: 10 Sep 2009

srjones67

Bottom line end of subject, not all USB sticks are created equal. A 32gb USB 2.0 is only that speeds not guaranteed. I've been there done that.
Posts: 16
Joined: 19 Jul 2012

remmian

@srjones I realize speeds aren't guaranteed but it was more than just being slow (at the beginning of this thread). Now Zenju is just trying to help me shave some time off, which I appreciate. Sorry if its been irritating.

@Zenju I tried the 3.2 GB routine again with locked files and fail-safe writing disabled... did not make a difference. In fact took a few minutes longer (32min vs 29min for same 3.2 GB data - within the normal varied range I guess), but at least the stick is stable now since formatting it the sloooow way to FAT32. That is, Win is recognizing it right off the bat with no weird hanging, burping, deciding if it wants to see it or not.

MORE IMPORTANTLY I started this thread because it was taking FFS 2 MINUTES to transfer 1.8 MB ... what I *normally* transfer each night... and now those small updates are fast again... even with "fail-safe copy" and "copy locked files" enabled, and even on this lame stick. So thank you for following through with me. Even if it was the formatting <?> at least I got to learn about all these FFS settings. :) I very much appreciated your time and efforts on my behalf!