I just finished a network drives upgrades and Chkdsk maintenance to make sure everything is working at maximum performance as I had to do a complete re-install of Windows 7 64 on my backup machine and noticed that I'm getting greatly fluctuating Sync speeds on my 1GB network. All hardware is functioning normally (complete D-Link 1 GB network) on 2.9Ghz Dual-cores and up. Doing manual copies via explorer is yielding 90Mbps+ copy speeds. Network buffers both transmit and receive are optimized from defaults and using Jumbo frame rates (4088 or 4k) which has always been the best setting for my LAN and worked very well.
On large files such as ISO's and large video's, archives, etc. I was able to tweak my NIC buffering to achieve 80-90Mbps which is pretty good, but I noticed that when it came to a large # of smaller files (.PDFs, images, .JPG, >BMP, etc) the transfer speed drops to a snails pace of 6-10Mbps extending the estimated time to complete by many hours. Not Good... At the moment I'm only seeing 10-12Mbps on 5 GB .MP4's and FFS is saying 9 hours to complete what normally requires only just over an hour under normal circumstances.
I remember back around 10.0 I was getting consistent transfer speeds of 80Mbps+ which was acceptable and normal in all other file transfers across my LAN, both during large and small files.
What happened? Why is the speed only 1/8th of what it used to be because this isn't working anymore. What changed?
Thanks!
FFS10.19
- Posts: 4
- Joined: 30 Jan 2020
- Posts: 4
- Joined: 30 Jan 2020
Imagine it takes you 1 second to go from point A to point B carrying up to 20Kg. If you move a 5Kg rock from point A to point B, it will take you 1 second. If you move a grain of sand it will still take you 1 second. The amount of stuff you transport in 1 second from point A to point B is in Kg per second. When it's a 5 Kg rock, it's 5Kgps, if it's a grain of sand it would be maybe 0.0001 Kgps. Your max bandwidth is still 20Kgps.
Now if you use a bag that can contain up to 5Kg of stuff and you fill that bag on point A before you transport it to point B, the only delay is the time it takes you to fill that bag. In any case you will transport it at 5Kgps. That bag would be a buffer.
In your case, it could be that FFS does not use a buffer, or if it does, the buffer is not optimally tuned.
Now if you use a bag that can contain up to 5Kg of stuff and you fill that bag on point A before you transport it to point B, the only delay is the time it takes you to fill that bag. In any case you will transport it at 5Kgps. That bag would be a buffer.
In your case, it could be that FFS does not use a buffer, or if it does, the buffer is not optimally tuned.
- Posts: 4
- Joined: 30 Jan 2020
I'm pickin' up what you're puttin' down.. :)
Thanks,
Thanks,
- Posts: 4
- Joined: 30 Jan 2020
Follow-Up:
This is a REAL bummer... It appears that I and other have outgrown FFS. :( Due to it's limited ability to sync up two folder that already have identical folders/files there is no viable, acceptable way of facilitating a synced folder or file.
Network maxed out performance wise. Thanks for your additional illumination on that suject. It prompted me to finish and further my understand of how things work that I hadn't considered previously. Now "I know what I know" and know it's at it's peak performance. Got sustained, solid transfers across the LAN of 89.5MB/sec. That works for me in spades.
Because my Docs folder is over 410GB with over 157,000 files & folders using either Windows COPY or FFS is time prohibitive. :( FFS gave me an estimated 16 days to sync up the two over the LAN even @ 90MB/sec because of the filesize issues involving small files no matter what settings I used in NIC properties. Under any circumstances that is not acceptable.
So I experimented and tried creating an ISO, tranferring it over the LAN and the entire process took only 3-4 hours roughly. That was acceptable considering the shear size of the job. It sure beat 16 days!!! Also, that didn't work. UltroISO created the ISO, but it was broken when I checked TWICE. Transfer rate was great, so there is a lesson there for future FFS development. Nothing gets it, not Win Explorer FFS... It's just excruciatingly too slow. When I hit the graphics and PDF's the transfer rates dropped to .7Mbps/sec. The grass grows faster than that. :)
So, I tried creating a single RAR archive using the STORE flag (no compression. That worked GREAT! Structure was intact and correct and even meta of the files were mirrored (create date, modified date, etc.). So now I'm feeling so hope... No dice. After I extracted to the destination, checked and everything looked perfect, deleted the old FFS db files in both dirs it was time to run the job and see if it would simply compare the files in the two folder. It did something weird, instead of comparing and creating new FFS db files (157,000+ in source folder) it combined the two folders and indexed over 314,000 files (BOTH FOLDER'S CONTENTS) which seems strange to me when it should be comparing.
So unless FFS development strategies can up it's game a bit and ADD a function for 1st time comparisons (because we all have lives and can't leave our systems locked on running one APP & a resource intensive app at that (target system slowed to a sluggish, choppy craw where it takes minutes just to position your mouse on it to try and run something. * Windows Explorer did not exhibit that behavior when browsing or execution apps when the LAN transfer was at full capacity receiving one enormous file) folks like myself suffering this insane amount of time doing the 1st sync will have to look elsewhere for remedy.
I can't see why it would be so difficult to add a "custom" 1st time scan option for both folders 1st to see what is identical and setting the FFS db files based on the results correct;y. Then run the job and do the updates folder/file updates. Otherwise add some logic that streams the folder/file contents as one file to address the LAN packet issue on small files. There has to be a way to address this really poor slow file manipulation process. It's slow folder to folder on the same drive and even slower on the LAN (which is normal. "My" drives transfer at 120-165 MB/sec, my LAN transfers at 90MB/sec. Both respectable rates, but they both slow down on single small files under a few MBs. Maybe a 1st time automated option to create the single huge file for transfer at high speeds then call Windows explorer to transfer the file to the destination, then extract it to the destination, then do the compare to build the FFS db files proper. That IS the fastest, acceptable way of working with large groups of files in the 100's of GB's. My process took only 4 hours start to finish to create and move 421GB (157,000 files & folders) to get ready to start FFS...
Can you achieve this what seems pretty straight forward capability??? or should should I leave FFS to the small occasional light user on thier PC because it doesn't support anything resembling a power user's needs on a home network sadly in it's current state and I sent a contribution in for what I felt was fair for your efforts and I don't do that often because most just aren't worth it... I think FFS is worth supporting. Have I outgrown it???
This is a REAL bummer... It appears that I and other have outgrown FFS. :( Due to it's limited ability to sync up two folder that already have identical folders/files there is no viable, acceptable way of facilitating a synced folder or file.
Network maxed out performance wise. Thanks for your additional illumination on that suject. It prompted me to finish and further my understand of how things work that I hadn't considered previously. Now "I know what I know" and know it's at it's peak performance. Got sustained, solid transfers across the LAN of 89.5MB/sec. That works for me in spades.
Because my Docs folder is over 410GB with over 157,000 files & folders using either Windows COPY or FFS is time prohibitive. :( FFS gave me an estimated 16 days to sync up the two over the LAN even @ 90MB/sec because of the filesize issues involving small files no matter what settings I used in NIC properties. Under any circumstances that is not acceptable.
So I experimented and tried creating an ISO, tranferring it over the LAN and the entire process took only 3-4 hours roughly. That was acceptable considering the shear size of the job. It sure beat 16 days!!! Also, that didn't work. UltroISO created the ISO, but it was broken when I checked TWICE. Transfer rate was great, so there is a lesson there for future FFS development. Nothing gets it, not Win Explorer FFS... It's just excruciatingly too slow. When I hit the graphics and PDF's the transfer rates dropped to .7Mbps/sec. The grass grows faster than that. :)
So, I tried creating a single RAR archive using the STORE flag (no compression. That worked GREAT! Structure was intact and correct and even meta of the files were mirrored (create date, modified date, etc.). So now I'm feeling so hope... No dice. After I extracted to the destination, checked and everything looked perfect, deleted the old FFS db files in both dirs it was time to run the job and see if it would simply compare the files in the two folder. It did something weird, instead of comparing and creating new FFS db files (157,000+ in source folder) it combined the two folders and indexed over 314,000 files (BOTH FOLDER'S CONTENTS) which seems strange to me when it should be comparing.
So unless FFS development strategies can up it's game a bit and ADD a function for 1st time comparisons (because we all have lives and can't leave our systems locked on running one APP & a resource intensive app at that (target system slowed to a sluggish, choppy craw where it takes minutes just to position your mouse on it to try and run something. * Windows Explorer did not exhibit that behavior when browsing or execution apps when the LAN transfer was at full capacity receiving one enormous file) folks like myself suffering this insane amount of time doing the 1st sync will have to look elsewhere for remedy.
I can't see why it would be so difficult to add a "custom" 1st time scan option for both folders 1st to see what is identical and setting the FFS db files based on the results correct;y. Then run the job and do the updates folder/file updates. Otherwise add some logic that streams the folder/file contents as one file to address the LAN packet issue on small files. There has to be a way to address this really poor slow file manipulation process. It's slow folder to folder on the same drive and even slower on the LAN (which is normal. "My" drives transfer at 120-165 MB/sec, my LAN transfers at 90MB/sec. Both respectable rates, but they both slow down on single small files under a few MBs. Maybe a 1st time automated option to create the single huge file for transfer at high speeds then call Windows explorer to transfer the file to the destination, then extract it to the destination, then do the compare to build the FFS db files proper. That IS the fastest, acceptable way of working with large groups of files in the 100's of GB's. My process took only 4 hours start to finish to create and move 421GB (157,000 files & folders) to get ready to start FFS...
Can you achieve this what seems pretty straight forward capability??? or should should I leave FFS to the small occasional light user on thier PC because it doesn't support anything resembling a power user's needs on a home network sadly in it's current state and I sent a contribution in for what I felt was fair for your efforts and I don't do that often because most just aren't worth it... I think FFS is worth supporting. Have I outgrown it???
-
- Posts: 3603
- Joined: 11 Jun 2019
If it is chugging your destination that much, maybe that is your bottleneck? Also, compressing/compacting, transferring, then decompressing would maybe help if you have the hardware for it, but a dual core 1.5GHz processor isn't going to benefit. It sounds like you have outgrown pretty much every method in existence. You need NVME drives and a 10gb network
- Posts: 4
- Joined: 30 Jan 2020
Hi xCSx,
It's "chugging my destination"? You lost me friend...
Last evening I discovered during testing that it is advantageous to simple compress directly to the destination (I use WinRAR and used the "store" option. It actually took no more time then compressing locally! That cut the processing time in half. Just 2 hrs to and another 2 hours to decompress on the target into it's destination folder.
Sadly FFS chokes on the comparison portion of it's coding logic.
Two identical folders, one on source and one on target. FFS when run will not compare, it creates db's of both folders. There a flaw in the logic. 10 files in folder A on source A compared to the same exact files is folder B on target B do NOT equal 20 files total synced... That what I get when I "pre-load" the folders. Fixing that error in logic would go a long way to solve this issue of moving large amounts of small files in compressed format to speed up the task. Compressing across the LAN overcomes Windows and FFS limitations when transferring large amounts of small files efficiently at the greatest transfer speeds and if FFS could properly compare folder pairs and properly update db's on both ends we'd be right as rain... Maybe in the future FFS could internally address a compression/decompression scheme of it's own, but calling WinRAR on both ends would be the smart way to go because it is already a reliable, robust, proven solution...
It's "chugging my destination"? You lost me friend...
Last evening I discovered during testing that it is advantageous to simple compress directly to the destination (I use WinRAR and used the "store" option. It actually took no more time then compressing locally! That cut the processing time in half. Just 2 hrs to and another 2 hours to decompress on the target into it's destination folder.
Sadly FFS chokes on the comparison portion of it's coding logic.
Two identical folders, one on source and one on target. FFS when run will not compare, it creates db's of both folders. There a flaw in the logic. 10 files in folder A on source A compared to the same exact files is folder B on target B do NOT equal 20 files total synced... That what I get when I "pre-load" the folders. Fixing that error in logic would go a long way to solve this issue of moving large amounts of small files in compressed format to speed up the task. Compressing across the LAN overcomes Windows and FFS limitations when transferring large amounts of small files efficiently at the greatest transfer speeds and if FFS could properly compare folder pairs and properly update db's on both ends we'd be right as rain... Maybe in the future FFS could internally address a compression/decompression scheme of it's own, but calling WinRAR on both ends would be the smart way to go because it is already a reliable, robust, proven solution...
