Real-time sync severely slows down large sequential writes (RAR extraction)

[Bob4K]

On Windows 10 with FFS 14.4 , I’ve noticed that when FreeFileSync’s real-time sync is active, extracting a large RAR archive becomes extremely slow. Without FreeFileSync, extraction streams at full NVME disk speed (hundreds of MB/s). With FreeFileSync running, the same job drops to a crawl, with a 60s stall before it even starts.

I asked ChatGPT why, and it generated this answer:

What seems to be happening is this:

RAR extraction produces large sequential writes.

FreeFileSync is watching the filesystem (via USN journal / directory change notifications). Every write event wakes FreeFileSync up.

To decide if a sync is needed, FreeFileSync performs metadata queries (timestamps, sizes, directory scans).

These extra metadata I/O operations interleave with the sequential writes, turning the workload into many random seeks on HDDs (or at least extra overhead on SSDs).

The result is that the extractor’s throughput collapses, even though FreeFileSync itself isn’t actively syncing data.

By contrast, tools like RoboCopy avoid this interference: they don’t monitor writes in real time, they scan in batch and then stream data directly, so large sequential writes remain fast.

So, if the above is indeed correct:

Would it be possible (again ChatGPT suggestion) to add a “low-interference” or “deferred scan” mode for real-time sync? For example:

Batch notifications for a few seconds, then rescan once instead of reacting to every write immediately.

Only rescan when the system is idle.

Throttle directory queries during heavy I/O.

This would let FreeFileSync keep its real-time sync functionality without crippling other processes that are doing large sequential writes.

I know that if I use something like:

robocopy "B:\dvhs\arch\sta119" "d:\sta\sta122" /MIR /J /FFT /COPY:DAT /DCOPY:T /R:1 /W:2 /XD "@Recycle" ".@__thumb" "lost+found" "System Volume Information" "$RECYCLE.BIN" /ETA /TEE /LOG:D:\logs\nas_mirror.log

that works without slowing down WinRAR extraction.

Implementing robocoopy like options would solve the problem.

[xCSxXenon]

Get AI "help" out of here. Garbage in, garbage out.
RTS isn't going to affect extraction performance, it is just listening for change notifications from the OS. Your post isn't really clear either, FFS and RTS are different programs. You say with FFS's RTS, it's slow, but without FFS it is fast. "FFS's RTS" and "FFS" are, again, separate programs so your statement is unclear or irrelevant. If FFS is running, of course other processes using the disk are going to be impacted, they'll be sharing resources.

I can't stress this enough, ChatGPT gave you complete nonsense info.
We should just filter/ban any post containing any reference to AI tools.

[Bob4K]

Ok, forget ChatGPT.

The fact is: when FFS runs a copy from my NAS to a regular HDD at 200MBs, extracting a RAR archive from a GEN 5 SSD to another GEN 5 SSD, normally extremely fast, first stalls for 60s to the point you can't even kill it with Process Hacker if you try.

Then, after this delay, it finally starts extracting, but 5 times slower than usual.

My PC uses an AMD Ryzen Threadripper 9960X with an ASRock TRX50 WS motherboard under Windows 10 and 128GB of ECC RAM. Plenty of PCI lanes and horse power.

If instead of using FFS to copy from the NAS I use robocopy, the RAR extraction works without any slowness.

So, robocopy is doing things differently than FFS.

Whatever the explication, it would be nice to have an option with FFS that mimics what robocopy is doing to be able to run a 14h long job without hampering other jobs like RAR extraction.

There is certainly a good reason for FFS to work that way, it must be useful most of the time. But in that case, it's not.

[xCSxXenon]

Certainly odd behavior, as I'm able to use FFS without affecting other storage devices not being used by FFS.
Which LAN port are you using, 2.5 or 10gb?
I see that the SATA, 2.5g LAN, and at least one M.2 all have to be on the chipset, which has a Gen4x4 connection to the CPU. I can't imagine that this would be a bottleneck bandwidth-wise, but might worth looking into. Seems like you are running into a latency issue though, not throughput. BIOS/firmware up to date? What does disk usage look like when using FFS vs robocopy? And other system metrics, like CPU

[Bob4K]

I am using a 10G LAN. The NAS is very fast, reads up to 3GBs. When FFS starts, it shows 2.5GBs first, and quite slowly drops to the 200MBs HHD speed.

I apologize in advance, but I think that Chat is giving me here a valid explanation.
Using AI is useful, but only with a very large grain of salt. They often sprout garbage, but my experience is that weaning it out leads you to interesting findings.

So, here it is. It might steer you to look into the right place even if "as is" it's wrong:

1. How FreeFileSync copies

FFS is built on ReadFile/WriteFile loops at the application layer.

By default, it uses relatively large buffered I/O and memory-mapped files for comparison and transfer.

This hits the Windows system cache very aggressively. The NAS data streams through RAM before going to the HDD.

Result: your precious DDR5 RAM is clogged with NAS → HDD traffic, so other apps (like WinRAR) see memory starvation and I/O scheduler contention. Windows starts to stall anything else that wants big contiguous memory and I/O

2. How robocopy copies

Robocopy, especially with /J (unbuffered I/O) or /MT (multithreaded), bypasses most of the system file cache.

It streams data directly from network → disk, less RAM pollution.

Because it’s chunked and async, Windows I/O scheduler can interleave operations with other workloads (your NVMe → NVMe extraction).

That’s why WinRAR doesn’t choke when robocopy is running — the caches and scheduler aren’t monopolized.

3. Why no native “unbuffered” in FFS?

The developer explicitly decided against it because unbuffered I/O requires strict alignment of buffers to disk sector size. That makes the code much uglier and prone to misalignment errors on odd storage devices. FFS leans toward safety and simplicity at the cost of hogging the page cache.

That looks to me quite plausible. It explains the behavior.

My suggestion if this is actually not garbage, and although that when reading the last Chat statement it seems a lot of work, is an option to use unbuffered I/O.

I would certainly understand if it's a too major change to consider it.

[Bob4K]

Well, further testing shows that Chat is globally correct.

I tried making a TBs copy to an old 60MBs fast HDD. FFS first shows a 500MBs speed, that slowly drops to 60MBs. Certainly not enough to stress the system, and yet, when trying to extract a 90GB RAR archive between 2 unrelated GEN 5 SSDs that run 20 times faster than the HDD, it stalls first for a long time and then proceeds slowly.

I tried to use robocopy (with ARGUI, that gives it a rudimentary GUI) or FastCopy, no problem.

Now, FFS is a fantastic program, I could not live without it. Its GUI and features are just unrivaled. And it's free! (Although I use the donator edition).

But in this particular usage, a multi TB copy that lasts hours, it's not usable.

Adding an option to do unbuffered IO would really be a must.

[az99]

decided against it because unbuffered I/O requires strict alignment of buffers to disk sector size. That makes the code much uglier and prone to misalignment errors on odd storage devices.

Seems like there would be open-source code to do that. Many backup programs use rsync under the covers. I've read ffs doesn't (it's written to do what it does). I wonder if rsynch takes a long time to copy your file too? (If not, it seems like there would be some code from rsynch that could be used without a lot of development effort.). Or ssh? scp?

What happens if you chunk a 1tb file into 10,000 10meg files? Does it take as long to back all those up? Do the other tools outperform that too?

Maybe a pragmatic solution would be for ffs to use an external tool like rsync when a file is larger than some limit where the time-difference will be worthwhile?

[Bob4K]

Yes, there are plenty of file copy software, open-source or not. I already tried 2 that work perfectly.

But FFS is just the best overall, by so wide a margin that other stuff look pretty lame. It just seems a pity that it misses this option.

It's no problem using other software when I need it, but it would be nicer to be able to use FFS for everything rather than having to switch.

It's not a matter of performance here, FFS is always fast. One large file or many small ones makes no difference, FFS handles everything very well.

It's a matter of hampering other tasks which could be avoided with an unbuffered I/O option in the settings to be used in the rare cases it's needed.

[xCSxXenon]

hello guys

Is this all just AI bot spam???

3GB/s isn't POSSIBLE on a 10gb connection.
1) I have 10gb networking and RAID arrays capable of saturating it, unlike OP's HDD
2) 10gb is 1.25GB/s, half of what you claim to be getting onto the HDD at the beginning

Nothing in this thread makes sense. You never reported back on system metrics, either. You have 128GB of RAM, enough to handle this workload. Again, I have 64GB and have zero issues with fully saturating 10gb and accessing my 3xRAID0 close to its 20GB/s peak. Maybe your RAM is failing. There is something wrong with your environment if write-caching is affecting anything. Maybe try 7zip or NanaZip instead of WinRAR?

[Bob4K]

Here:

https://e.pcloud.link/publink/show?code=XZiisUZegD223pbDAfTPmivWp3BEzCwX4c7

That's a screen cap showing 2.19GB speed when I captured it.

I was hasty about 10G, this my general lan speed. The NAS uses a dedicated 40G adapter:

https://e.pcloud.link/publink/show?code=XZbrsUZg7sqcqkblnQVbyo9uW8E2yXd8QO7

You can see above 2 ConnectX-4 adapters, one for each of my 2 NASs.

I don't know what more to tell you. ChatGPT was clear about the reasons it happens.

And doing what Chat told me, using FastCopy in Sync/Mirror + Non-OS Cache mode does not trigger the problem while FFS always do.

I made at least half a dozen tries to compare with FastCopy or Robocopy with the /J option.

Maybe that would not happen with something else than WinRAR, but I like it and don't want to use something else.

This forum is about making FFS better. I'm pointing out a small way to maybe add something to an already great program. What's wrong about asking? I'm not expecting anything, it's a free program and I'm glad to use it as it is if it's too much of a problem. No need to get so defensive about it.

[Bob4K]

Here is why having a lot of memory does not help:

FFS uses buffered I/O. Its writes land in the Windows file cache first (dirty pages). Your HDD is slow to flush those pages, so the Lazy Writer builds a backlog.

Windows throttles globally when dirty pages pile up. The Memory/Cache manager doesn’t throttle “per-disk,” it throttles the whole box to keep dirty pages under control. Result: WinRAR’s writes to your Gen-5 SSD get stalled even though they’re to a different, very fast device. That’s why you see a pause, then a crawl.

“128 GB RAM not used” is a mirage. The file cache mostly shows up as Standby/Modified memory, not “In Use.” Task Manager can look comfy while the cache is actually stuffed and the system is in write-throttle mode.

Robocopy /J (and FastCopy with Non-OS cache) don’t poison the cache. They write unbuffered, so there’s no mountain of dirty pages and no system-wide throttle—hence your WinRAR job keeps flying.

It's not the first time I see a knee-jerk reaction about using AIs. As long as you're conscious they are just a tool that can make incomprehensible mistakes sometimes, there is no reason for not to use them. You just have to be careful.

Here, I have verified these findings by testing the suggested alternatives, and they work.

[Bob4K]

I just tested Goodsync that also uses buffered I/Os.

It make WinRAR stalls the same way than FFS.

That confirms previous findings.

I'm just an user reporting an issue. Whether or not it's worth addressing is of course at the discretion of the developers, I certainly have no say in that.

Just don't shoot the messenger...

[Bob4K]

3GB/s isn't POSSIBLE on a 10gb connection.
1) I have 10gb networking and RAID arrays capable of saturating it, unlike OP's HDD
2) 10gb is 1.25GB/s, half of what you claim to be getting onto the HDD at the beginning

You pointing my mistake (not thinking at the time of my post that I was using a 40G dedicated connection), and saying the problem cannot be reproduced made me try something:

Rather than using my 40G connection, I used the 10 G one.

The NAS is accessed with \\tn4\data with the 40G adapter and \\tn\data (different IPs) with 10G.

Well, I tried 3 times each, the stall only happens with 40G.

With 10G, you're right, I only get 1G speed before dropping to the HDD real speed. With 40G, it goes up to 3.3G:

https://e.pcloud.link/publink/show?code=XZpeDUZyk3hS7j3xTRgDoDcQLFjj0OnQ7X7

And there is no stall at all at 10G.

So, it's a speed related problem: Windows cache is saturated only with very high speeds, you cannot see it with *only* 10G.

[Zenju]

Last I checked, and that was a long time ago, disabling kernel-level I/O buffering during file copy lead to data corruption in some cases: viewtopic.php?t=1857

It triggered various driver bugs, because they weren't tested under this condition. There are also technical challenges like appropirate copy buffer alignment and copy buffer size - underspecified details that carefully need to be accounted for, or pay the price: which is, again, silent data corruption for the FFS user.

Then there are vague statements from Microsoft, that when disabling kernel I/O buffering "it is not recommended to pause copies" - whatever that means. Most likely also the risk of driver bugs.

Also note that without kernel buffering, file prefetching is also disabled on the input side of file copying, which can decrease performance to some degree.

So the upside of this flag is moderate, and only relevant in very specific scenarios, like yours, while the downsides are unacceptable for a file sync tool, where data correctness must come first.
If anything at all this could be an "expert setting" at the level of folder pair.

[Bob4K]

Ok, that is a very reasonable reason not to consider this option.

Thanks for pointing it out, I will use different tools for long run copies.

I guess that Microsoft's Robocopy has been thoroughly validated, I will use that.

And thank you for not biting my head off, that was an unexpected reaction from other posters!

[Zenju]

I guess that Microsoft's Robocopy has been thoroughly validated, I will use that.

Some small correction, which is also the point I was trying to make. Robocopy has not been validated, nor has any other tool that disables kernel buffering, because the potential issue is in "other people's code", notably driver code. And this depends on the specific hardware you're using.

[Bob4K]

So, risky!

A very good solution will be to use my 10G LAN and FFS, it won't affect the speed copy to a regular HHD and won't trigger the cache problems being to slow(!) for that.

So simple, but it came from getting some input here.

Edit: I tried again using the 10G adapter, perfect, WinRAR does not slow at all. Thank!

[az99]

Edit: I tried again using the 10G adapter, perfect, WinRAR does not slow at all. Thank!

What was the solution? This topic has been confusing to me (I wish it were distilled into a faq or something). Is there a flag you can use with ffs to get unbuffered io? (I can't figure out what the solution was.).

From what I gathered (from all the threads): unbuffered IO would fail for someone, so it had to be removed (as the default? or entirely?)

There is a command-line option (config file option) to verify files after copy. It seems like that would be a valid safeguard instead of disallowing unbuffered io for everyone (is it?). I can understand that corruption would be a serious problem if it happens just once. But, it seems like unbuffered could be allowed with the caveat that you should use the verify option for awhile to ensure your system can handle it. If you change your hardware, maybe ffs could detect that a hardware signature has changed, and warn the user that they're running unbuffered in a new environment, this could be a problem.

It doesn't seem like it should be disallowed because it could hurt someone. But, I can see how it's a serious topic to allow it too. (But, mainly I'm confused now if it's disallowed. Everything I read talked like it is. But, you replied that it's working now. So, I guess that's my question: What's your solution?).

[Bob4K]

Yes, the solution is to use a 10G connection rather than my fast 40G one.

It allows to use FFS as is, without the need for unbuffered IO. It's a very elegant and simple solution.

Instead of using \\tn4\data as a source, I use \\tn\data. tn4 is 192.9.204.224 while tn is 192.9.200.224, corresponding to the 2 IPs of my TrueNAS. 192.9.204.0/24 is 40G, 192.9.200.0/24 is 10G.

With a 10G connection, FFS starts with a 1GBs speed and rapidly drops to the HDD speed, less than 100MB with my latest test case.

With a 40G connection, FFS starts with a 3GBs speed, and from the results I get, it saturates the "Windows file cache" as described by ChatGPT.

I don't know this mechanism, I'm just reporting what Chat wrote.

But the result is obvious: with the 40G connection, a WinRAR extractions stalls for almost 60s when running in parallel with an FFS job, with a 10G it does not.

It's 100% repeatable on my system.

[az99]

Thank you! It sounds like the hardware/transport layer is slowing it enough that it doesn't have to go to a level of buffering that's problematic.

I wonder if there was a tool to throttle how fast data can be written to your 40g connection, if it too would work better like writing to your 10g. The only reason I mention that is that perhaps if such a throttle were adjustable, you might find 20g works, or 39g. There could be some speed that works, doesn't fall into the deep buffering your 40g does. (Depending on how far above 10g that speed is, it could be worth it?).

FWIW: Completely removing unbuffered doesn't seem like the best option to me. If those other tools you mentioned do unbuffered without reports of corruption, it seems like it's something a ffs user should be able to choose (and carefully monitor, be aware of the risk). If someone knows the risk, they could run a compare immediately after the backup to see if anything's different (or use the verify option as it backs up). It seems like overkill to me to remove the option because 1%(?) might run into corruption? (It must be a small percentage if people using the tools you mentioned aren't complaining?). If it's really a small number who are vulnerable, it seems like a manageable risk.

[Bob4K]

From what Zenju wrote here, the tools themselves have no control.

Corruption can happen with some devices depending on their driver.

Click on the link Zenju posted, it's informative. Now that I have a perfect solution, it's just not worth the risk. I don't need to use 40G to write on a drive that will max out at 250Mbs.

[xCSxXenon]

And thank you for not biting my head off, that was an unexpected reaction from other posters!

I want to clarify that I wasn't trying to be too critical. As you noted, AI tools can be dangerous if not handled with caution. You can imagine, encounters with forum posters that actually know how a computer works, let alone these advanced topics, are not frequent lol
There were definitely discrepancies in the data you were sharing, as I correctly identified and pointed out, but they were attempts to close in on a solution for you. I also came back to share that Windows doesn't count the buffer as "in-use" memory, but you found that already. I was going to suggest RAMMap to analyze, but you have found your solution already. When considering that you were actually using a 40gb link, the metrics you shared make sense.

Interesting that Windows can't handle 40gb, though. I was venturing into 40gb myself, but this may hold me back until some more research is done. I'm sure there is some solution involving Linux, and my NAS is Linux-based hosted on Proxmox, but I can't convert my main desktop off of Windows and likely will never be able to. It is nice to see why unbuffered is so risky, though it seems like a disaster waiting to happen having Microsoft be the sole bandaid ensuring data integrity. :C

[Bob4K]

No problem, I understand. You just could have been a tad more tactful...

And being seen myself as and AI is a first, that was funny. :)

40G is well worth it most of the time. It would be a pity to use only 1/3 of my NAS speed.

Another metric that might be the culprit: to get 3 GB speed, I need to set the MTU to 9000. I can't do that for the global 10G network, every device should use it which is impossible. With a standard 1504 one, speed is barely above 10G.

I made an another test that shows how useful it is: making a FFS copy between the NAS and an SSD rather than an HDD no longer impacts WinRAR operations. In fact, I extract over the same SSD that FFS is already writing at 2.3G (real speed, not a swap illusion) and WinRAR remains very fast.

Extracting and remuxing videos over the network are operations that I do all the time. They run 3 times as fast, it's well worth it.

Just avoid ConnectX-3 cards on Windows. It's been my experience that they have problems. You'll find plenty of ConnectX-4 cards on eBay that work fine.

My TrueNAS uses a ConnectX-3 card, no problem here.

Windows on your main PC should not hold you back for 40G, it really works fine aside this very specific swap problem with slow devices.

[xCSxXenon]

Wasn't calling you an AI, there was another poster that led me there, but their reply was removed lol
Good to hear! I remember switching off Connectx cards when we started having issues with Windows 11 and newer Intel CPUs. They were causing long POST delays and were dropping out of systems. Switched over to newer Mellanox Marvell Felicity based NICs and they have been awesome so far. Nice to see that ConnectX-4 is working so far, something to keep an eye on.

[Bob4K]

Glad to see that this conversation led to something positive for every one.

Cheers.