FreeFileSync Crash Apple Silicon Mac

[jrp]

When I make backup over network on my Mac Mini first everything work ok, then everything start hanging and if I don't stop backup Mac crash. That happend every time when backup size is severeal gigabytes. Small backup is work. Problem not exist when I make backup Thunderbolt or usb drive.

[Zenju]

Is there a crash report? https://freefilesync.org/faq.php#crash

[jrp]

No. Mac crash and no option to send or save crash report.

[jrp]

I am looking memory usage when I make backup. When I make local backup Mac not swap memory at all.
When I make backup over network swapped memory start increase and when is going over 2-3 gigabyte Mac is start hanging.

[xCSxXenon]

What are you backing up to?

[jrp]

I make backup my photos, videos and other personal files. Nothing special.

[xCSxXenon]

What are you backing up to?

Not what are you backing up, what are you backing up to?

[jrp]

I make backup to my Windows 10 NAS. Before Mac Mini I have Mac Pro and running OSX Mojave without any problem. Mac Pro and FreeFileSync work fine without any issue.

[xCSxXenon]

Any notable changes between when it worked and now? Trying reinstalling FFS? It doesn't sound like FFS, it sounds like there is something caching the data before sending it. Anything installed that scans files, like antivirus, cleaners, etc.?

[jrp]

Mac Mini is new but I use like old Mac Pro. Fresh install and first FreeFileSync version was 11.4 and now updated 11.5. Both has same issue. No cleaners, antivirus, etc.
I think Apple has some problem with OSX Big Sur,but FreeFileSync is only program what crash system repeatedly.

[Zenju]

Did you defaults write com.apple.CrashReporter DialogType crashreport?

[jrp]

Now i get crash report and is attached this message.

[Zenju]

Looks like a system crash and not a FreeFileSync application crash. If FreeFileSync somehow triggers the system crash it still needs to be fixed by Apple.

[jrp]

Yes, system crash, but only when I make backup FreeFileSync over network. I have another backup software, Intego Backup Express, which is old and GUI is not work perfectly, but backup over network works well. Every time when I make large backup over network with FreeFileSync Mac crash.

I hope Apple fis this issue soon. FreeFileSync is great program and hope I can use again without issue.

[schneidexe]

Just wanted to chime in as I'm facing exactly the same issue on new MacBook Air M1 with Big Sur (backing up to Synology DS218+ via SMB). Too bad, as I really got used to FFS. So if there is any chance for a fix or workaround would be great. Also happy to support if I can provide any data or sth...

[nuanda74]

Me too. My Macbook Air M1 crashes after few minutes when trying to make big backups (70GB) via LAN on a Synology NAS. Same crash report as jrp (watchdog timeout: no checkins from watchdogd...)

[Zenju]

Let's test the new FFS M1 build! Chances are good, this will avoid the crashes. It seems Rosetta 2 is consuming heaps of memory that a native build could avoid.

Please let me know how the FFS test version for M1 works for you: viewtopic.php?t=7997#p30217

[JKPhoto]

Hi all,
Just wanted to report I am also seeing this crashing issue on my 2020 M1 Macbook Pro.

I just updated to version 11.13, which I think has the M1 build mentioned as a beta in previous posts, as in Activity Monitor FFS shows as Apple now under "Kind". Unfortunately even after this update my system is still crashing when syncing large files (about 93GB in my most recent case).

FFS will start fine, and it will begin copying files over to the network drive, but after 5-10 minutes the MBP completely crashes and re-boots like I had held the power button. In my case I am using FFS Donation Edition, running Big Sur 11.6. The network drive is a shared Windows computer connected to the MBP via SMB, with an ethernet connection.

Looking at Console, I'm not seeing any crash reports unfortunately. As a result, I don't know if this is an Apple issue or a FFS issue, but I thought I would mention it here again. Everything seems to work fine connecting to the drive and copying files via the Finder. It's only using FreeFileSync that the system crashes. Hoping this can get resolved eventually, as FFS is a great utility. Thanks.

[alph-ffs]

I've been dealing with this same issue for a while now. After some apple care support, messing around with rsync and all kinds of testing finally I was able to find the culprit.

For all you of who are facing this, I would run apple built in diagnostic (making sure hardware is fine).

Then you can do one of three options:
a) Change the communication protocol to AFP
PRO: easier if you have already the volume all set
CONS: it's slower than SMB 3.1

b) Create a blank disk from disk utility - you can either choose a Read/ Write or a Sparse
PRO: it's faster than AFP + if you create a sparse of let's say 1TB, you can change all you have in the AFP/SMB volumes because the space will only grown after you start copying files (there's no need to move them to external ssd and move them back to NAS volume)
CONS: You can only mount one dmg per user

c) run a rsync script with automator
PRO: might be faster if you use the update block feature
CONS: it hasn't the UI and all the features that make FFS so good

Those kernel panic reports already hit Apple, I believe they are aware of the issue. Let's hope Monterey will address this. If not, the workaround might work just fine for a couple of weeks.

It's never enough to stress this, nothing wrong with FFS 11.14.

Quoting:
"Apple's SMB filesystem client causes system stalls on M1 Macs, leads to kernel panics"
We have received several reports from M1 Mac users of kernel panics that occur while copying files to an SMB-mounted NAS volume. The kernel panic reports have confirmed that the SMB filesystem client (implemented via the smbfs.kext kernel extension) was stalled, which led to a "watchdog" panic.

[RUBEN SALAZAR]

I have the same problem with new M1 Mac Minis, sending data over the 10Gbps network:
- From one raid to other Raid in other M1 Mac Mini or...
- To Google Drive, through the cache folder but finally comparing against the 'cloud' synced data

First I thought it was the hardware (OWC Pro Dock with 10Gbps ethernet) but the same happened with other adapter (Sonnet Thunderbolt 3 to 10Gbps ethernet).

As it was failing through FFS we tested Finder and the same data crashing was moved correctly.

The data transfer speed average can be over 300MBps with peaks of 620MBps (HDDs Raid 5 in both sides).
But at some point it crashes. Curiously the last 3 tests close to the same point after transferring between 720 and 740GB (from 800GB) and a test in between worked fine moving 989GB in less than one hour.
The failed tests were moving and average of 16,500,000 packets although the successful test moved over 21,000,000 packets and 293MBps.

Disks are not sleeping.

I installed last version of FFS and activated Rosetta, and restarted computer.

It is happening in the same environment and different countries with same M1 Big Sur Minis, OWC Raids, and 10Gbps ethernet.

[alph-ffs]

This is a known issue for SMB protocol in macOS. You should change to AFP and get it done just fine.
Other software parties have already adapted in order to make it easier to switch.

LINK:
https://bombich.com/kb/ccc6/macos-monterey-known-issues

Apple's SMB filesystem client causes system stalls, random application crashes, and may lead to kernel panics. We have received several reports (primarily from Apple Silicon Mac users) of unruly macOS behavior that occurs while copying files to an SMB-mounted NAS volume.

Eject the NAS volume if it's currently mounted
Open CCC and select the applicable backup task
Click on the Source or Destination selector (whichever is applicable for your particular task)
Hold down the Option key and choose "Switch to AFP" (provide the credentials for the NAS volume again if prompted)
Save and run the task

[RUBEN SALAZAR]

So... is it supposed that M1 with Big Sur (or Monterey) will use AFP instead SMB? In my case volumes are DAS connected to other M1 Minis. And the flow is syncing data from independent Raids to bigger disk for technicians. Data size can be an average of 2TB.

[alph-ffs]

I believe so. I'm not using Free File Sync anymore so I cannot tell if there is still an issue.

[mauricev]

I am seeing kernel panics when copying using FreeFileSync to an SMB share on an M1 Pro laptop running 12.3.

[RUBEN SALAZAR]

We were able to avoid this issue by accessing from other computers, but now, with a new pipeline of Mac Minis M1 connected to a NAS with 10Gbps the problem is back...

I tried Big Sur, Monterey and a beta of Ventura.

I managed a nsmb.conf file to force Samba 2, Samba 1, Samba 3, Samba 2 and 3,....

My next chance is upgrading all the devices -switchs, NAS, computers- to Jumbo MTU and check if reduced number of packages is better.

It seems to be always a problem linked to ethernet adapters connected to Thunderbolt 3, but not the port when using DAS devices and data transfer between disks.
Any suggestion? THANKS!

[RUBEN SALAZAR]

I have been testing again... Always the average is 30GB till the system starts suffering. And kernel_task increases the % till the computer hangs. See here:

M1 FREEFILESYNC KERNEL TASK 01.jpg (323.94 KiB) Viewed 6406 times

In this screenshot the data transferred arrives to 30GB -as usual before crashing-, the speed is reduced to 35MBps, the CPU becomes red, the Kernel_task is over 100%.

But, after stopping FFS the system still is collapsing, till I decided sending the data with a copy / paste from Finder. Then the system automatically recovered and the high speed transfer kept. In our case through a 10Gbe getting close to 800MBs from a Raid 5 with 4 disks to a NAS:

M1 FINDER NETWORK 01.jpg (205.63 KiB) Viewed 6406 times

In this screenshot I pasted the copied files to the NAS through Finder, the % CPU got automatically reduced, the data transfer speed is 792MBps, the computer didn't hang.

So no problem with Thunderbolt 3 ports or adapters to 10Gbe. No problem with Samba 3 or Samba 2 as I tried to manage. No problem with MTU for ethernet card.

I have been downloading and installing all the versions from the past and testing. The same issue and behaviour.
The version tested here is the 11.12, the first not Intel.
11.25 had the same bad result. Any suggestion?

[RUBEN SALAZAR]

However, getting content from the NAS to the M1 is working fine:

M1 GETTING DATA FROM NAS.png (585.31 KiB) Viewed 6397 times

So the problem is sending data from the M1, with the app, through the network but not with Finder

[Zenju]

@RUBEN SALAZAR:
Great testing and write up! From your descriptions it sounds like macOS is caching all copied data in the file system cache, even to the degree that the system runs out of memory and becomes unstable.
This clearly is a bug in the OS, but it's also something that FFS could avoid to trigger.

Here's a test version that doesn't write to the file system cache. Do you see any improvement?

https://www.mediafire.com/file/wztecf4aoqbgavj/FreeFileSync_11.26_beta_macOS%25282%2529.zip

[RUBEN SALAZAR]

BINGO! It keeps running without stressing the computer!! Yeah! You got the solution to people with M1s! THANKS!!

The only issue could be the speed, under 400MBps, while Finder is able to achieve 800MBps (in our case, reading from 4x HDDs Raid 5 and writing to a NAS).

[Plerry]

This may also be the result of using "Fail-safe file copy" (FFS GUI, Tools, Options, top left).
You can test, just for the sake of experiment.
But you may consider if it is wise to always turn this off; see e.g. here.

[Zenju]

The only issue could be the speed, under 400MBps, while Finder is able to achieve 800MBps (in our case, reading from 4x HDDs Raid 5 and writing to a NAS).

And before you got 700MB/sec with FFS according to your screenshots?

I suspect disabling file system cache also disables prefetching.
Here's a version that tries to enable prefetching even with disabled OS file system caching.
Do you see an improvement?
https://www.mediafire.com/file/5xna14zw59y4qcc/FreeFileSync_11.26_beta3_macOS.zip

[Zenju]

@RUBEN SALAZAR:
BTW: when you make performance measurements/comparsions, *restart* your mac between each run. Otherwise the numbers will be garbage due to buffers at various levels (OS, and hardware).

At the very least, do a

sudo purge


to clear OS cache before each test.

[RUBEN SALAZAR]

Thank you so much for this beta testing chance!!

I have been testing, the app is stable in M1. Finder is still working faster, but I guess that verification between files 'is not free'. So close to 400MBps could be great for one touch process!

FFS 11.26 BETA 3 APP VS FINDER 220912.jpg (283.83 KiB) Viewed 7015 times

In the picture, Data sent with Finder, stopping, and syncing the same content with FFS 11.26 b3

And, yes, restarting computer before testing! THANKS! Any other improvements are welcome, and I am available for testing them!

[Zenju]

I have a hunch that prefetching is enabled by default on macOS, so that enabling it explicitly does nothing.
But there's one more thing we could try:

Disable caching only for the target file.

This should partially reduce the load on the OS file system cache, and also get FFS performance back to what it was in 11.25. Can you confirm?
https://www.mediafire.com/file/fyhkzfhm6k40c3n/FreeFileSync_11.26_beta4_macOS.zip

[RUBEN SALAZAR]

FFS 11.26 BETA 4 Faster.png (351.44 KiB) Viewed 6997 times

In the picture, syncing from Raid 5 to NAS, while kernel is not in panic

It is working faster! CONGRATS! Now is achieving the speed of Finder.

I got 800MBps on weekend that the NAS was not working. I will stress it!

Curiously the 5th core is now working in its way

[Zenju]

Curiously the 5th core is now working in its way

This is probably because the input stream is still OS-buffering in this beta.

It is working faster! CONGRATS! Now is achieving the speed of Finder.

Interestingly, I'm not able to reproduce the perf-degradation when OS buffering is disabled. Quite contrary, FFS is significantly faster (about 30% shorter runtime) for local SSD-based file copying.

But there's a different bottleneck. FFS could get rid of needless intermediary buffers. This gives another good speed improvement in my tests. I believe we can match Finder with the following version which even has OS file cache disabled for both source and target.

What do you see in your tests? Can we have our cake and eat it, too?
https://www.mediafire.com/file/3hzxvvu0lg8gbed/FreeFileSync_11.26_beta5_macOS.zip

[RUBEN SALAZAR]

Thank you for this 5th beta. The behaviour is similar to 3rd. And winner is 4th!

FFS BETA 5 vs BETA 4 220913.png (451.96 KiB) Viewed 6958 times

In the picture, first installed Beta 5 and test, 400MBps average. Then overwritten with old Beta 4 and run, 600MBps average, the same as Finder copying-pasting

After testing 5th, re-installing 4th and testing, I re-installed 5th again and same result, going down to 400MBps again:

FFS BETA 5 then 4 then 5 220913.png (452.25 KiB) Viewed 6958 times

In the picture FFS Beta 5 second round. After first round I overwrote with Beta 4 getting the best performance.

[RUBEN SALAZAR]

Testing Beta 5 vs. Beta 4 with content from SSD, instead Raid 5 HDD.
Beta 4 wins!
Compared twice, both versions, with purged cache before all tests.

[Zenju]

Hm, this clearly needs more testing. I don't have other ideas at this point, other than maybe if the terminal command "cp" is as fast as Finder, we could analyze what clever thing it does: https://freefilesync.org/faq.php#trace

File copy while skipping the OS file cache is the default on macOS (both Finder and copyfile() function), so this seems to be the right direction. Disabling cache only for one side will problably not help in scenarios where source and target devices are reversed, so it's all or nothing. Therfore I'll include beta5 for the next release. If feedback is such that performance suffers in most cases, we can still go back to OS caching + system instability.

[RUBEN SALAZAR]

Yes, maybe not all users will notice the differences between 4 and 5 in terms of speed transfer but stability is the main goal!

[Zenju]

After further testing it became clear that buffer size might be the remaining issue. My SMB network share advertizes an f_iosize of 7405568 bytes, but for optimal speed matching Finder this needs to be 32 MB at least! I have no idea why buffer sizes that large make a difference, but apparently each network write is unusually expensive.

I've doubled the f_iosize sizes, which gets me in the range of only 5% slower than Finder in my tests.
With 4 x f_iosize I can match Finder. But I don't want to deviate too much from what is defined as "optimal transfer block size for the file system". Not so optimal, aparently. What do you get?

https://www.mediafire.com/file/vbaqie55wh1iwx7/FreeFileSync_11.26_beta6_macOS.zip

[RUBEN SALAZAR]

Beta 4 is still the faster vs. Beta 6. As in B3 or B5 the speed is 'limited', in our case, to 400MBps instead 600MBps achieved by B4:

FFS 11.26 BETA 4 vs BETA 6 220915.png (224.77 KiB) Viewed 6877 times

In the picture, first block Beta 4, second round with Beta 6. A non identified peak of data there (700MBps) but average of 400MBps instead 600MBps.

[Zenju]

Slower speed after buffer size increase doesn't make much sense. Unless prefetch is not happening, and FFS is switching back and forth between read and write.
Here's a version implementing prefetch via a FFS worker thread:
https://www.mediafire.com/file/oymxy279w0ypjks/FreeFileSync_11.26_beta7_macOS.zip

BTW is your target network connected using SMB or AFP? Are you using fast SSDs only or also slower devices?

[RUBEN SALAZAR]

The behaviour of slow / fast is proportional to the use of the main storage, or the use of a SSD for testing.

The main is 4x HDDs in Raid 5 (thunderbolt 3), to a NAS (1PB) over 10Gbps network with Mac Minis M1, with 10Gbe adapter.
Pending the test with 8-bay Raid 5 disks.

Protocol is SMB 3. We have been forcing SMB 2 in the past (till 3 days ago) to avoid crashes with M1s' data transferring to other M1s.
But NAS is using 3.1.1.

Using Big Sur and also testing Monterey and Ventura. Speed behaviour is the same with beta versions in the 3 O.S.

Only happens with Minis M1. iMacs work fine!

Also pending the tests with Jumbo MTU. Now is 1500. But have to change several switches / ports

[RUBEN SALAZAR]

Tested. Beta 7 is as fast as Beta 4.

[antag]

I just discovered this thread, so glad to hear this is being worked on.
I am having this issue on a Macbook Pro 16, Max 32core, 64gb ram.
My storage is a Synology 1621+ NAS over 1gbe for now.
I can help with testing from now on if needed.

[Zenju]

Tested. Beta 7 is as fast as Beta 4.

And beta 4 was the fast one in your setup! Does this mean Beta 7 is as fast as Finder on your system?

To not lose track:

beta3:  Disable caching source & target + F_RDAHEAD
beta4:  Disable caching only for the target
beta5:  Disable caching source & target + less mem copies
beta6:  Disable caching source & target + larger buffers
beta7:  Disable caching source & target + larger buffers +  prefetching thread


[Zenju]

I just discovered this thread, so glad to hear this is being worked on.
I am having this issue on a Macbook Pro 16, Max 32core, 64gb ram.
My storage is a Synology 1621+ NAS over 1gbe for now.
I can help with testing from now on if needed.

Perfect! Can you test betas 6 and 7 and compare the speed with Finder? Be sure to "sudo purge" before each test!

[RUBEN SALAZAR]

A curiosity... A Mac Mini M1 running 1 instance of FFS is getting about 250 or 300MBps from an external SSD connected through USB 10Gbps to a Thunderbolt 3 4x Raid 5.
And a second instance running at the same time, from other SSD to the same Raid, different folders, is achieving close to the same speed.
The sum of 2 FFSs running at once is faster, getting the speed of the bus.

[Zenju]

@RUBEN SALAZAR:
Any feedback on viewtopic.php?t=8039&p=35220#p35266 ?

@antag:
Have you made any tests so far? viewtopic.php?p=35316#p35267