Syncing with Extended Characters from a SSHFS on OSX

[cju18]

Hi,
thanks for the great application. Since FFS v8 I get an error on files with special characters on OSX (10.11) when syncing my sshfs (MacFuse) directory.
It concerns all folders and files with "ä, ü, ö" etc. The Errormessage:

Error Code 2: No such file or directory (lstat)


Those file exists, in an explorer its possible to copy them.
Everything works still fine on FFS v7.9.

Thanks! cj18


Seems to be the same bug:

On a Mac running OS/X El Capitan, I'm trying to sync from a Windows share to a native filesystem. I'm getting the following error on a lot of files. The one common factor is that all the filenames include extended characters such as accents or umlauts. Is there a workaround?

Cannot read file attributes of "/Volumes/Music/Various Artists/História da Música Eletroacústica CD 36 - 1982-1985".

Error Code 2: No such file or directory (lstat)

[smz]

I think I might know where the problem lies:

In UTF-8 some visually equivalent characters can be represented with different code points. Particularly accented characters can be represented in a composed form (a single codepoint representing the accented character) or in a decomposed form (the base character codepoint + the accent codepoint).

Mac OS (HFS+) is, AFAIK, unique as it is the only OS requiring file names to be in the decomposed form.

In Windows both forms are accepted but only the composed form is used by default. You can end up having two files with apparently identical names in the same folder...

Please see, amongst other interesting places:
https://en.wikipedia.org/wiki/Unicode_equivalence
http://apple.stackexchange.com/questions/83935/unicode-normalization-for-filenames-and-copied-text-from-pdfs

I hope this can be helpful...

Cheers!

smz

[smz]

A better example on http://apple.stackexchange.com:

http://apple.stackexchange.com/questions/95483/utf8-filenames-and-shell-utilities

[Zenju]

It seems the best solution is to go with decomposed UTF on OS X. Let me know if you are still having issues with the new version:
http://www.mediafire.com/download/3rqpn3yac5g1k3k/FreeFileSync_8.2_beta_Mac_OS_X.zip

[Alfont]

Similar problem still with 8.6

[Zenju]

I think this deserves a more fundamental and portable solution. Back on the todo list.

[chanquete]

I just joined the forum. Great program! I'm trying to get back on this dormant topic.
I can sync in network at work using a USB stick plugged into any computer (with the program installed on a Windows Server 2008 R2) but when trying to sync from home with my Mac, connected via VPN, I get all the sort of problems explained above. I was hopeful when one of the site administrators (Zenju) put the subject back in the todo list.

I was wondering if a filter could be created to ignore those "weird" characters, pretty much like converting them into wildcards while matching the names during comparison.

Cheers!!

[Zenju]

The next FFS version will be aware of different Unicode normalization forms and should solve all Unicode issues (... at least the ones that are solvable... :> )

On macOS there is one peculiar case that can be considered a bug of the OS:
If file names in NFD (decomposed form) are shared with macOS via Samba, the macOS network drivers convert the file names to NFC. This is essentially a data loss of the original names: Since other OS like Windows or Linux - where the files are hosted - DO care about Unicode normalization differences, these NFC names won't be found and lead to errors of the kind:

Error Code 2: No such file or directory (lstat)


If the macOS Samba implementation converted to NFC in all cases it wouldn't even be such a huge problem. But it only does this for direct file accesses, but *not* for directory listing. This non-sensical behavior is the root of the problem. The solution would have been so simple: Just don't corrupt user data an leave the Unicode normalization forms of file names be, like Linux, Windows do.

This is a bit ironic, since NFD is the natural Unicode normalization form of macOS's HFS+, but by trying to be "clever" and converting the original NFD to NFC, thinking this would improve compatibility with Windows, the macOS Samba implementation actually created an issue that is unsolvable from other software layers and made these files unreadable: Finder will simply not show them, while FFS at least shows error messages instead of assuming "non-existence" which could lead to real data loss during sync if FFS wouldn't properly detect the issue.

Here's a writeup from someone else describing the same Samba bug on macOS: https://gist.github.com/tempelmann/8f5232550c55a9d9640e5eb7cb3d7cd8#file-main-m-L50

It mentions a workaround which I haven't tested:

You should mount your NFS share using “nfc” parameter
http://www.openradar.me/radar?id=4984843016339456

Also, here's an ELI5: viewtopic.php?t=9986#p37007