How to sync "symbolic links" in linux?

Posts: 4 · CRD 9 Sep 2020, 20:21

Hi there!

I've just installed FFS and it seems really great. However, I'm trying to sync two linux machines and I cannot find, for my life, a way to sync symbolic links (which I want to be identical in both machines). I always get the error message "Operation not supported between different devices."

I've ticked "Direct" in "Include symbolic links" in the Synchronizing Settings.

Can someone, please, advice? Many thanks!

Posts: 7505 · Zenju 10 Sep 2020, 07:45

What is the full error message?

Posts: 4 · CRD 10 Sep 2020, 08:37

Hi Zenju,

The full message is:

Cannot copy symbolic link "sftp://server/folder/link" to "/folder/link".

Operation not supported between different devices.

Any ideas?

Posts: 4 · CRD 17 Sep 2020, 22:51

More details, in case it helps: I'm using FreeFileSync 11.1 on Linux Mint 20 Cinnamon...

Posts: 7505 · Zenju 25 Sep 2020, 17:11

FreeFileSync should support this. => ToDo

Posts: 4 · CRD 26 Sep 2020, 15:36

Thank you , Zenju!

Just for reference, in case some one else is in the same situation, I'm syncing the symbolic links in the meanwhile using rsync:

rsync -avzu -e ssh "remotehost:/path/remotesymlink" "/path/localsynmlink"

(see rsync man page for details...)

Posts: 2 · ryoushi86 13 Jan 2021, 07:32

Hello,

Any update about this issue? I'm trying to copy symbolic links between 2 devices and the error message is :
Info : Creating symbolic link "/mnt/whatever"
Error : Cannot copy symbolink link
"sftp://mnt/whatever to /mnt/whatever".
Operation not supported between different devices.

I'm using FreeFileSync 11.5 on Debian 10 with "Direct" box ticked

Thank you for your wonderfull software !

Posts: 65 · mdrmdr 15 Jan 2021, 11:17

I second this request to be able to sync symbolic links from (in my case) Linux to Windows. A Linux symbolic link should result at Windows in the file the link is pointing to.

Thanks a lot!

Posts: 7505 · Zenju 15 Jan 2021, 12:19

I second this request to be able to sync symbolic links from (in my case) Linux to Windows. A Linux symbolic link should result at Windows in the file the link is pointing to. mdrmdr, 15 Jan 2021, 11:17

This should already work. See "Symbolic Link Handling": https://freefilesync.org/manual.php?topic=comparison-settings

Posts: 65 · mdrmdr 15 Jan 2021, 12:25

Unfortunately not. I get "Operation not supported between different devices." I included symbolic links as "direct". I use FFS 11.5 on Windows and access Linux via ftp.

Posts: 7505 · Zenju 15 Jan 2021, 12:27

Unfortunately not. I get "Operation not supported between different devices." I included symbolic links as "direct". mdrmdr, 15 Jan 2021, 12:25

So this is also SFTP => Windows?

Posts: 65 · mdrmdr 15 Jan 2021, 12:28

Not SFTP, standard FTP.

Posts: 7505 · Zenju 15 Jan 2021, 12:29

You need to set symlink handling to "follow".

Posts: 65 · mdrmdr 15 Jan 2021, 12:36

No success. Then I get many errors like

CURLE_FTP_COULDNT_RETR_FILE: RETR response: 550

550 '/opt/automation/config_links/000-default-ssl.conf' is not a directory [curl_easy_perform]

Posts: 2 · ryoushi86 15 Jan 2021, 12:46

Syncing symbolic links between 2 linux boxes would be very nice :)

Posts: 65 · mdrmdr 15 Jan 2021, 12:51

FileZilla also shows the 550 error:

Command: CWD 000-default-le-ssl.conf

Answer: 550 000-default-le-ssl.conf: Not a directory

But then it does not stop and just copies the linked file....

Posts: 7505 · Zenju 15 Jan 2021, 16:06

FileZilla also shows the 550 error:
Command: CWD 000-default-le-ssl.conf Answer: 550 000-default-le-ssl.conf: Not a directory
But then it does not stop and just copies the linked file.... mdrmdr, 15 Jan 2021, 12:51

Indeed. The FTP protocol is very limited compared to SFTP regarding symlinks but FreeFileSync might be able to do something about this => ToDo

Posts: 65 · mdrmdr 15 Jan 2021, 16:14

Posts: 7505 · Zenju 16 Jan 2021, 10:19

The FTP protocol is very limited compared to SFTP regarding symlinks but FreeFileSync might be able to do something about this => ToDo Zenju, 15 Jan 2021, 16:06

Implemented! FreeFileSync now handles symbolic links pointing to files via FTP (before it only supported links pointing to directories)
https://www.mediafire.com/file/o9m2m2y0c61al7w/FreeFileSync_11.6_%255BBeta%255D_Windows_Setup%25284%2529.exe

Posts: 65 · mdrmdr 16 Jan 2021, 11:19

Installed 11.6 Beta and still the same error "Operation not supported between different devices". German error message attached.

Posts: 7505 · Zenju 16 Jan 2021, 11:50

You still need to set symlink handling to "follow".

Posts: 65 · mdrmdr 16 Jan 2021, 13:06

If I set it to "follow", I get this error for each linked file:

Posts: 65 · mdrmdr 16 Jan 2021, 13:18

In the affected directory on my Raspberry, I have links to files and directories. I'd like to only sync the linked files, not the directories. Excluding the directories still gives the above error.

Actually I would expect the following behaviour:
• "direct": sync linked files and ignore linked directories
• "follow": sync the whole tree of a linked directory (possibly with a "depth" parameter?) and sync linked files

Therefore I'd prefer "direct" with the above behaviour or "follow" with "depth=0"

Posts: 7505 · Zenju 16 Jan 2021, 13:24

If I set it to "follow", I get this error for each linked file: mdrmdr, 16 Jan 2021, 13:06

It seems "fullchain.pem" is incorrectly considered a directory. Here's a debug version of FreeFileSync, which error details do you get?
https://www.mediafire.com/file/t7mj94453u3zbs5/FreeFileSync_11.6_%255BBeta%255D_Windows_Setup%25285%2529.exe

Posts: 65 · mdrmdr 16 Jan 2021, 13:36

Same "is not a directory" error during "compare" as before for any linked file with "follow". No further details. Where should I see more details?

Posts: 7505 · Zenju 16 Jan 2021, 14:00

Did you run the same test case? I've added code if the item name is "fullchain.pem" and FreeFileSync will show a different error message. Also check you're really running the correct beta and not some other version.

Posts: 65 · mdrmdr 16 Jan 2021, 14:49

Now, the only link to be synced is:
cert -> /etc/letsencrypt/live/dyn.xxx.de/ (link)

Within "/etc/letsencrypt/live/dyn.xxx.de/":
fullchain.pem -> ../../archive/dyn.xxx.de/fullchain450.pem (file)

This gives:

: FFS3.jpg (43.07 KiB) Viewed 7065 times

If (for debugging only):
cert -> fullchain.pem (file)

it works and FFS syncs the the path "cert -> fullchain.pem"
So it seems that FFS only is able to follow 1 level?

if (for debugging only):
fullchain.pem -> /etc/letsencrypt/live/dyn.xxx.de/fullchain.pem (file)
I get:

: FFS4.jpg (31.54 KiB) Viewed 7065 times

Posts: 65 · mdrmdr 16 Jan 2021, 14:51

Upate: Now the attachments of the above post are correct...
--
(There seems to be an attachment inlining problem at your board. The second picture after "I get" should be
FFS4 not FFS3 again...)

Posts: 65 · mdrmdr 16 Jan 2021, 14:56

But again: My standard case is to follow links down just by one level to exactly one file. If a link points to a directory, the danger under Linux is, that a huge tree is affected. Therefore I'd exclude links to directories and only let FFS sync links which directly points to a file. For the above debugging example, I had to move some files out of "/etc/letsencrypt/live/dyn.xxx.de" so that FFS only finds "fullchain.pem".

Posts: 7505 · Zenju 16 Jan 2021, 16:43

Now, the only link to be synced is:
cert -> /etc/letsencrypt/live/dyn.xxx.de/ (link)

Within "/etc/letsencrypt/live/dyn.xxx.de/":
fullchain.pem -> ../../archive/dyn.xxx.de/fullchain450.pem (file)

This gives: [SIZE: 550] mdrmdr, 16 Jan 2021, 14:49

This is either:
1. an issue with the FTP server: it cannot find the file at /opt/automation/config_links/cert/fullchain.pem although it should, or...
2. /opt/automation/config_links/cert/fullchain.pem is a broken symlink

If (for debugging only):
cert -> fullchain.pem (file)

it works and FFS syncs the the path "cert -> fullchain.pem"
So it seems that FFS only is able to follow 1 level? mdrmdr, 16 Jan 2021, 14:49

This is FFS's new file-symlink support => okay

if (for debugging only):
fullchain.pem -> /etc/letsencrypt/live/dyn.xxx.de/fullchain.pem (file)
I get: [SIZE: 213] mdrmdr, 16 Jan 2021, 14:49

This is also FFS's new file-symlink support => okay
(the error message only comes up because it is the debug version)

Posts: 65 · mdrmdr 16 Jan 2021, 17:44

This is either:
1. an issue with the FTP server: it cannot find the file at /opt/automation/config_links/cert/fullchain.pem although it should, or...
2. /opt/automation/config_links/cert/fullchain.pem is a broken symlink

Don't think so, since e.g. FileZilla has no problems following the 2 links and copying the resulting file. And e.g. "ls" shows that it's a valid link:

PI_PROD$ ls -l /opt/automation/config_links/cert/fullchain.pem

lrwxrwxrwx 1 root root 41 Jan 15 00:00 /opt/automation/config_links/cert/fullchain.pem -> ../../archive/dyn.xxx.de/fullchain450.pem

This is FFS's new file-symlink support => okay
This is also FFS's new file-symlink support => okay

Sounds great :-) So I'll wait, until the new final 11.6 (donation) version is online.

Thanks a lot for your efforts so far and your great FFS...

Posts: 7505 · Zenju 16 Jan 2021, 17:55

This is either:
1. an issue with the FTP server: it cannot find the file at /opt/automation/config_links/cert/fullchain.pem although it should, or...
2. /opt/automation/config_links/cert/fullchain.pem is a broken symlink
Don't think so, since e.g. FileZilla has no problems following the 2 links and copying the resulting file. And e.g. "ls" shows that it's a valid link: mdrmdr, 16 Jan 2021, 17:44

FreeFileSync issues a SIZE /opt/automation/config_links/cert/fullchain.pem which fails with 550. You can test this by manually entering this command with FileZilla. You should see the same 550 error (if not we have something interesting to investigate)

Assuming "fullchain.pem" is not broken this indicates an FTP server bug and FileZilla just got lucky to not trigger it.

Posts: 65 · mdrmdr 16 Jan 2021, 18:01

I wrote already, that FileZilla does see the error but continues:

Status:    Retrieving directory listing of "/etc/letsencrypt/live/dyn.xxx.de/fullchain.pem"...

Command:    CWD fullchain.pem

Response:    550 fullchain.pem: Not a directory

Error:    Failed to retrieve directory listing

Status:    Connecting to 10.0.0.10:21...

Status:    Connection established, waiting for welcome message...

Status:    Initializing TLS...

Status:    Retrieving directory listing of "/etc/letsencrypt/live/dyn.xxx.de"...

Status:    Directory listing of "/etc/letsencrypt/live/dyn.xxx.de" successful

Status:    Verifying certificate...

Status:    TLS connection established.

Status:    Server does not support non-ASCII characters.

Status:    Logged in

Status:    Starting download of /etc/letsencrypt/live/dyn.xxx.de/fullchain.pem

Status:    File transfer successful, transferred 3.761 bytes in 1 second

Posts: 7505 · Zenju 16 Jan 2021, 19:18

I wrote already, that FileZilla does see the error but continues:

Status: Retrieving directory listing of "/etc/letsencrypt/live/dyn.xxx.de/fullchain.pem"... Command: CWD fullchain.pem Response: 550 fullchain.pem: Not a directory Error: Failed to retrieve directory listing Status: Connecting to 10.0.0.10:21... Status: Connection established, waiting for welcome message... Status: Initializing TLS... Status: Retrieving directory listing of "/etc/letsencrypt/live/dyn.xxx.de"... Status: Directory listing of "/etc/letsencrypt/live/dyn.xxx.de" successful Status: Verifying certificate... Status: TLS connection established. Status: Server does not support non-ASCII characters. Status: Logged in Status: Starting download of /etc/letsencrypt/live/dyn.xxx.de/fullchain.pem Status: File transfer successful, transferred 3.761 bytes in 1 second
mdrmdr, 16 Jan 2021, 18:01

That's not what I was talking about. FileZilla let's you enter FTP commands manually (right-click on the log window).

Posts: 65 · mdrmdr 16 Jan 2021, 22:05

Sorry, I did execute the FileZilla command before, but did not post it, as it also gave a 550 error:

Command: SIZE /opt/automation/config_links/cert/fullchain.pem

Response: 550 /opt/automation/config_links/cert/fullchain.pem: No such file or directory

And don't ask me why, but STAT works:

Command: STAT /opt/automation/config_links/cert/fullchain.pem

Response: 213-Status of /opt/automation/config_links/cert/fullchain.pem:

Response: 213-lrwxrwxrwx   1 root     root           41 Jan 16 16:56 /opt/automation/config_links/cert/fullchain.pem -> /opt/automation/archive/dyn.xxx.de/fullchain450.pem

Response: 213 End of status

Weird, that "/opt/automation/archive/dyn.xxx.de/fullchain450.pem" is an invalid path... FileZilla seems to combine the containing path of the 1st symlink with the resulting path the 2nd symlink. See the full path path resolution above.

Posts: 65 · mdrmdr 16 Jan 2021, 22:29

Found an explanation how FileZilla does it:

The directory listings returned by the server may contain information about symbolic links, though this is not covered by neither the FTP specifications nor the SFTP specifications. If FileZilla encounters a symbolic link, it first tries to treat it like a directory, and if that fails, treats it like a file. To prevent endless loops during recursive operations, symlink targets must be within the root of the operation. There is no option to control how symlinks are handled.

[https://forum.filezilla-project.org/viewtopic.php?t=30485]

Posts: 7505 · Zenju 16 Jan 2021, 23:03

fullchain.pem -> ../../archive/dyn.xxx.de/fullchain450.pem (file)
and fullchain.pem is located in "/etc/letsencrypt/live/dyn.xxx.de/",
so the resolved path is

/etc/letsencrypt/archive/dyn.xxx.de/fullchain450.pem

If this is an existing file, then the SIZE FTP command should return its size, if not => FTP server bug

The STAT command seems to return successfully (213), but the shown resolved path (/opt/automation/config_links/cert/fullchain.pem) is incorrect => FTP server bug

Found an explanation how FileZilla does it: mdrmdr, 16 Jan 2021, 22:29

FreeFileSync does the reverse: it checks if the path points to a file (via SIZE) and assume a folder symlink on failure (550).

The problem with FileZilla's approach is that treating a file symlink like a folder by traversing it does not always fail. e.g. Pure-FTPd returns garbage data with success code for MLSD (e.g. "/<symlink name>" with a bogus / at the beginning of a nonexistent file at <parent path>/<symlink name>//<symlink name>)

Posts: 65 · mdrmdr 16 Jan 2021, 23:12

/etc/letsencrypt/archive/dyn.xxx.de/fullchain450.pem
If this is an exising file, then the SIZE FTP command should return its size, if not => FTP server bugZenju, 16 Jan 2021, 23:03

Yes, this path is correct! I'll check if I can install a better/newer ftp server on my Raspberry. Currently it's the latest available ProFTPD Version 1.3.6

Posts: 65 · mdrmdr 16 Jan 2021, 23:22

ProFTPD v1.3.7a seems to solve it. I'll try to compile it myself... (tomorrow - it's late now :-)

Posts: 65 · mdrmdr 18 Jan 2021, 20:17

After having some issues with building ProFTPD V1.3.7a from source, I finally managed it. And voilà, it works with your initially provided FFS beta version. Direct links to files are now properly synced. And the size is properly shown in the Size column.

One strange thing: There is "sometimes" an issue with the link:

apache2.conf -> /etc/apache2/apache2.conf

I've to further check under what circumstances this happens, so that the problem is reproducible...

Posts: 65 · mdrmdr 18 Jan 2021, 20:28

I can't see a pattern. "Compare" usually works. But sometimes with 2-3 links to normal ASCII files error 550 (is not a directory) is displayed. Very strange...

If i do

size /opt/automation/config_links/apache2.conf

in FileZilla, it works now all time.

Posts: 7505 · Zenju 19 Jan 2021, 11:13

If comparison works ("usually" => not always?), then this is probably an unrelated issue. What is the full error message?

Posts: 65 · mdrmdr 19 Jan 2021, 16:16

It's always the same error on random 2-3 files out of 21 files in the directory containing all the symlinks. I'll try to enable a full debug log of ProFTPD and check if I can see something. It's funny, that most of the time "apache2.conf" and "crontab" shows the problem.

Posts: 65 · mdrmdr 19 Jan 2021, 18:37

One finding from the detailled ProFTPD logfile: For each failing file a

SIZE not allowed in ASCII mode

is in the log. Any idea?

Posts: 7505 · Zenju 19 Jan 2021, 19:37

Perfect, you found the issue! FFS needs to set "TYPE I" before "SIZE". Will see to post an update tomorrow.

Posts: 65 · mdrmdr 19 Jan 2021, 20:23

For both, compare and sync, what rules does FFS apply to select either ASCII or binary mode?

Posts: 7505 · Zenju 20 Jan 2021, 10:51

Fixed (FFS never uses ASCII, only binary mode)
https://www.mediafire.com/file/snuj27yy3ndyamx/FreeFileSync_11.6_%255BBeta%255D_Windows_Setup%25287%2529

Posts: 65 · mdrmdr 20 Jan 2021, 14:40

Can confirm, that it works now. Also following (compare & sync) of a directory tree works well. Will make more tests tonight, but looks good so far :-)

Update Jan 22nd: Found no further errors. Works as expected :-)