How to sync "symbolic links" in linux?

[mdrmdr]

This is either:
1. an issue with the FTP server: it cannot find the file at /opt/automation/config_links/cert/fullchain.pem although it should, or...
2. /opt/automation/config_links/cert/fullchain.pem is a broken symlink

Don't think so, since e.g. FileZilla has no problems following the 2 links and copying the resulting file. And e.g. "ls" shows that it's a valid link:

PI_PROD$ ls -l /opt/automation/config_links/cert/fullchain.pem
lrwxrwxrwx 1 root root 41 Jan 15 00:00 /opt/automation/config_links/cert/fullchain.pem -> ../../archive/dyn.xxx.de/fullchain450.pem


This is FFS's new file-symlink support => okay
This is also FFS's new file-symlink support => okay

Sounds great :-) So I'll wait, until the new final 11.6 (donation) version is online.

Thanks a lot for your efforts so far and your great FFS...

[Zenju]

This is either:
1. an issue with the FTP server: it cannot find the file at /opt/automation/config_links/cert/fullchain.pem although it should, or...
2. /opt/automation/config_links/cert/fullchain.pem is a broken symlink

Don't think so, since e.g. FileZilla has no problems following the 2 links and copying the resulting file. And e.g. "ls" shows that it's a valid link:

FreeFileSync issues a SIZE /opt/automation/config_links/cert/fullchain.pem which fails with 550. You can test this by manually entering this command with FileZilla. You should see the same 550 error (if not we have something interesting to investigate)

Assuming "fullchain.pem" is not broken this indicates an FTP server bug and FileZilla just got lucky to not trigger it.

[mdrmdr]

I wrote already, that FileZilla does see the error but continues:

Status:    Retrieving directory listing of "/etc/letsencrypt/live/dyn.xxx.de/fullchain.pem"...
Command:    CWD fullchain.pem
Response:    550 fullchain.pem: Not a directory
Error:    Failed to retrieve directory listing
Status:    Connecting to 10.0.0.10:21...
Status:    Connection established, waiting for welcome message...
Status:    Initializing TLS...
Status:    Retrieving directory listing of "/etc/letsencrypt/live/dyn.xxx.de"...
Status:    Directory listing of "/etc/letsencrypt/live/dyn.xxx.de" successful
Status:    Verifying certificate...
Status:    TLS connection established.
Status:    Server does not support non-ASCII characters.
Status:    Logged in
Status:    Starting download of /etc/letsencrypt/live/dyn.xxx.de/fullchain.pem
Status:    File transfer successful, transferred 3.761 bytes in 1 second


[Zenju]

I wrote already, that FileZilla does see the error but continues:

Status:    Retrieving directory listing of "/etc/letsencrypt/live/dyn.xxx.de/fullchain.pem"...
Command:    CWD fullchain.pem
Response:    550 fullchain.pem: Not a directory
Error:    Failed to retrieve directory listing
Status:    Connecting to 10.0.0.10:21...
Status:    Connection established, waiting for welcome message...
Status:    Initializing TLS...
Status:    Retrieving directory listing of "/etc/letsencrypt/live/dyn.xxx.de"...
Status:    Directory listing of "/etc/letsencrypt/live/dyn.xxx.de" successful
Status:    Verifying certificate...
Status:    TLS connection established.
Status:    Server does not support non-ASCII characters.
Status:    Logged in
Status:    Starting download of /etc/letsencrypt/live/dyn.xxx.de/fullchain.pem
Status:    File transfer successful, transferred 3.761 bytes in 1 second


That's not what I was talking about. FileZilla let's you enter FTP commands manually (right-click on the log window).

[mdrmdr]

Sorry, I did execute the FileZilla command before, but did not post it, as it also gave a 550 error:

Command: SIZE /opt/automation/config_links/cert/fullchain.pem
Response: 550 /opt/automation/config_links/cert/fullchain.pem: No such file or directory


And don't ask me why, but STAT works:

Command: STAT /opt/automation/config_links/cert/fullchain.pem
Response: 213-Status of /opt/automation/config_links/cert/fullchain.pem:
Response: 213-lrwxrwxrwx   1 root     root           41 Jan 16 16:56 /opt/automation/config_links/cert/fullchain.pem -> /opt/automation/archive/dyn.xxx.de/fullchain450.pem
Response: 213 End of status


Weird, that "/opt/automation/archive/dyn.xxx.de/fullchain450.pem" is an invalid path... FileZilla seems to combine the containing path of the 1st symlink with the resulting path the 2nd symlink. See the full path path resolution above.

[mdrmdr]

Found an explanation how FileZilla does it:

The directory listings returned by the server may contain information about symbolic links, though this is not covered by neither the FTP specifications nor the SFTP specifications. If FileZilla encounters a symbolic link, it first tries to treat it like a directory, and if that fails, treats it like a file. To prevent endless loops during recursive operations, symlink targets must be within the root of the operation. There is no option to control how symlinks are handled.

[https://forum.filezilla-project.org/viewtopic.php?t=30485]

[Zenju]

fullchain.pem -> ../../archive/dyn.xxx.de/fullchain450.pem (file)
and fullchain.pem is located in "/etc/letsencrypt/live/dyn.xxx.de/",
so the resolved path is

/etc/letsencrypt/archive/dyn.xxx.de/fullchain450.pem

If this is an existing file, then the SIZE FTP command should return its size, if not => FTP server bug

The STAT command seems to return successfully (213), but the shown resolved path (/opt/automation/config_links/cert/fullchain.pem) is incorrect => FTP server bug

Found an explanation how FileZilla does it:

FreeFileSync does the reverse: it checks if the path points to a file (via SIZE) and assume a folder symlink on failure (550).

The problem with FileZilla's approach is that treating a file symlink like a folder by traversing it does not always fail. e.g. Pure-FTPd returns garbage data with success code for MLSD (e.g. "/<symlink name>" with a bogus / at the beginning of a nonexistent file at <parent path>/<symlink name>//<symlink name>)

[mdrmdr]

/etc/letsencrypt/archive/dyn.xxx.de/fullchain450.pem
If this is an exising file, then the SIZE FTP command should return its size, if not => FTP server bug

Yes, this path is correct! I'll check if I can install a better/newer ftp server on my Raspberry. Currently it's the latest available ProFTPD Version 1.3.6

[mdrmdr]

ProFTPD v1.3.7a seems to solve it. I'll try to compile it myself... (tomorrow - it's late now :-)

[mdrmdr]

After having some issues with building ProFTPD V1.3.7a from source, I finally managed it. And voilà, it works with your initially provided FFS beta version. Direct links to files are now properly synced. And the size is properly shown in the Size column.

One strange thing: There is "sometimes" an issue with the link:

apache2.conf -> /etc/apache2/apache2.conf


I've to further check under what circumstances this happens, so that the problem is reproducible...

[mdrmdr]

I can't see a pattern. "Compare" usually works. But sometimes with 2-3 links to normal ASCII files error 550 (is not a directory) is displayed. Very strange...

If i do

size /opt/automation/config_links/apache2.conf


in FileZilla, it works now all time.

[Zenju]

If comparison works ("usually" => not always?), then this is probably an unrelated issue. What is the full error message?

[mdrmdr]

It's always the same error on random 2-3 files out of 21 files in the directory containing all the symlinks. I'll try to enable a full debug log of ProFTPD and check if I can see something. It's funny, that most of the time "apache2.conf" and "crontab" shows the problem.

[mdrmdr]

One finding from the detailled ProFTPD logfile: For each failing file a

SIZE not allowed in ASCII mode


is in the log. Any idea?

[Zenju]

Perfect, you found the issue! FFS needs to set "TYPE I" before "SIZE". Will see to post an update tomorrow.

[mdrmdr]

For both, compare and sync, what rules does FFS apply to select either ASCII or binary mode?

[Zenju]

Fixed (FFS never uses ASCII, only binary mode)
https://www.mediafire.com/file/snuj27yy3ndyamx/FreeFileSync_11.6_%255BBeta%255D_Windows_Setup%25287%2529

[mdrmdr]

Can confirm, that it works now. Also following (compare & sync) of a directory tree works well. Will make more tests tonight, but looks good so far :-)

Update Jan 22nd: Found no further errors. Works as expected :-)