FreeFileSync FTP command channel idle timeout CURLE_OPERATION_TIMEDOUT etc.

[koto0k]

When FreeFileSync try to sync files via xFTP, huge size files (in my case 100...200+ Gigs) via midle speed channels or midsize files via slow channels can not sync due to timeout of TCP connections via a control connection to submit commands and receive replies while data channel continue transfer data.

If you can transfer small files without any issues, but transfers of larger files end with a timeout, a broken router and/or firewall may exist between the client and the server and is causing a problem.

It is a nature of FTP protocol where the command channel needs to be alive while data channel makes data transfer to from 0 to 100% and TCP window is still opened.

In an attempt to solve this problem, the TCP specifications include a way to send keep-alive packets such as NOOP commands on otherwise idle TCP connections, to tell all involved parties that the connection is still alive and needed.

As i see FreeFileSync does not have machanism to keep FTP control channel alive or in work condition because NOOP command does not send periodically to FTP as i see in FTPs log. And big files cant be transferred due to closed (by router, by OS, etc.) TCP command channel by timeout.

To my mind it is no problem to send HELP command via ftp and if 214-supported commands:... NOOP,... present - try to use it to keep alive TCP command channel.

[Zenju]

FFS is using CURLOPT_TCP_KEEPALIVE with 60 sec default.

[koto0k]

When I try to download files via ftp with size 100+ Gbytes the
CURLE_OPERATION_TIMEDOUT: Operation too slow. Less than 1 bytes/sec transferred the last 120 seconds
150 Starting data transfer. [curl_easy_perform]
and operation restart on high speed and stay linearly, after some time (2,4 or 6 hours) download speed decreases histrogramically and download operation restart again and again and upper error logs

In second sample, when I try to download files in test local network area there are no problems with big files 200+ Gbytes, but speeds are nead 1Gbit (for example switch is 1 Gbit)

But in real environment where files are stored in other country and network is unknown but speed is near 20..40 Mbits/sec this error is present

[koto0k]

all that I could not do with FreeFileSync during the week
I did it (download my 300 Gb big files) with good old ftp software in one day.

The todays FreeFileSyncs curl problem is that it is not adapted to work with huge size files for a long time in real and not fully ideal networks

Problem CURLE_OPERATION_TIMEDOUT: Operation too slow. Less than 1 bytes/sec transferred the last 120 seconds 150 Starting (restarting) data transfer. [curl_easy_perform] still exist

i am disappointed in FreeFileSync

[Zenju]

Maybe you could find out what the problem is. Perhaps the CURLOPT_TCP_KEEPALIVE isn't working as expected, although it worked for FTP in this case: viewtopic.php?t=6928

[koto0k]

As I see in your post - problem is CURLE_OPERATION_TIMEDOUT: FTP response timeout

But in my situation Curl can not download huge files with constant speed. It downloads with the is speed decreases to zero over long time.

problem is that ftp software support correct redownload from broken connections etc + uses ftp protocol commands such as noop, retr, etc. to keep the channel in good shape

May be in curl huge files is problem when files are downloading over a long period of time via real networks wich need restart download from previous step on broken connections etc.

I noticed that even with web servers too, large downloads occur with a drop in speed and there are many servers whose specifications say so - tested for working with large files

[Zenju]

Without proper analysis of FTP traffic logs this is only guesswork.

[koto0k]

I try to download same files from ftp server in same time with ftp client and freefilesync on the same PC with shaping traffic 50/50 to ftp client and FFS. FTP client work good while FFS CURLE_OPERATION_TIMEDOUT: Operation too slow. Less than 1 bytes/sec transferred the last 120 seconds 150 Starting (restarting) data transfer. [curl_easy_perform] still exist periodically when ftp client continued to download files at that same time

[koto0k]

I think I spent so much of my time describing the problem - I'd rather use other proven utilities

[koto0k]

logs from ffs and ftp server attached

[Zenju]

(000055) 18.02.2025 14:34:28 - bck_usr376 (ext_ip_376.int)> disconnected.
(000053) 18.02.2025 15:58:29 - bck_usr376 (ext_ip_376.int)> 426 Connection closed; aborted transfer of "/disk_e/imgs/cb_376.dmp.tgz"

(000061) 18.02.2025 16:00:26 - bck_usr376 (ext_ip_376.int)> TLS connection for data connection established
(000061) 18.02.2025 16:53:28 - bck_usr376 (ext_ip_376.int)> 426 Connection closed; aborted transfer of "/disk_e/imgs/cb_376.dmp.tgz"

(000064) 18.02.2025 16:55:26 - bck_usr376 (ext_ip_376.int)> TLS connection for data connection established
(000064) 18.02.2025 17:03:28 - bck_usr376 (ext_ip_376.int)> 426 Connection closed; aborted transfer of "/disk_e/imgs/cb_376.dmp.tgz"

(000065) 18.02.2025 17:05:26 - bck_usr376 (ext_ip_376.int)> TLS connection for data connection established
(000065) 18.02.2025 17:23:30 - bck_usr376 (ext_ip_376.int)> 426 Connection closed; aborted transfer of "/disk_e/imgs/cb_376.dmp.tgz"

(000066) 18.02.2025 17:25:27 - bck_usr376 (ext_ip_376.int)> TLS connection for data connection established
(000066) 18.02.2025 18:29:35 - bck_usr376 (ext_ip_376.int)> 426 Connection closed; aborted transfer of "/disk_e/imgs/cb_376.dmp.tgz"


The time durations until the "Connection closed" occurs vary a lot. This probably rules out "keep alive" as being the issue. It's unclear which side is initiating the connection close. Other software like firewalls might be causing the disconnects, which would be worth a test.