FTP Timeout when sync bigger files

[dklev]

When I sync to my (standard) FTP server eveything works like a charm - except if there is a bigger file to by synced (>200MByte). Then I always get a timeout error.

When doing same with my FTP client, there is no error.

19:34:25 Error File "ftp://user@server:77/Data8TB/DataSyncDavidLocalWork/!!!To Upload To Vimeo!/2023-11-15d Closer Gesprächsanalyse Sales Call Sandra mit Toni (Teil 2).mp4" can not be written.
CURLE_OPERATION_TIMEDOUT: FTP response timeout
150 Opening BINARY mode data connection for '/Data8TB/DataSyncDavidLocalWork/!!!To Upload To Vimeo!/2023-11-15d Closer Gesprächsanalyse Sales Call Sandra mit Toni (Teil 2).mp4'. [curl_easy_perform]

Any idea how to fix that?

Thank you very much
David

PS: I already found this, but I guess it doesn't help me, as I am using newst version already: viewtopic.php?t=6928&hilit=CURLE_OPERATION_TIMEDOUT%3A+FTP+response+timeout#p23065

[therube]

What version of FFS?
(I would not think 200MB to be particularly "big".)

Might also be related, FTPS error CURLE_PARTIAL_FILE and Google error CURLE_OPERATION_TIMEDOUT?

Any idea of connection/transfer speed?

When doing same with my FTP client, there is no error.

Hmm...

[dklev]

Thank you for your help.

Version is 13.1 64bit 23.10.2023 Donation

Of course not, 200 MB is not that much, but with smaller files this timeout doesn't happen. Transfer speed is 15 MBit/s, not that bad. A 200MB file is uploaded within 2,5 minutes, but after uploading nothing happens the next minute, then the timeout occurs.

Here I made a screen recording:
https://youtu.be/JCsWt373NYA

I tried to exclude <Program Files>\FreeFileSync\bin folder from my antivirus, but that didn't change anything.

[xCSxXenon]

So the file transfers fine, but whatever finishing operations that are being attempted aren't going through. I hate to ask the simple questions, but is the server up-to-date and has it been power cycled? What happens if you try copying that file over manually?

[dklev]

You are right, transfer was completed successfully, but anyway, then "something" times out.

Server is most current version, and with ftp client everything works fine, even with a list of 3GByte files.

What can it be? What is FreeFileSync doing or waiting after transferring the file successfully? (Normally there is a rename happening, but when I switch off "Fail-safe file copy" not. Video shows the behaviour of "Fail-safe file copy" off. This setting does not solve my problem)

[Zenju]

FTP uses two connections, one to transfer the data, the other to send control commands. In your video it almost looks as if the control connection timed out. This used to be an issue in the past, but should be fixed:
viewtopic.php?t=6928

FFS set's CURLOPT_TCP_KEEPALIVE which defaults to sending keepalive every 60 secs.

On your server, are there any settings regarding idle time after which the server starts to close connections?

Also for the record, what is your setting for "access timeout" in FreeFileSync?

For testing here's a beta version that sends keepalive every 30 seconds:
https://www.mediafire.com/file/qqr8w4gi13zvjfc/FreeFileSync_13.2_%255BBeta%255D_Windows_Setup%25284%2529.exe

[dklev]

Thank you Zenju and all !!

As my FTP server is a "FritzBox" Router, I can not make settings for FTP server except users and directories.

My setting for access timeout is ... Hmmm, I can not find setting for "access timeout" in FreeFileSync somehow... (I checked Tools/Options and the 3 comparison/sync settings...)

But: I updated to 13.2 (stable, not your beta), now the error message changed (but also I am at a new location with better internet and lower ping if that's helpful somehow).

Here the new error message:

Cannot write file "ftp://david@server:77/Data8TB/DataSyncDavidLocalWork/!!!To Upload To Vimeo!/Upl/2023-11-14b Closer Gesprächsanalyse Sales Call Sandra mit Toni (Teil 1)-5f91.ffs_tmp".

CURLE_OPERATION_TIMEDOUT: Operation too slow. Less than 1 bytes/sec transferred the last 60 seconds
150 Opening BINARY mode data connection for '/Data8TB/DataSyncDavidLocalWork/!!!To Upload To Vimeo!/Upl/2023-11-14b Closer Gesprächsanalyse Sales Call Sandra mit Toni (Teil 1)-5f91.ffs_tmp'. [curl_easy_perform]

And the problem "Less than 1 bytes/sec" I saw already somewhere in a thread with Google Drive if I remember right...

Looks as if we get closer now. :-)

[Plerry]

And how did the beta version provided above perform?
Zenju does not provide those for fun.

[dklev]

Thank you Zenju,

I needed a while because I was at home already where connection is too fast to reproduce the problem again. Now I have slower internet again, the problem is again there with 13.2, now I installed your 13.2 beta version to see if your new timeout fixes my problem. Aaannnd?

No, same problem, after 100% upload FFS waits another 60 seconds, and then again

Cannot write file "ftp://server:77/Data8TB/DataSyncDavidLocalWork/!!!To Upload To Vimeo!/2023-12-05b Closer gemeinsame Übungen Schmerz und Messer-120a.ffs_tmp".

CURLE_OPERATION_TIMEDOUT: FTP response timeout
150 Opening BINARY mode data connection for '/Data8TB/DataSyncDavidLocalWork/!!!To Upload To Vimeo!/2023-12-05b Closer gemeinsame Übungen Schmerz und Messer-120a.ffs_tmp'. [curl_easy_perform]

If you like you can send me an email. Then I can send you my credentuials and you can reproduce it with my FTP- Server.

Thank you for your cool help and effort you already did to help me to get it running again.

David

[Zenju]

If you like you can send me an email. Then I can send you my credentuials and you can reproduce it with my FTP- Server.

Yes, if I could reproduce the issue from my end, chances are much higher to find out what this time out is really about.

[dklev]

Great Zenju, then let's do it. Unfortunately I don't know your email and I don't find an option to send a private message to you. You can write me to dkdk9000@googlemail.com

[Plerry]

Zenju's E-mail address can be found at the bottom of the FFS Home page.

[Zenju]

Thanks dklev! I've made a few tests and observations:

1. the timeout of the FTP control connection occurs for uploads that take longer than 120 seconds

2. it's the FritzBox FTP server that silently terminates the FTP control connection at this time, but doesn't stop the FTP data connection from still uploading the file!

3. "TCP keep alives" that are supposed to keep the control connection busy during long idle times have no effect on this server

4. The same issue occurs when testing FileZilla, with the difference that FileZilla does not delete the uploaded file after the control connection is closed unexpectedly. FreeFileSync OTOH always cleans up the unsuccessfully uploaded file upon encountering an error, because it has to be considered at least "truncated" if not "corrupted".

5. when "explicit TLS" is enabled, the behavior is different for FreeFileSync and FileZilla:
a) FileZilla:

421 Timeout (120 seconds): closing control connection.
GnuTLS-Fehler -110 in gnutls_record_recv: The TLS connection was non-properly terminated.
ECONNABORTED

=> basically same as without TLS: control connection fails to respond, and FileZilla doesn't clean up the file despite this, which incidentally happens to be uploaded completely.

b) FreeFileSync: the file upload just works!!! I don't know why that is.

So to sum up:
I) The FritzBox FTP server is broken: The FTP control connection is not kept open by "TCP keep alives". If you want to do something good, you may consider spending some of your time to report this issue to AVM.

II) Setting "Explicit TLS" in FreeFileSync not only makes the FTP connection secure by encrypting it, but also works around the time out issue, for whatever reason.

[dklev]

Zenju!! You are right: After activating Explicit TLS it also works on my side immediately. I created a support ticket for AVM, too.

Thank you for your help and workaround!

David

[mandrk]

Problem can be solved if you set/use sFTP not FTP for bigger files.