Multi-threading for file transfer

[Pako]

I realize this qualifies for a feature request, but I would like to see some discussion on the topic as opposed to hanging lonely in the requests section.

For me and the people I know of, FFS would be the ideal solution to our needs if it had two more features:
1. Virtual file system to handle syncing to cloud (amazon s3, google, microsoft azure etc.)
2. Multi-threaded file transfer so that several files would be transferred simultaneously.

I understand that the implementation of the first is too mush effort. In addition, it would make the product too useful to remain open source so we may loose it if such feature gets integrated in FFS.

But what about the multi-threading? I am on the sysadmin side so I cannot estimate the amount of work required for that. I have tested multi-threaded file transfer with powershell (20 threads) and managed to reduce the file copy time up to 4 times compared to FFS. This is big. We sync from 4 locations over internet nearly 20GB per week, you can imagine how much time is spent just on waiting for sync jobs to complete.

So I would like to hear what other folks think, do you actually need more speed? Do you sync locally or over internet, what is your experience?

[mikeloeven]

i definitely believe this is needed as someone who synchronizes over a VPN the process is painfully slow because it needs to wait for the overhead on each file reducing a broadband connection to dialup simply because it cannot execute multiple transfers simultaneously.

it should never ever EVER take 3 days to sync 2 gigs of data over a broadband connection

multi threading opens the door to configuring multiple simultaneous transfers allowing large quantities of tiny files to be done at the same time allowing the bandwidth to be utilized fully

this is a feature of any ftp client and would be useful for synching over vpns

[Pako]

i definitely believe this is needed as someone who synchronizes over a VPN the process is painfully slow because it needs to wait for the overhead on each file reducing a broadband connection to dialup simply because it cannot execute multiple transfers simultaneously.

it should never ever EVER take 3 days to sync 2 gigs of data over a broadband connection

multi threading opens the door to configuring multiple simultaneous transfers allowing large quantities of tiny files to be done at the same time allowing the bandwidth to be utilized fully

this is a feature of any ftp client and would be useful for synching over vpns

You have some serious bottleneck. My residential clients can dl/upload 1GB for about 1hr/2hrs over vpn with FFS. Oddly, robocopy's remote-to-local speed matches FFS no matter how many threads you set. Copying local-to-remote, though, is noticeably faster (don't have my logs on this PC).

[mikeloeven]

well considering that a linear read or linear write over the network for me at least normally goes at between 256 and 512 depending on the line. i belive the bottle neck is the overhead of the sync program where it is writing temp file checking temp file deleting original and renaming. all of this seems to be done in sequence so when you have say 1000 files under 10k it will take a very long time where as synching 5 large files will go quite fast. adding multithreading will allow the program to do multiple transfers at the same time. with a 512 upload you could theoretically perform this operation on a large chunk of small files at the same time using the full bandwidth and making it go much much faster.

[dabbatelli]

I realize this qualifies for a feature request, but I would like to see some discussion on the topic as opposed to hanging lonely in the requests section.

For me and the people I know of, FFS would be the ideal solution to our needs if it had two more features:
1. Virtual file system to handle syncing to cloud (amazon s3, google, microsoft azure etc.)
2. Multi-threaded file transfer so that several files would be transferred simultaneously.

I understand that the implementation of the first is too mush effort. In addition, it would make the product too useful to remain open source so we may loose it if such feature gets integrated in FFS.

But what about the multi-threading? I am on the sysadmin side so I cannot estimate the amount of work required for that. I have tested multi-threaded file transfer with powershell (20 threads) and managed to reduce the file copy time up to 4 times compared to FFS. This is big. We sync from 4 locations over internet nearly 20GB per week, you can imagine how much time is spent just on waiting for sync jobs to complete.

So I would like to hear what other folks think, do you actually need more speed? Do you sync locally or over internet, what is your experience?

I'm trying to sync 1TB of data over VPN or FTP, and somehow the transfer speed for a single file is much slower than the one I get transferring several files at once.
A multithreaded Freefilesync would be exactly what I need!

[philipgoddard]

I've just discovered the SFTP functionality in FFS, which latter I've been using for some time now for local sync jobs (an excellent program). Having set up sync profiles for all my websites, I then found that there is no multithreading, so for the moment I'll continue using Beyond Compare for my SFTP syncs. I definitely require multiple connections for SFTP / FTP, and an option to choose the maximum number of simultaneous connections (important!). My hosting company blocks any IP that tries to make more than three simultaneous connections, and presumably, for security reasons, others would have a similar policy.

[Zenju]

Just for the record: multithreaded file copy (and comparison) has been added in FFS 10.0!