Hello,
I'm using FFS on Ubuntu 22.04.3 LTS (Desktop). FFS is set up as Mirror from an SFTP server (remote) to this machine. Remote machine is a server. Mostly everything works fine - I'm just looking for a snapshot backup. However, log files aren't syncing as their size changes between Compare and Sync.
I've read around the forums and I can see that this could be a problem in some situations. However, in this one, it means the file isn't copied and an error is thrown.
Ideally I would like to ignore the size mismatch (maybe show a warning) and have the file as it comes when downloaded. Currently an error and no file saved in the mirror.
I've tried a couple of options suggested on similar threads without luck. Being log files they will change every few seconds at most so it's unlikely to get a compare and sync done before they change.
Any help appreciated.
thanks,
Unexpected size of data stream
- Posts: 6
- Joined: 28 Aug 2023
- Posts: 6
- Joined: 28 Aug 2023
Just a little bump here.
If this is a dead end with this software, let me know and I'll find a different solution.
thanks,
If this is a dead end with this software, let me know and I'll find a different solution.
thanks,
Let me see if I understand correctly. You are trying to sync a bunch of files from SFTP/Linux down to your workstation And some of the files that you're trying to get are server log files that are constantly being appended to every second of the day and are actually physically changing from the time that you click on the compare button to the sync button?
And, the result is no log file being mirrored down to your work station?
I'm only an end user and by no means an expert with this software however I am familiar with Linux and the continuously changing log files.
I can see and understand where this might be a problem with FFS.
one solution would be to have a script run on the server copying the log files to a static folder that FFS can mirror to your workstation.
Another would be to pursue some sort of server log file copying utility app that I don't know the name of.
My last thought for grabbing a copy of an Instance in time of the log files would be to use a different app.
Just for the log files you might consider using an application that doesn't have a separate analyze and synch function but rather a single function that just grabs the files as they are when they get to them.
I would suggest giving a program that runs on Linux Windows and Mac as a command line utility called Rclone a try. It functions similarly to rsync only it has the built in intelligence to know how to talk to cloud servers including SFTP servers. It is open source and free to use. Community support is provided just as it is here.
You would need to create a config for your SFTP server and then you could run a command like this:
This is the Windows syntax. you can adapt it easily for a Linux file system. Instead of the local drive letter it would be a local folder such as /home/joe/serverlogs
or "rclone copy remote-server:/var/log /home/joe/serverlogs"
You can compare the results of that copy command to the "sync" command and see which one works better.
"rclone copy" or "rclone sync"
They should both work, I think.
On many systems, those files are in /var/log. You need to make sure you log into the server with a user ID that has read permissions to that directory.
or If your local work station is Linux use your local folder syntax instead of the C drive for the destination.
Personally, I've never tried copying active log files using this app but it's the best option that I can think of that can do a one step file copy from an SFTP server down to your local PC.
Rclone can be tricky to use for a novice, so let me know if you get stuck somewhere.
Run "rclone config" to create the config for your SFTP server.
And, the result is no log file being mirrored down to your work station?
I'm only an end user and by no means an expert with this software however I am familiar with Linux and the continuously changing log files.
I can see and understand where this might be a problem with FFS.
one solution would be to have a script run on the server copying the log files to a static folder that FFS can mirror to your workstation.
Another would be to pursue some sort of server log file copying utility app that I don't know the name of.
My last thought for grabbing a copy of an Instance in time of the log files would be to use a different app.
Just for the log files you might consider using an application that doesn't have a separate analyze and synch function but rather a single function that just grabs the files as they are when they get to them.
I would suggest giving a program that runs on Linux Windows and Mac as a command line utility called Rclone a try. It functions similarly to rsync only it has the built in intelligence to know how to talk to cloud servers including SFTP servers. It is open source and free to use. Community support is provided just as it is here.
You would need to create a config for your SFTP server and then you could run a command like this:
This is the Windows syntax. you can adapt it easily for a Linux file system. Instead of the local drive letter it would be a local folder such as /home/joe/serverlogs
rclone copy remote-server:/log/file/directory c:\users\joe\logs
You can compare the results of that copy command to the "sync" command and see which one works better.
"rclone copy" or "rclone sync"
They should both work, I think.
On many systems, those files are in /var/log. You need to make sure you log into the server with a user ID that has read permissions to that directory.
or If your local work station is Linux use your local folder syntax instead of the C drive for the destination.
Personally, I've never tried copying active log files using this app but it's the best option that I can think of that can do a one step file copy from an SFTP server down to your local PC.
Rclone can be tricky to use for a novice, so let me know if you get stuck somewhere.
Run "rclone config" to create the config for your SFTP server.
- Posts: 6
- Joined: 28 Aug 2023
Hi LeoW,
Thanks for the detailed reply!
Whilst I referred to log files - they are the clearest example of a file that changes between analysing and syncing - this happens with any file that changes between the two processes. Which means for a live directory, say one being used by other peope on a shared folder FFS wouldn't work as there would constantly be replication errors and incomplete backups.
I hoped there would be a way around this - an option to ignore changes between analyse and sync ideally.
I was hoping for a quick and easy GUI based solution. rclone is great and I'll possibly go that way once I find a little time to work on this more.
I'll stay subscribed to this topic and maybe there will be a change in the future. I'll also add a feature request as I can't be the first person to run up against this and a checkbox to ignore size mismatch should be straight forward for the devs.
thanks again,
Thanks for the detailed reply!
Whilst I referred to log files - they are the clearest example of a file that changes between analysing and syncing - this happens with any file that changes between the two processes. Which means for a live directory, say one being used by other peope on a shared folder FFS wouldn't work as there would constantly be replication errors and incomplete backups.
I hoped there would be a way around this - an option to ignore changes between analyse and sync ideally.
I was hoping for a quick and easy GUI based solution. rclone is great and I'll possibly go that way once I find a little time to work on this more.
I'll stay subscribed to this topic and maybe there will be a change in the future. I'll also add a feature request as I can't be the first person to run up against this and a checkbox to ignore size mismatch should be straight forward for the devs.
thanks again,
- Posts: 1037
- Joined: 8 May 2006
Wouldn't Tools | Options -> Copy locked files (VCSS) handle that?a file that changes between analyzing and syncing - this happens with any file that changes between the two processes. Which means for a live directory, say one being used by other people on a shared folder FFS wouldn't work as there would constantly be replication errors and incomplete backups
Things should be accurate as of the point where the program (& VCSS) was started?
Oops, you're on Linux, so no, that would not help.
Yah, I doub't a GUI based tool would work as well as rclone can.Hi LeoW,
Thanks for the detailed reply!
I was hoping for a quick and easy GUI based solution. rclone is great and I'll possibly go that way once I find a little time to work on this more.
maeterlinck, 06 Sep 2023, 08:02
Rclone does has a new GUI user interface for creating/editing configs, and mounting drives.
Use this command to launch a browser based GUI for Rclone:
rclone rcd --rc-web-gui
Also, if you use the "--interactive" flag, you will get lots of good feedback while it runs.
