Files disappear from destination after upload

[AdrianVergin]

Hi all

I just started using the program so please let me know if I've configured something wrong. I'm attempting to use this program as a way to synchronize my NAS which houses all my backups with a GoFlex disk which can be taken offsite. I've got a Ubuntu 14.04 VM with the sole purpose of transferring the data. FreeFileSync can see all the source files and can write to the destination. I can watch the files written on the destination drive with the file size slowly counting up.

The problem is that as soon as the file size reaches the same amount as what it is on the source drive, it disappears. FreeFileSync just happily goes on to the next file like nothing happened and there's nothing untoward in the log. I've checked to make sure it wasn't getting moved anywhere else and even the admin console of the GoFlex drive says the total used space is equal to the size of the current upload file (not the previous + current).

I don't even know where to start to diagnose this further - I'm just attempting to move the file using Windows to make sure this isn't a Ubuntu issue, then I'll probably use the portable Windows version of FreeFileSync to check it's not the program's fault. I can't imagine I'll gain any information from either of these tests, probably just more questions.

Hopefully someone's had a similar issue in the past and you can point toward a fix.

Many thanks
Adrian

[AdrianVergin]

Hello again

I thought I'd post my findings as they occur:

Copying files via Windows Explorer results in the copied file staying put in the destination folder
Copying files via Ubuntu Files directly results in the copied file staying put in the destination folder
Copying files via FreeFileSync results in the files existing only as long as the copy operation is active

I think I have a new theory though. Among the errors captured at the end of the the sync was a bunch of errors to do with setting modified time. Apparently the destination drive doesn't allow such a change. I think the problem might be that since I'm running FreeFileSync in Mirror mode it sees the copied files with incorrect modified dates and deletes them.

I tried setting the comparison variant to 'File Content' but comparing 2 files was going to take 7 hours. When complete there will be many files, most over 25GB and both on network drives... I want to run this fairly regularly, like once a day, so I can't spend a week comparing first each time. :)

I'm going to test further by creating a custom sync plan with the option "Where Left side is Newer: Do Nothing". I take it there's no way to set an "Ignore Time Shift: *:*" to just ignore modified time altogether?

Many thanks
Adrian

[AdrianVergin]

Ok, so I don't think this is working. Even with the new sync plan the file gets deleted after the copy operation.

This time, for some reason, it thought that the file on the source side was newer than the destination despite it being two months old. It moved copied the file over and as soon as it was finished, the file disappeared as before.

Here's the log (usernames redacted):

16:07:50 Warning The following folders are significantly different. Make sure you have selected the correct folders for synchronization.
/run/user/1000/gvfs/smb-share:port=139,server=tcm-backup.local,share=backup <->
/run/user/1000/gvfs/afp-volume:host=GoFlexHome.local,user=****,volume=****/GoFlex Home Public
16:07:50 Info Synchronizing folder pair: [Custom]
/run/user/1000/gvfs/smb-share:port=139,server=tcm-backup.local,share=backup
/run/user/1000/gvfs/afp-volume:host=GoFlexHome.local,user=****,volume=****/GoFlex Home Public
16:07:50 Info Updating file "/run/user/1000/gvfs/afp-volume:host=GoFlexHome.local,user=****,volume=****/GoFlex Home Public/AOMEI-Images/APPSERVER3-System/APPSERVER3-System.adi"
16:57:28 Error Cannot write modification time of "/run/user/1000/gvfs/afp-volume:host=GoFlexHome.local,user=****,volume=****/GoFlex Home Public/AOMEI-Images/APPSERVER3-System/APPSERVER3-System.adi".
Error Code 95: Operation not supported (futimens)
16:57:28 Info Creating file "/run/user/1000/gvfs/afp-volume:host=GoFlexHome.local,user=****,volume=****/GoFlex Home Public/AOMEI-Images/P002 Disk Backup/P002 Disk Backup.adi"
16:57:42 Error Synchronization stopped (by me)

I'm going to try one more time with the previous sync plan option change plus "Where Right side is Newer: Do Nothing" but I expect the result will be the same.

Any input will be greatly appreciated as I don't seem to be getting anywhere on my own.

Many thanks
Adrian

[AdrianVergin]

As predicted, the sync plan with two "Do Nothing" options set had the same effect as before. I was looking for anything else that might contribute to this weird problem and I thought it may be something to do with the way each share was mounted in Ubuntu.

The NAS is mounted as: smb://tcm-backup:139/backup/
The GoFlex is mounted as: afp://****@GoFlexHome.local/****/

I had no idea, maybe there was some incompatibility between the smb and afp protocols... nope, it seems to be this software. Just to be sure, I tried mounting the GoFlex a different way. It didn't want to accept authentication for SMB because Ubuntu requires a domain name and then rejects the credentials no matter what you put in. I used FTP instead which worked perfectly the first time. Weirdly, the transfer over FTP was about twice as fast. The SMB to AFP transfer moved at an average of 10.5MB/s and the SMB to FTP transfer moved at 21MB/s.

It didn't do much good though, because as soon as the file was uploaded it disappeared from the destination again.

Again, there wasn't anything to indicate the file was intentionally deleted in the log:

19:55:25 Warning Cannot set directory lock for "/run/user/1000/gvfs/ftp:host=10.0.0.26/GoFlex Home Public".
Cannot read file "/run/user/1000/gvfs/ftp:host=10.0.0.26/GoFlex Home Public/sync.ffs_lock".
unexpected end of stream
19:55:25 Warning The following folders are significantly different. Make sure you have selected the correct folders for synchronization.
/run/user/1000/gvfs/smb-share:port=139,server=tcm-backup.local,share=backup <->
/run/user/1000/gvfs/ftp:host=10.0.0.26/GoFlex Home Public
20:18:44 Error Cannot write modification time of "/run/user/1000/gvfs/ftp:host=10.0.0.26/GoFlex Home Public/AOMEI-Images/APPSERVER3-System/APPSERVER3-System.adi".
Error Code 95: Operation not supported (futimens)
20:26:43 Error Synchronization stopped (by me)

Since the time the file disappeared corresponds to the log entry for the modification time error I've got to assume it's something to do with whatever "futimens" is. I understand that modified time is kind of a fundamental part of this program but I wish there was a way to disable it for mirror operations... So if a file exists and is the same size, leave it alone. If the file size is different, overwrite the destination. If a file is deleted in source, delete it in destination. If a file doesn't exist in destination, add it. I'm mirroring large backups so each backup has a different file name. The only changes are that are made include backups of a certain age being deleted and replaced with new ones so I don't really care about modified dates.

Unless anyone else has any ideas I'm going to have to try using a different product. Hopefully this information can be used to improve this software or to help someone struggling with the same issue.

Many thanks
Adrian

[Zenju]

The problem simply is that the target device doesn't support setting of modification times (whether this is unexpected or a limitation of GoFlex is something to check). FFS copies files as a transaction: it either finishes successfully or does nothing at all (=> copying the modification times fails, so the temporary file is deleted and an error is shown).

[AdrianVergin]

and is there any way to not delete the temporary file? even if i have to recompile the source code...

[Zenju]

Generally it makes sense to show an error because synchronisation (at least when comparing by time and size) is not possible without valid modification times. But now with the new "compare by file size" variant, this may be something that FFS should support (and also for the binary comparison variant).

[AdrianVergin]

I agree that in probably 99% of cases the way in which FFS works is perfectly appropriate. Especially that incorrect modified times would create false comparison results in future sync operations. My problem seems to be that my device doesn't support changing modified times and creates the situation in which files get deleted as a result of the error.

Since I can't enable file modification change support on the target device (it's a "for home use only" product and their support staff aren't trained to deal with issues more complicated than setting a static IP address) there are only two available courses of action:
1. Modify the software to prevent it deleting temporary files when an error occurs
2. Change the direction of the migration in hopes that my other NAS supports changing modified dates

Would Option 1 be difficult for you guys to implement? I'm usually quite happy to jump in and start coding despite not having done it since I was a university student, but I'm dealing with my company's backups here and I don't really want to mess up and accidentally cause irreparable damage somehow...

In the meantime I'll stick with Option 2 which involves far less risk. I'm currently manually copying all of the files with an ETA of 24h to completion. The copied files should have drastically different created/modified dates so when I run a sync back to the original NAS it should start overwriting the old files. Let's hope once it does that they stay put.

[AdrianVergin]

This is unrelated to the issue but I found a utility that works and has a GUI front end. It's just a front end for the rsync command but it has all the options I need, like ignore time and only compare by filesize. I've included screenshots to demonstrate what options I'm using.

grsync-basic-options.jpg (89.66 KiB) Viewed 6983 times

It works somewhat slowly but that may be due to the fact that it's a VM with minimal resources and I've set it to compress data. At least the uploaded files don't get deleted though.

grsync-running.jpg (157.59 KiB) Viewed 6983 times

I like the interface for FFS better and would prefer to use it in the future if similar functionality was added.

Any chance of a premium version coming out with a scheduling feature? Like a run as a service, set schedule, enable email notifications type of thing...

[drghughes]

Scheduling is possible - see https://freefilesync.org/manual.php?topic=schedule-a-batch-job

Or maybe real time sync is more what you want: https://freefilesync.org/manual.php?topic=realtimesync

[AdrianVergin]

Thanks, I'll play around with the settings once I have the program working. I already saved a batch job but never got a chance to activate it because of the disappearing files issue.

I'm currently still copying the files over using that rsync application (7+ hours left) and afterwards I'll swap the destination of the backups so they go to the disk that doesn't accept modification time changes and I'll try setting up FFS to copy files from there to the NAS instead. Hopefully the NAS won't struggle so much with mod times.

I'll read through the real time sync documentation more thoroughly because it seems more appropriate. Do you know how real time sync would handle files currently being written? I believe the backup utility just creates a file and slowly inflates it with data so the file size is constantly changing. Would RealTimeSync try to copy the backup files as they're being written or would it try to read them and get an access denied error or something to that effect? If my backup utility wrote temporary files like FFS can then it would be easy to filter them out but I think it just creates the final filename right away.

[AdrianVergin]

Just wanted to post an update with the latest happenings. I've just finished re-initializing my NAS so I've got blank disks again. I tried using FFS to sync the data back from the drive that didn't allow me to do mod time updates to the NAS which I assumed would be more lenient and I'm sad to say it still didn't work.

For some reason, copying the files in the other direction using FFS was much slower (only around 2MB/s) but that's an unrelated problem. Either way, after waiting over an hour for one file, I got the same error, "unable to change modified time", and the file was deleted as a result.

I read somewhere that there's bound to be issues when copying between EXT3 and NTFS filesystems so maybe that's part of the cause... any ideas?

[Zenju]

Would Option 1 be difficult for you guys to implement?

No and as already mentioned above it makes sense. It's on the todo list.

Do you know how real time sync would handle files currently being written?

It doesn't, RTS just detects that files changed and starts the command line (usually a FFS sync job) after the specified idle time has passed.

[AdrianVergin]

Perfect, so RTS sounds exactly like what I need.

I obviously can't use the program until the functionality from the to-do list is added so have you got an ETA for this or is it just a case of waiting until "added option to not delete files on error" appears in the change log for a new version?

[Zenju]

Failure to set modification times will be treated as a warning instead of an error in the next FFS 9.0.

[AdrianVergin]

Legend. Thanks. I'll test with the new version as soon as it's released.