Timestamp issues with NTFS external drives

[travelid]

I use FreeFileSync every day; it's incredible!

I just wanted to point out a bug that occurs only when syncing to NTFS external disks (external HDDs and Flash-memory USB keys). If the external disk or USB key are formatted with FAT32, everything works perfectly. But not with NTFS.

Since v6.12 FreeFileSync, after a sync finishes, and I press the "Compare" button again (just to check if all went well), FreeFileSync keeps finding ~5% of the files it just synced as still needing updating. So I click "Synchronize" and let it update them again. But then after I click "Compare" for the 2nd time, it still finds those same files as needing updating. And those are mostly video, audio and PDF files that don't change at all. Maybe a time stamp issue? I'm on a Windows XP SP3 laptop with all updates installed.

Not only that but FreeFileSync is also now creating several empty .ffs_tmp files that have to be deleted manually after each sync, or by clicking "Synchronize" again. When that is done and I click "Compare" again, it finds a few new temp files of the same type to delete. FreeFileSync deletes a few and creates a few new temp files after each sync, which need to be deleted at the next sync. Those .ffs_tmp files are 0 KB in filesize, and carry the name of the one of the just synced video files. This only seems to be happening when I do syncs to a NTFS external disks.

Hope this is enough info to find the bug!

Thank you so much for creating FreeFileSync!

[Zenju]

> FreeFileSync keeps finding ~5% of the files it just synced as still needing updating

From the symptoms it could be a virus scanner writing hidden ADS data to the files just after FFS has copied them, thereby changing the modification time:
viewtopic.php?t=334

> several empty .ffs_tmp files that have to be deleted manually after each sync

Did you properly unmount the NTFS drive before pulling the plug?

[travelid]

Thank you for that link. I use ESET NOD32 v8 as Antivirus. I've never had this issue with FreeFileSync v6.11 or lower. (I've been using FreeFileSync for 3 years, and I've had the same version of ESET for more than a year).

The issues happen only when syncing to a NTFS disk and NTFS USB keys. My USB key was NTFS and the issues were there after every sync. As soon as I re-formatted my USB key to FAT32, the issues disappeared. But I cannot re-format my external HDDs to FAT32 because I have no place to temporarily put so much data.



"several empty .ffs_tmp files that have to be deleted manually after each sync"
Did you properly unmount the NTFS drive before pulling the plug?

-- This happens immediately after each sync. So unmounting is not the issue. Also, I always unmount drives properly before I unplug them.

Thank you very much!

[Zenju]

Can you simplify your scenario and reproduce the problem for syncing a single file only? In this case we could have a deeper look and you could provide me with a
Process Monitor (https://technet.microsoft.com/de-de/sysinternals/bb896645.aspx) trace file for analysis.

[platon91]

I have exactly the same problem after a recent update on fff. I am on linux arch/manjaro and got fff from AUR.

I have been using fff for ever, but after the latest update (6.15) when I sync to my external USB thumb drive that is NTFS formatted, the time stamp changes and the files that are copied on the receiving end get the time stamp from the time they got copied/synced and don't retain the original time stamp. That way if you do a compare just after you have done the sync, the exact same files that were just synced are reported as newer on the receiving end. It is the same behavior as running rsync without the -t option...

[needcoffee200]

I have the same problem too with 6.15 using Linux Mint 64 bit. Downgraded back to 6.14 and problem went away.

[gerinho]

Same problem in Arch Linux x64; in windows 8.1 x64 everything is fine for now

[Zenju]

> after the latest update (6.15) [...] the time stamp changes

This could be due to recent changes discussed in this thread:
viewtopic.php?t=387

Can you simplify your test case to copying a single file with FreeFilesync and provide a "strace" for it, as well as one for "cp --preserve=timestamps" doing the same operation?

[platon91]

Hi Zenju

Thanks for your reply. I did a test on a single file as requested and below are the link to the straces.

Here is the strace for the copy command with preserving the time stamps, which works:

[404, Invalid URL: https://onedrive.live.com/redir?resid=a5c93bba58033a99!2505&authkey=!AE7kuSpHq5bd_Wc&ithint=file%2Ctxt]

Here is the strace of FFS and I could confirm that when syncing/copying a single file it will get the time stamp of the sync time on the receiving end and not the original time stamp:

[404, Invalid URL: https://onedrive.live.com/redir?resid=a5c93bba58033a99!2506&authkey=!AEUU-g70ET6-hfw&ithint=file%2Ctxt]

The file I copied/synced is just called Test1.txt in folder Test.

Cheers

[Zenju]

Hi Zenju

Thanks for your reply. I did a test on a single file as requested and below are the link to the straces.

Here is the strace for the copy command with preserving the time stamps, which works:

[404, Invalid URL: https://onedrive.live.com/redir?resid=a5c93bba58033a99!2505&authkey=!AE7kuSpHq5bd_Wc&ithint=file%2Ctxt]

Here is the strace of FFS and I could confirm that when syncing/copying a single file it will get the time stamp of the sync time on the receiving end and not the original time stamp:

[404, Invalid URL: https://onedrive.live.com/redir?resid=a5c93bba58033a99!2506&authkey=!AEUU-g70ET6-hfw&ithint=file%2Ctxt]

The file I copied/synced is just called Test1.txt in folder Test.

Cheers

Thanks for the trace! I see a few small differences between the two, so let's test one after the other:

1. Retry with the official FFS 6.15 but disable menu -> tools -> options: fail-safe file copy. Then test if the time stamps are still set to current time in your test case.

2. Test with the following versions and see if your test case works or not:
[404, Invalid URL: https://freefilesync.org/FFS_Test1.tar.gz]
[404, Invalid URL: https://freefilesync.org/FFS_Test2.tar.gz]
[404, Invalid URL: https://freefilesync.org/FFS_Test3.tar.gz]

[platon91]

Thanks for looking into this!

Disabling fail safe copy in the official 6.15 didn't work.

Tested the various FFS versions you attached.

FFS1 WORKS, GREAT!!!
FFS2 didn't work
FFS3 didn't work

Cheers

[Zenju]

Looks good: version 1 is only a trivial change explicitly setting the file access time, which was (rightfully) omitted in the old version, but this seemed to trigger bugs in "no so well tested" system libraries.

Here's the new version:
[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.16_beta_Ubuntu_14.10_64-bit.tar.gz]

[platon91]

Great, thanks a lot for the new version. Will be using this one till it becomes the new official one.

Cheers

[travelid]

Can you simplify your scenario and reproduce the problem for syncing a single file only? In this case we could have a deeper look and you could provide me with a
Process Monitor (https://technet.microsoft.com/de-de/sysinternals/bb896645.aspx) trace file for analysis.

I see the issue has been resolved for Linux, which is great!
for Windows: I have too many things to work on for now, so unfortunately it looks like I simply won't have the time to learn how to use Process Monitor and provide a trace file for sync on NTSF external disks.

But it's also possible that my specific issue is caused by Comodo Firewall & HIPS (Defense+) v8.1.0.4426 (latest version). Even though I do use ESET NOD32 as Antivirus, Comodo seems to be the best professional firewall and HIPS a person can find. Looks like they didn't fix the issue with adding hidden ADS data to files even though I have no Comodo Antivirus, so Comodo is not supposed to be scanning anything, so I do not understand why it would add any ADS data to some of my files...

Thank you so much for everything Zenju!

[Zenju]

I see the issue has been resolved for Linux, which is great!
for Windows: I have too many things to work on for now, so unfortunately it looks like I simply won't have the time to learn how to use Process Monitor and provide a trace file for sync on NTSF external disks.

But it's also possible that my specific issue is caused by Comodo Firewall & HIPS (Defense+) v8.1.0.4426 (latest version). Even though I do use ESET NOD32 as Antivirus, Comodo seems to be the best professional firewall and HIPS a person can find. Looks like they didn't fix the issue with adding hidden ADS data to files even though I have no Comodo Antivirus, so Comodo is not supposed to be scanning anything, so I do not understand why it would add any ADS data to some of my files...

Thank you so much for everything Zenju!

If you're using Comodo v8, then we don't need to analyze further anyway. I've confirmed myself with Process Monitor that v8 is writing ADS to some files, but not to others during file creation, which breaks synchronization fundamentally.
I was using Comodo myself until v7, but the time-stamp issue was serious enough to finally have to uninstall it.

[rangergr]

Looks good: version 1 is only a trivial change explicitly setting the file access time, which was (rightfully) omitted in the old version, but this seemed to trigger bugs in "no so well tested" system libraries.

Here's the new version:
[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.16_beta_Ubuntu_14.10_64-bit.tar.gz]

Hi Zenju. I have same problem with timestamps after upgrading to 6.15.
I am using arch linux, can you give the source for this beta version so I can build a package?

[Zenju]

Hi Zenju. I have same problem with timestamps after upgrading to 6.15.
I am using arch linux, can you give the source for this beta version so I can build a package?

Here you go:
[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.15_bugfix_Source.zip]

[rangergr]

Here you go:
[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.15_bugfix_Source.zip]

Thanks

I confirm this version works. I had to overwrite the files I had sync with previous version in order to have the correct timestamps.
After that it works like before

Keep up the great work