OS X - Upgraded to FFS 6.13, now certain files being sync'd over and over again even when FFS recognizes they are the same...

Posts: 73 · htismaqe 22 Jan 2015, 05:18

Subject says it all. Didn't have a problem with 6.12 but 6.13 is repeatedly syncing the same files over and over again. Strangely, it isn't all files.

Posts: 7211 · Zenju 22 Jan 2015, 12:28

What types of disks/volumes and file systems are you syncing? Does this happen always or only in some cases, e.g. not when syncing on the local hard disk?

Posts: 73 · htismaqe 22 Jan 2015, 14:25

I typed a response, hit "post" and it disappeared. So if you end up with duplicates, I'm sorry. :)

I'm mirroring personal folders on an HFS+ volume (my Mac's local HDD) to a NAS using SMB (RTS with AFP is still broken in 6.13 calling FFS every 20 seconds even though it doesn't identify anything to be synced).

I did figure something out - the files in question all have temporary copies (files that start with dot underscore, e.g. note.txt and ._note.txt) on the NAS. FFS identifies them as needing to be deleted during a Compare but they aren't deleted. Interestingly enough, FFS continually identifies the actual files on the NAS as being newer, so it's trying to overwrite the NAS copies with local copies over and over and over again.

Posts: 7211 · Zenju 22 Jan 2015, 14:58

I'm not sure if you are seeing multiple problems (files not deleted???), but let's focus on one of them first and solve one after the other:

As I understand there are certain pairs of files, where one file is existing locally and its counterpart on your NAS where the NAS file is allegedly updated during sync but the next comparison still shows the files with different dates. The modification time of the file on the NAS is not what it should be after sync, but it matches what is also shown in Finder. So it seems as if the problem is that the modification time on the NAS is not set correctly during sync.

If this is the case, then I have two versions to test for you. Let me know which one works or if both work (meaning: they correctly sync file modification times on the NAS):

[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.14_beta_1_Mac_OS_X_64-bit.zip]
[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.14_beta_2_Mac_OS_X_64-bit.zip]

Posts: 73 · htismaqe 22 Jan 2015, 15:02

I went ahead and added

*/.*

to my filters to prevent it from trying to sync any of the temp files. That should allow us to focus on the time stamps.

I am installing beta 1 now.

Posts: 73 · htismaqe 22 Jan 2015, 15:48

With beta 1, the last modified date is carried over. However, the created date is incorrect - it appears to be resetting the created date to the last modified date.

I used two files as tests.

Original file #1:
Created: 1/20/15, 1:44PM
Modified: Today, 8:30AM

On NAS, after sync:
Created: Today, 8:30AM
Modified: Today, 8:30AM

Original file #2:
Created: 1/31/13, 10:17AM
Modified: 12/31/14

On NAS, after sync:
Created: 12/31/14
Modified: 12/31/14

Posts: 73 · htismaqe 22 Jan 2015, 15:52

Beta 2 appears to be setting the created date to the last modified date as well. In addition, it is setting the last modified date to the time the file is copied.

Original file #1:
Created: 1/20/15, 1:44PM
Modified: Today, 8:30AM

On NAS, after sync:
Created: Today, 8:30AM
Modified: Today, 9:49AM

Original file #2:
Created: 1/31/13, 10:17AM
Modified: 12/31/14

On NAS, after sync:
Created: 12/31/14
Modified: Today, 9:49AM

Posts: 73 · htismaqe 22 Jan 2015, 15:55

For the sake of comparison, FFS release 6.13 shows the same behavior as Beta 2.

Original file #1:
Created: 1/20/15, 1:44PM
Modified: Today, 8:30AM

On NAS, after sync:
Created: Today, 8:30AM
Modified: Today, 9:53AM

Original file #2:
Created: 1/31/13, 10:17AM
Modified: 12/31/14

On NAS, after sync:
Created: 12/31/14
Modified: Today, 9:53AM

Posts: 7211 · Zenju 22 Jan 2015, 15:58

Nice representation of results, very useful!

> For the sake of comparison, FFS release 6.13 shows the same behavior as Beta 2.

And beta 1 the same as FFS 6.12? Can you confirm? I think the problem with the create date is not new.

Posts: 73 · htismaqe 22 Jan 2015, 16:28

Nice representation of results, very useful!

> For the sake of comparison, FFS release 6.13 shows the same behavior as Beta 2.

And beta 1 the same as FFS 6.12? Can you confirm? I think the problem with the create date is not new.Zenju

Thanks, I try! I am a technologist but my specialty is network and compute infrastructure, certainly not applications/programming. :)

As you suspected, FFS 6.12 behaves identically to beta 1.

Original file #1:
Created: 1/20/15, 1:44PM
Modified: Today, 8:30AM

On NAS, after sync:
Created: Today, 8:30AM
Modified: Today, 8:30AM

Original file #2:
Created: 1/31/13, 10:17AM
Modified: 12/31/14

On NAS, after sync:
Created: 12/31/14
Modified: 12/31/14

Posts: 73 · htismaqe 22 Jan 2015, 16:33

By the way, I just realized why only certain files are affected. All of the files being sync'd repeatedly were either created or modified after 6.13 was installed and thus sync'd with the wrong timestamps. Files sync'd prior to 6.13 have the correct modified date, so FFS isn't identifying the NAS copies as being newer.

Posts: 7211 · Zenju 22 Jan 2015, 17:51

Thanks for your tests! The problems with setting modification times are caused by an old bug in Samba code. Up to FFS 6.12, OS X was sharing the file copy routine with Linux and had a work around implemented for this bug. In 6.13 I've introduced a native file copy algorithm in order to support OS X-specific features like ACLs and extented attributes, but was not sure the workaround was needed. Obviously the issue is shared by OS X and Linux as your tests have confirmed.
Therefore I've patched the copy routine accordingly. Here's the fixed version:

[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.14_beta_Mac_OS_X_64-bit.zip]

Posts: 73 · htismaqe 22 Jan 2015, 17:54

I wonder if this problem would exist then when using AFP instead of SMB? (Of course, that re-introduces the other issue...)

Posts: 73 · htismaqe 22 Jan 2015, 19:29

I also now know why the Sync process is creating files that start with "dot underscore".

All of the main files associated with them have extended attributes (denoted by the @ in -rw-rw-rw-@), namely some form of metadata associated with the file.

I have a feeling this is also related to the recent changes in the copy routine, as I never noticed metadata being preserved before. In this case, I have a filter in place that should prevent the copying of any file starting with a period. This means the "dot underscore" files are being CREATED on the NAS at the time the file with extended attributes is copied. I further validated this by verifying that the "dot underscore" files do not exist in the local directory prior to sync.

EDIT: I figured this out. It's related to SMB. The "dot underscore" files are the resource forks of the main files. HFS+ stores all extended attributes, like Finder info, in the main file but in order to maintain compatibility with Linux/Unix NAS devices, Apple's implementation of SMB splits the resource fork into a separate file starting with "dot underscore".

http://support.grouplogic.com/?p=1496

Posts: 73 · htismaqe 22 Jan 2015, 19:35

Thanks for your tests! The problems with setting modification times are caused by an old bug in Samba code. Up to FFS 6.12, OS X was sharing the file copy routine with Linux and had a work around implemented for this bug. In 6.13 I've introduced a native file copy algorithm in order to support OS X-specific features like ACLs and extented attributes, but was not sure the workaround was needed. Obviously the issue is shared by OS X and Linux as your tests have confirmed.
Therefore I've patched the copy routine accordingly. Here's the fixed version:

[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.14_beta_Mac_OS_X_64-bit.zip]Zenju

Bad news. I'm using the beta 6.14 you just posted and it's not working. It's exhibiting the 6.13 behavior. Did you post the wrong one by chance?

Posts: 73 · htismaqe 22 Jan 2015, 20:56

FWIW, the behavior of beta 1 and 6.12 are not unique to FFS. If I do a copy from within Finder using SMB, the modified date of the file is preserved but the created date is changed to the modified data.

If I use AFP instead of SMB, both the created and modified timestamps are preserved correctly. I will not be using SMB going forward. I'd rather live with RTS calling FFS every 20 seconds and know that my files are being copied correctly, with timestamps and extended attributes intact.

With that in mind, which version of the application should I use going forward?

EDIT: I just tried a quick sync with the last 6.14 beta you posted - using AFP <s>the timestamps are correctly preserved</s> FFS now exhibits a completely new behavior. The modified timestamps are preserved correctly. However the created timestamp is reset to the time at which the file was sync'd.

Original file #1:
Created: 1/20/15, 1:44PM
Modified: Today, 8:30AM

On NAS, after sync:
Created: Today, 2:59PM
Modified: Today, 8:30AM

Original file #2:
Created: 1/31/13, 10:17AM
Modified: 12/31/14

On NAS, after sync:
Created: Today, 2:59PM
Modified: 12/31/14

Posts: 7211 · Zenju 23 Jan 2015, 12:10

Bad news. I'm using the beta 6.14 you just posted and it's not working. It's exhibiting the 6.13 behavior. Did you post the wrong one by chance?htismaqe

I had to update the 6.14 beta once to remove the old code setting the modtime that was not needed anymore. Setting the modtime twice shouldn't make any difference, but maybe this is the problem here. So there are two versions of the beta 6.14:

I): = "beta1" behavior + FFS 6.13 setting of modtime
II): should be equivalent to "beta1" from above

Perhaps you were using the first version of beta14? What time stampe does the "FreeFileSync" binary file have that you used in tests?

Posts: 73 · htismaqe 23 Jan 2015, 12:28

I had to update the 6.14 beta once to remove the old code setting the modtime that was not needed anymore. Setting the modtime twice shouldn't make any difference, but maybe this is the problem here. So there are two versions of the beta 6.14:

I): = "beta1" behavior + FFS 6.13 setting of modtime
II): should be equivalent to "beta1" from above

Perhaps you were using the first version of beta14? What time stampe does the "FreeFileSync" binary file have that you used in tests?Zenju

I just downloaded it from the link again. The timestamp is yesterday at 6:57pm central US time, which is 5 hours after I posted it wasn't working so you've updated it, yes.

Here is the behavior now using AFP for the network protocol:

Original file #1:
Created: 1/20/15, 1:44PM
Modified: Yesterday, 11:41PM

On NAS, after sync:
Created: Today, 6:23AM
Modified: Yesterday, 11:41PM

So the modified timestamp is correct but the created time stamp is being reset to the time the file is copied. Is there any way to not have that happen? If I copy the file via Finder or via the CLI, the created timestamp remains correct.

Posts: 7211 · Zenju 23 Jan 2015, 12:38

FWIW, the behavior of beta 1 and 6.12 are not unique to FFS. If I do a copy from within Finder using SMB, the modified date of the file is preserved but the created date is changed to the modified data.

If I use AFP instead of SMB, both the created and modified timestamps are preserved correctly. I will not be using SMB going forward. I'd rather live with RTS calling FFS every 20 seconds and know that my files are being copied correctly, with timestamps and extended attributes intact.

With that in mind, which version of the application should I use going forward?

EDIT: I just tried a quick sync with the last 6.14 beta you posted - using AFP <s>the timestamps are correctly preserved</s> FFS now exhibits a completely new behavior. The modified timestamps are preserved correctly. However the created timestamp is reset to the time at which the file was sync'd.

Original file #1:
Created: 1/20/15, 1:44PM
Modified: Today, 8:30AM

On NAS, after sync:
Created: Today, 2:59PM
Modified: Today, 8:30AM

Original file #2:
Created: 1/31/13, 10:17AM
Modified: 12/31/14

On NAS, after sync:
Created: Today, 2:59PM
Modified: 12/31/14htismaqe

Currently FFS does nothing with regards to file creation times, so I'm surprised there is so much variation going on. Usually I'd expect create time to always be "now" or equal to mod time. In practice however create time is not supported by all file systems and is equal to ctime in these cases which is inode modification time. This may explain your results.

Posts: 7211 · Zenju 23 Jan 2015, 12:42

I just downloaded it from the link again. The timestamp is yesterday at 6:57pm central US time, which is 5 hours after I posted it wasn't working so you've updated it, yes.

Here is the behavior now using AFP for the network protocol:

Original file #1:
Created: 1/20/15, 1:44PM
Modified: Yesterday, 11:41PM

On NAS, after sync:
Created: Today, 6:23AM
Modified: Yesterday, 11:41PM

So the modified timestamp is correct but the created time stamp is being reset to the time the file is copied. Is there any way to not have that happen? If I copy the file via Finder or via the CLI, the created timestamp remains correct.htismaqe

> Here is the behavior now using AFP for the network protocol

Could you test this with SMB? Since I expect a Samba bug to be the cause of the original issues there probably was never any problem for AFP in any of the beta versions or the releases.

> created time stamp is being reset

I'll see if there is a way to preserve the creation time on OS X.

Posts: 73 · htismaqe 23 Jan 2015, 12:53

I just tested it with both SMB and AFP.

A file copy via the Finder preserves both create time and modified time after copying to the NAS.

With FFS, the create time is overwritten in both cases. With AFP, the create time is set to the time when the file is copied. With SMB, the create time is set to the modified time.

There's not any way to mimic the native file copy routine?

Posts: 73 · htismaqe 23 Jan 2015, 12:56

> Here is the behavior now using AFP for the network protocol

Could you test this with SMB? Since I expect a Samba bug to be the cause of the original issues there probably was never any problem for AFP in any of the beta versions or the releases.

> created time stamp is being reset

I'll see if there is a way to preserve the creation time on OS X.Zenju

Sorry, we have two conversations going now inside this thread. My fault! :) I will keep all responses here from now on.

As you can see from my previous post below, I tested via both AFP and SMB. With the latest 6.14 beta, the modified time is now correct regardless of protocol. The create time is being reset however and differently depending on which protocol is used.

Thanks for looking into this for me. You're awesome.

Posts: 7211 · Zenju 23 Jan 2015, 12:57

File modification time and creation time are really separate issues:

1. mod time not copied correctly: => bug, this should never happen (Is this solved now on SMB and the current beta?)
2. create time: => not yet supported by FFS. No bug, but a todo.

Posts: 7211 · Zenju 23 Jan 2015, 13:00

Sorry, we have two conversations going now inside this thread. My fault! :) I will keep all responses here from now on.

As you can see from my previous post below, I tested via both AFP and SMB. With the latest 6.14 beta, the modified time is now correct regardless of protocol. The create time is being reset however and differently depending on which protocol is used.

Thanks for looking into this for me. You're awesome.htismaqe

Great, so the modtime issue on SMB can be considered solved!

As far as create time goes, I'm working on it. I'll (probably) post a new build later.

Posts: 73 · htismaqe 23 Jan 2015, 13:01

File modification time and creation time are really separate issues:

1. mod time not copied correctly: => bug, this should never happen (Is this solved now on SMB and the current beta?)
2. create time: => not yet supported by FFS. No bug, but a todo.Zenju

1. yep, modified time is now correct on both SMB and AFP.
2. excellent, to know you're working on it is good enough for me!

Posts: 73 · htismaqe 23 Jan 2015, 13:20

Great, so the modtime issue on SMB can be considered solved!

As far as create time goes, I'm working on it. I'll (probably) post a new build later.Zenju

You're the best, truly. Thanks for all of your time and effort.

Posts: 7211 · Zenju 23 Jan 2015, 20:00

Alright, I've created a new version that copies file creation times in addition to modification times. Also FFS is now using a new API that supports nano-second precision.
Since this is no small change, it would be great if you could test this with AFP and SMB and let me know if you find any issues. Here's the new beta:

[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.14_beta3_Mac_OS_X_64-bit.zip]

Posts: 7211 · Zenju 24 Jan 2015, 11:53

One more update and refinements:
[404, Invalid URL: https://freefilesync.org/FreeFileSync_6.14_beta4_Mac_OS_X_64-bit.zip]

Posts: 73 · htismaqe 26 Jan 2015, 15:29

It appears to be working as of right now. Thanks!

Posts: 73 · htismaqe 26 Jan 2015, 17:33

So far, so good.

I completely formatted the remote drive, wiping out the previous directory structure entirely. I then completely removed FFS from my machine, including the app support directory and preference files.

Regardless of whether I use SMB or AFP, the created and modified dates are both reserved.

EDIT: For right now, I'm going to stick with AFP. Despite the fact that RTS still calls FFS for no reason every 20 seconds, AFP is faster (by quite a bit actually, transferring about 100GB of data took 15 minutes longer with SMB) and maintains the resource fork in the main file rather than writing an external file like SMB.