First off, I apologize for being such a pain. Hopefully, I have figured something out. If you can just validate my assumptions here, I think I may have a permanent workaround.
I was hesitant to just remove the remote folders from RTS monitoring for fear of losing data but then it dawned on my that RTS doesn't actually SYNC anything, all it does it monitor folders and call FFS when necessary.
I have created an FFS batch file that mirrors 5 local folders to the same 5 folders on a remote NAS. By default, if I drag that batch file into RTS, RTS is automatically setup to monitor BOTH sets of folders (10 in all, 5 local, 5 remote).
In normal operation, RTS would monitor both local and remote folders and call FFS to compare and mirror (local to remote) if any of them changed. If for example I accidentally deleted a file on the remote NAS, RTS would immediately call FFS and put the file back because it still exists locally.
My fear was that if I remove the remote folders from monitoring, RTS would only call FFS when changes were made local and FFS would only mirror changes made to local directories. In the above example, I was worried the accidentally deleted file would disappear forever. However, I believe that assumption is incorrect.
If I remove those folders from RTS, RTS would only call FFS when changes are made to the local folders. However, if I had made changes to the remote folders (such as the accidental deletion of a file in the example above), those changes would be immediately mirrored the next time FFS is called.
Going back to my example, if I accidentally deleted a remote file, RTS would not see it and do nothing. However, the next time I change a local file, RTS will call FFS. FFS will do a compare and see that the accidentally deleted file is missing and put it back.
Is all of this thinking correct? If so, I will consider this issue closed.
Using Mac OS X RealTimeSync with AFP
- Posts: 73
- Joined: 23 Feb 2007
- Posts: 73
- Joined: 23 Feb 2007
I was able to validate this myself. Here are my results.
I removed the remote folders from RTS. I let it sit for 30 minutes and verified that it did not call FFS even one time.
1. On the local drive, I created folder1 and folder2, each containing file1 and file2, and allowed RTS to call FFS, mirroring them to the NAS drive.
2. I deleted folder 2/file2 and waited. As expected, RTS did not run.
3. I added folder 1/file3 and allowed it to sync. After the sync, folder2/file2 was restored.
I'm not sure how exhaustive this test was but I think it's sufficient at this point.
EDIT: The only downside of this approach is the potential for long periods of time to pass between local changes. For myself, this is remedied by the fact that when I sleep/wake the Mac, the NAS volume is disconnected/reconnected. This causes RTS to immediately trigger a compare/sync, so I'm guaranteeing that my files get synced every morning when I wake up the machine.
I removed the remote folders from RTS. I let it sit for 30 minutes and verified that it did not call FFS even one time.
1. On the local drive, I created folder1 and folder2, each containing file1 and file2, and allowed RTS to call FFS, mirroring them to the NAS drive.
2. I deleted folder 2/file2 and waited. As expected, RTS did not run.
3. I added folder 1/file3 and allowed it to sync. After the sync, folder2/file2 was restored.
I'm not sure how exhaustive this test was but I think it's sufficient at this point.
EDIT: The only downside of this approach is the potential for long periods of time to pass between local changes. For myself, this is remedied by the fact that when I sleep/wake the Mac, the NAS volume is disconnected/reconnected. This causes RTS to immediately trigger a compare/sync, so I'm guaranteeing that my files get synced every morning when I wake up the machine.
- Posts: 73
- Joined: 23 Feb 2007
Of course, it turned out to be too good to be true.
If I remove all of the remote folders, RTS stops checking for the remote Volume altogether. So if the NAS goes offline, RTS stays active. If a local change is made, it tries to sync which of course fails. Not only that but it creates a new volume with the same name as the original destination volume. So when the NAS comes back online, it now has to mount itself with a different path because its original path already exists.
Of course, as soon as a I add a single remote folder back to RTS, it starts trying to sync every 20 seconds again.
If I remove all of the remote folders, RTS stops checking for the remote Volume altogether. So if the NAS goes offline, RTS stays active. If a local change is made, it tries to sync which of course fails. Not only that but it creates a new volume with the same name as the original destination volume. So when the NAS comes back online, it now has to mount itself with a different path because its original path already exists.
Of course, as soon as a I add a single remote folder back to RTS, it starts trying to sync every 20 seconds again.
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
Okay, this was your result:Continued from the other thread...
I've already gathered the output from the variables you outlined in one of my posts above.
RTS thinks the folders on my share have changed and triggers FFS. FFS then reports back that nothing has changed.
If there's additional troubleshooting I can do, I'm all for it. I'm just not sure where to go next.htismaqe
> change_action=UPDATE
> change_path=/Volumes/xxx/Documents
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
Yes your assumptions are correct. RTS is only used for detecting changes, FFS on the other hand will always sync irrespective of what RTS detected.First off, I apologize for being such a pain. Hopefully, I have figured something out. If you can just validate my assumptions here, I think I may have a permanent workaround.
I was hesitant to just remove the remote folders from RTS monitoring for fear of losing data but then it dawned on my that RTS doesn't actually SYNC anything, all it does it monitor folders and call FFS when necessary.
I have created an FFS batch file that mirrors 5 local folders to the same 5 folders on a remote NAS. By default, if I drag that batch file into RTS, RTS is automatically setup to monitor BOTH sets of folders (10 in all, 5 local, 5 remote).
In normal operation, RTS would monitor both local and remote folders and call FFS to compare and mirror (local to remote) if any of them changed. If for example I accidentally deleted a file on the remote NAS, RTS would immediately call FFS and put the file back because it still exists locally.
My fear was that if I remove the remote folders from monitoring, RTS would only call FFS when changes were made local and FFS would only mirror changes made to local directories. In the above example, I was worried the accidentally deleted file would disappear forever. However, I believe that assumption is incorrect.
If I remove those folders from RTS, RTS would only call FFS when changes are made to the local folders. However, if I had made changes to the remote folders (such as the accidental deletion of a file in the example above), those changes would be immediately mirrored the next time FFS is called.
Going back to my example, if I accidentally deleted a remote file, RTS would not see it and do nothing. However, the next time I change a local file, RTS will call FFS. FFS will do a compare and see that the accidentally deleted file is missing and put it back.
Is all of this thinking correct? If so, I will consider this issue closed.htismaqe
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
> So if the NAS goes offline, RTS stays active.
Usually RTS should go inactive, but it makes sense, since you don't monitor the NAS anymore in your new setup.
> If a local change is made, it tries to sync which of course fails. Not only that but it creates a new volume
That's bad. On Windows if a network drive is not existing FFS will fail and cannot create it. On Linux/Mac network drives are mounted to a local folder name. But why is this a problem? If a network share is mounted to an already existing local folder, it should just temporarily replace it, shouldn't it? In other words, the NAS should still be able to mount under the same local folder despite it already existing?
Usually RTS should go inactive, but it makes sense, since you don't monitor the NAS anymore in your new setup.
> If a local change is made, it tries to sync which of course fails. Not only that but it creates a new volume
That's bad. On Windows if a network drive is not existing FFS will fail and cannot create it. On Linux/Mac network drives are mounted to a local folder name. But why is this a problem? If a network share is mounted to an already existing local folder, it should just temporarily replace it, shouldn't it? In other words, the NAS should still be able to mount under the same local folder despite it already existing?
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
Okay let's try something. I've created a test version of RealtimeSync that prints out all the information it has to the console. This should help to find out what kind of notification "/Volumes/xxx/Documents" is receiving every 20 seconds:
1. Start the following version of RTS from console:
[404, Invalid URL: https://freefilesync.org/RealtimeSync_6.14_beta_Mac_OS_X_64-bit.zip]
2. Setup RTS with your "/Volumes/xxx/Documents" folder on the NAS that keeps triggering every 20 seconds, just as you did in your earlier tests. It doesn't matter what you enter as "command" in RTS.
3. Wait some time after you see a few entries in the console and let me know what it printed.
1. Start the following version of RTS from console:
[404, Invalid URL: https://freefilesync.org/RealtimeSync_6.14_beta_Mac_OS_X_64-bit.zip]
2. Setup RTS with your "/Volumes/xxx/Documents" folder on the NAS that keeps triggering every 20 seconds, just as you did in your earlier tests. It doesn't matter what you enter as "command" in RTS.
3. Wait some time after you see a few entries in the console and let me know what it printed.
- Posts: 73
- Joined: 23 Feb 2007
SF eats my first post and doubles my 2nd... :)> So if the NAS goes offline, RTS stays active.
Usually RTS should go inactive, but it makes sense, since you don't monitor the NAS anymore in your new setup.
> If a local change is made, it tries to sync which of course fails. Not only that but it creates a new volume
That's bad. On Windows if a network drive is not existing FFS will fail and cannot create it. On Linux/Mac network drives are mounted to a local folder name. But why is this a problem? If a network share is mounted to an already existing local folder, it should just temporarily replace it, shouldn't it? In other words, the NAS should still be able to mount under the same local folder despite it already existing?Zenju
- Posts: 73
- Joined: 23 Feb 2007
I really wish I could figure why sometimes SourceForge decides to re-prompt me for username and password instead of posting my reply. It takes a lot of time to re-type everything... :)> So if the NAS goes offline, RTS stays active.
Usually RTS should go inactive, but it makes sense, since you don't monitor the NAS anymore in your new setup.
> If a local change is made, it tries to sync which of course fails. Not only that but it creates a new volume
That's bad. On Windows if a network drive is not existing FFS will fail and cannot create it. On Linux/Mac network drives are mounted to a local folder name. But why is this a problem? If a network share is mounted to an already existing local folder, it should just temporarily replace it, shouldn't it? In other words, the NAS should still be able to mount under the same local folder despite it already existing?Zenju
> If a network share is mounted to an already existing local folder, it should just temporarily replace it, shouldn't it? In other words, the NAS should still be able to mount under the same local folder despite it already existing?
Unfortunately, no. If a folder already exists in /Volumes, OS X mounts the new volume as folder-1. For example, if I mount xxx via AFP, it creates /Volumes/xxx. If I then mount xxx via SMB, it creates /Volumes/xxx-1.
In the case of RTS/FFS, this is doubly bad. If the drive is unavailable, the call to /Volumes/xxx by FFS causes the creation of /Volumes/xxx. When the drive reconnects, the network share is now available via /Volumes/xxx-1, which of course completely breaks FFS because it's looking in the wrong path.
I've created a temporary workaround for now. I've mapped xxx via AFP and then created a dummy share call "SMB" and mapped it via SMB. I have RTS monitoring SMB but FFS is setup to sync xxx. Now if the NAS goes offline, my Mac goes to sleep, or I otherwise lose my network connection, BOTH drives are unmounted and RTS goes into suspend mode until they come back online. Since AFP connects much faster than SMB, there won't be many (if any) instances where xxx is available but SMB isn't so that should work for now.
I'm super busy with some other stuff but will begin testing the new beta you just posted as soon as I possibly can. Thanks again for all of your help!
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
> I really wish I could figure why sometimes SourceForge decides to re-prompt me for username and password instead of posting my reply. It takes a lot of time to re-type everything... :)I really wish I could figure why sometimes SourceForge decides to re-prompt me for username and password instead of posting my reply. It takes a lot of time to re-type everything... :)
> If a network share is mounted to an already existing local folder, it should just temporarily replace it, shouldn't it? In other words, the NAS should still be able to mount under the same local folder despite it already existing?
Unfortunately, no. If a folder already exists in /Volumes, OS X mounts the new volume as folder-1. For example, if I mount xxx via AFP, it creates /Volumes/xxx. If I then mount xxx via SMB, it creates /Volumes/xxx-1.
In the case of RTS/FFS, this is doubly bad. If the drive is unavailable, the call to /Volumes/xxx by FFS causes the creation of /Volumes/xxx. When the drive reconnects, the network share is now available via /Volumes/xxx-1, which of course completely breaks FFS because it's looking in the wrong path.
I've created a temporary workaround for now. I've mapped xxx via AFP and then created a dummy share call "SMB" and mapped it via SMB. I have RTS monitoring SMB but FFS is setup to sync xxx. Now if the NAS goes offline, my Mac goes to sleep, or I otherwise lose my network connection, BOTH drives are unmounted and RTS goes into suspend mode until they come back online. Since AFP connects much faster than SMB, there won't be many (if any) instances where xxx is available but SMB isn't so that should work for now.
I'm super busy with some other stuff but will begin testing the new beta you just posted as soon as I possibly can. Thanks again for all of your help!htismaqe
Oh yeah, this is a PITA for as long as I can remember. Not preserving the message text (when the cookie is outdated?) is Sourceforge's biggest (design-)bug IMHO.
I've created a bug tracker item on Sourceforge. Feel free to add your expierence thereby raising the priority:
https://sourceforge.net/p/forge/site-support/9579/
> Unfortunately, no. If a folder already exists in /Volumes, OS X mounts the new volume as folder-1.
Okay, I guess it depends on how you mount, then. I tested on OS X by mounting manually via command line and specifying a target folder for "mount" and it didn't matter that this folder existed and/or was not empty.
> In the case of RTS/FFS, this is doubly bad
So the best solution would then be to fix-up RTS. Looking forward to the new logs.
- Posts: 73
- Joined: 23 Feb 2007
Here's the output. I setup RTS to only monitor one remote directory. If I setup more than one, it lists all of them. It doesn't look particularly verbose to me but hopefully you'll know what those hex offsets are...
xxx-IMAC:MacOS xxx$ ./RealtimeSync
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:53:36.857 FreeFileSync[2081:49716] mbraHelper Loaded
2015-01-29 12:53:50.738 RealtimeSync[2079:49122] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:54:06.820 FreeFileSync[2084:50485] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:54:36.798 FreeFileSync[2088:51241] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:55:06.761 FreeFileSync[2090:52004] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:55:36.804 FreeFileSync[2092:52762] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:56:06.783 FreeFileSync[2094:53487] mbraHelper Loaded
- Posts: 73
- Joined: 23 Feb 2007
> Okay, I guess it depends on how you mount, then. I tested on OS X by mounting manually via command line and specifying a target folder for "mount" and it didn't matter that this folder existed and/or was not empty.> I really wish I could figure why sometimes SourceForge decides to re-prompt me for username and password instead of posting my reply. It takes a lot of time to re-type everything... :)
Oh yeah, this is a PITA for as long as I can remember. Not preserving the message text (when the cookie is outdated?) is Sourceforge's biggest (design-)bug IMHO.
I've created a bug tracker item on Sourceforge. Feel free to add your expierence thereby raising the priority:
https://sourceforge.net/p/forge/site-support/9579/
> Unfortunately, no. If a folder already exists in /Volumes, OS X mounts the new volume as folder-1.
Okay, I guess it depends on how you mount, then. I tested on OS X by mounting manually via command line and specifying a target folder for "mount" and it didn't matter that this folder existed and/or was not empty.
> In the case of RTS/FFS, this is doubly bad
So the best solution would then be to fix-up RTS. Looking forward to the new logs.Zenju
I normally mount directly from Finder using Automator. The command line works ok for SMB but has a myriad of other issues. Certainly, you can prevent OS X from auto-creating volume names by forcing it to mount to a specific folder but I have yet to get it to work that smoothly. All of my auto-mount routines are done via the GUI for that reason.
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
Thanks! Unfortunately the log results are not conclusive: Hex code "0x00020401" stands forHere's the output. I setup RTS to only monitor one remote directory. If I setup more than one, it lists all of them. It doesn't look particularly verbose to me but hopefully you'll know what those hex offsets are...
htismaqexxx-IMAC:MacOS xxx$ ./RealtimeSync
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:53:36.857 FreeFileSync[2081:49716] mbraHelper Loaded
2015-01-29 12:53:50.738 RealtimeSync[2079:49122] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:54:06.820 FreeFileSync[2084:50485] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:54:36.798 FreeFileSync[2088:51241] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:55:06.761 FreeFileSync[2090:52004] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:55:36.804 FreeFileSync[2092:52762] mbraHelper Loaded
0x00020401 /Volumes/xxx/Documents
2015-01-29 12:56:06.783 FreeFileSync[2094:53487] mbraHelper Loaded
kFSEventStreamEventFlagItemIsDir |
kFSEventStreamEventFlagMustScanSubDirs |
kFSEventStreamEventFlagItemInodeMetaMod
I cannot find any documentation on what "kFSEventStreamEventFlagItemInodeMetaMod" means and "kFSEventStreamEventFlagMustScanSubDirs" is documented as follows:
Your application must rescan not just
the directory given in the event, but all its children,
recursively. This can happen if there was a problem
whereby events were coalesced hierarchically. For
example, an event in /Users/jsmith/Music and an
event in /Users/jsmith/Pictures might be coalesced
into an event with this flag set and path=/Users/jsmith.
If this flag is set you may be able to get an
idea of whether the bottleneck happened in the
kernel (less likely) or in your client (more
likely) by checking for the presence of the
informational flags kFSEventStreamEventFlagUserDropped or
kFSEventStreamEventFlagKernelDropped.
I had hoped to see an event type that is nonsensical and can generally be ignored, but this is probably not the case with "kFSEventStreamEventFlagMustScanSubDirs".
- Posts: 73
- Joined: 23 Feb 2007
I think for right now, I'll just continue to use my workaround with the dummy SMB share. If you think of anything else I can try, let me know. When do you think 6.14 will be released?
-
- Site Admin
- Posts: 7211
- Joined: 9 Dec 2007
Probably in about a week or two.I think for right now, I'll just continue to use my workaround with the dummy SMB share. If you think of anything else I can try, let me know. When do you think 6.14 will be released?htismaqe
- Posts: 73
- Joined: 23 Feb 2007
Great. For right now, I'm just going to chill and wait for 6.14 release. I've got about 15 versions on my machine right now. :)
