I found viewtopic.php?t=10156&hilit=idle+time#p37761 while looking for explaination why changing from default idle time of 10 seconds to anything longer resulted in FFS not getting triggered. Now I understand - my file was getting updated every 10 seconds, so FFS sync job would not (& should not) run until all the updates are done. I'd really like to have a job run every few minutes (so intermediate changes get synced), but only when there are changes to be made. Scanning on a schedule would create a lot of unnecessary network traffic.
So could we add a max time option to the currently specified minimum idle time? This would be the maximum time to wait before the job is triggered after a file change is detected.
Feature request: min and max idle time
- Posts: 3
- Joined: 17 May 2023
-
- Posts: 3582
- Joined: 11 Jun 2019
Interesting idea! I could see this people a convenient feature to add.
For the time being, or if it doesn't get implemented, you could write a script to achieve the same result. Have the minimum idle time to whatever you want, but have RTS call a batch script instead of FFS. In the script, use a 'timeout' to wait however long you would want your maximum time to be and then the next line calls FFS.
One issue I see with this is that RTS will probably call the script over and over. For example, your file changes every 10 seconds so you have to set minimum idle time to 9 seconds, but that will result in the script being ran every 10 seconds as well! No good. You could fix this by having the script place a "lock file", similar to FFS itself, in a temp directory. The first thing the script would do is check for the lock and terminate immediately if it's there. If it's not there, it should put it there, call the FFS sync, then have it remove the lock once done.
For the time being, or if it doesn't get implemented, you could write a script to achieve the same result. Have the minimum idle time to whatever you want, but have RTS call a batch script instead of FFS. In the script, use a 'timeout' to wait however long you would want your maximum time to be and then the next line calls FFS.
One issue I see with this is that RTS will probably call the script over and over. For example, your file changes every 10 seconds so you have to set minimum idle time to 9 seconds, but that will result in the script being ran every 10 seconds as well! No good. You could fix this by having the script place a "lock file", similar to FFS itself, in a temp directory. The first thing the script would do is check for the lock and terminate immediately if it's there. If it's not there, it should put it there, call the FFS sync, then have it remove the lock once done.
-
- Site Admin
- Posts: 7049
- Joined: 9 Dec 2007
It seems the feature request is for an option to avoid running synchronization too frequently:
"Minimum time since last sync"
"Minimum time since last sync"
-
- Posts: 2271
- Joined: 22 Aug 2012
> It seems the feature request is for an option to avoid running synchronization too frequently:
I believe the original request was the opposite:
Preventing that FFS will never be launched by RTS because of changes continuously keeping occurring during the Idle Time.
Hence WilliamB's request for the addition of a max time option.
In my translation: a timer that is started when the first change is detected but, unlike the Idle timer, is not reset upon detecting consecutive changes. A FFS sync (or actually: the RTS command line) would then be executed upon expiration of the Idle Time OR the Max Time.
xCSxXenon's reaction describes the present way to overcome the problem of RTS never starting FFS, by setting the Idle Time sufficiently short. Then the risk is that FFS may be run very/too frequently.
Which may be overcome by some form of Min Time since last launch of FFS (by the current RTS instance).
I believe the original request was the opposite:
Preventing that FFS will never be launched by RTS because of changes continuously keeping occurring during the Idle Time.
Hence WilliamB's request for the addition of a max time option.
In my translation: a timer that is started when the first change is detected but, unlike the Idle timer, is not reset upon detecting consecutive changes. A FFS sync (or actually: the RTS command line) would then be executed upon expiration of the Idle Time OR the Max Time.
xCSxXenon's reaction describes the present way to overcome the problem of RTS never starting FFS, by setting the Idle Time sufficiently short. Then the risk is that FFS may be run very/too frequently.
Which may be overcome by some form of Min Time since last launch of FFS (by the current RTS instance).
-
- Posts: 2271
- Joined: 22 Aug 2012
Additional remark to my above reaction:
The 'Min Time since last launch of FFS' option will prevent RTS runs FFS very/too frequently.
However, it will not prevent RTS from never launching FFS in case the Idle Time has been set too long.
Conversely, the 'max time' option will prevent that RTS will never launch FFS (after having detected a change), but will only assure that FFS will not be launched too frequently by setting the Idle Time relatively long.
Conceptually, having both options seems feasible. They don't bite each other.
Provocative suggestion:
I even see no fundamental reason why both times could not be one and the same time ...
The 'Min Time since last launch of FFS' option will prevent RTS runs FFS very/too frequently.
However, it will not prevent RTS from never launching FFS in case the Idle Time has been set too long.
Conversely, the 'max time' option will prevent that RTS will never launch FFS (after having detected a change), but will only assure that FFS will not be launched too frequently by setting the Idle Time relatively long.
Conceptually, having both options seems feasible. They don't bite each other.
Provocative suggestion:
I even see no fundamental reason why both times could not be one and the same time ...
-
- Posts: 2271
- Joined: 22 Aug 2012
Some further thoughts on this topic:
The 'max time" (as described above) would guarantee that ultimately FFS will be run in case the Idle Time would be repetitively reset by recurring changes.
However, if changes occurred during the last FFS sync that was launched by RTS (during which time RTS is blind for any changes), RTS would still never trigger a next FFS sync unless any later changes occur after FFS has finished and RTS is actively monitoring for changes again.
An option to also address this issue in RTS could be to have a 'Max Time since last launch of FFS by RTS', next to, or even instead of a 'Max Time since detection of first change by RTS'.
A both a 'Min Time since last launch of FFS by RTS' and a 'Max Time since last launch of FFS by RTS' would be implemented, it would conceive a maximum, respectively minimum frequency at which RTS will launch FFS.
Unfortunately, the 'Min Time since last launch of FFS by RTS' and a 'Max Time since last launch of FFS by RTS' times can not be the same time ...
In case of multiple instances of RTS, these timings should obviously apply independently per RTS instance.
The 'max time" (as described above) would guarantee that ultimately FFS will be run in case the Idle Time would be repetitively reset by recurring changes.
However, if changes occurred during the last FFS sync that was launched by RTS (during which time RTS is blind for any changes), RTS would still never trigger a next FFS sync unless any later changes occur after FFS has finished and RTS is actively monitoring for changes again.
An option to also address this issue in RTS could be to have a 'Max Time since last launch of FFS by RTS', next to, or even instead of a 'Max Time since detection of first change by RTS'.
A both a 'Min Time since last launch of FFS by RTS' and a 'Max Time since last launch of FFS by RTS' would be implemented, it would conceive a maximum, respectively minimum frequency at which RTS will launch FFS.
Unfortunately, the 'Min Time since last launch of FFS by RTS' and a 'Max Time since last launch of FFS by RTS' times can not be the same time ...
In case of multiple instances of RTS, these timings should obviously apply independently per RTS instance.
-
- Posts: 3582
- Joined: 11 Jun 2019
Opposite I think. The request is for the option to specify a maximum time RTS will wait before syncing, even if changes have occurred within the minimum idle time. Personally, if the changes are so constant that RTS couldn't be used natively, I would set up a scheduled Task that runs every x minutes. The extra network traffic seems negligible in my opinion, but depends on the data set and network setup. For my environment, it wouldn't matter. Either way, the task could simply be set up to only run during working hours. I don't know, I think more information on the motivation for the OP would reveal a really niche situationIt seems the feature request is for an option to avoid running synchronization too frequently:
"Minimum time since last sync" Zenju, 20 Jun 2023, 17:20
-
- Site Admin
- Posts: 7049
- Joined: 9 Dec 2007
"Max time since last sync" seems to be more of a fix for an inappropriately high "idle time". In other words the same result can be achieved by setting a smaller "idle time". The downside if this OTOH could then be mitigated by "Minimum time since last sync".
Maybe "Minimum time since last sync" is even the more fundamental concept (from user's perspective): "idle time" could be hard-coded to 1 and removed from GUI, while only offering "Minimum time since last sync"?
Maybe "Minimum time since last sync" is even the more fundamental concept (from user's perspective): "idle time" could be hard-coded to 1 and removed from GUI, while only offering "Minimum time since last sync"?
-
- Posts: 2271
- Joined: 22 Aug 2012
> "Max time since last sync" seems to be more of a fix for an inappropriately high "idle time"
Agree. Not a problem for more experienced users, but only too often we see posts in this forum from less experienced users that misunderstand the Idle Time and/or its impact.
The attractiveness of the Idle Time is that if a "burst" (fast sequence) of file operations occurs, that the sync is postponed beyond to when that burst has ended.
Hard coding a fixed, very short Idle Time would have the disadvantage that if said burst occurs more than the "Minimum time since last sync" after the last sync, the renewed sync may be started during the burst and not wait until the burst is over ...
A hard nut to crack in a simple fashion, it seems.
Agree. Not a problem for more experienced users, but only too often we see posts in this forum from less experienced users that misunderstand the Idle Time and/or its impact.
The attractiveness of the Idle Time is that if a "burst" (fast sequence) of file operations occurs, that the sync is postponed beyond to when that burst has ended.
Hard coding a fixed, very short Idle Time would have the disadvantage that if said burst occurs more than the "Minimum time since last sync" after the last sync, the renewed sync may be started during the burst and not wait until the burst is over ...
A hard nut to crack in a simple fashion, it seems.
- Posts: 3
- Joined: 17 May 2023
As the OP, I greatly appreciate the discussion on my request.
As Zenju says, "Minimum time since last sync" may be the parameter of value to the user.
Maybe describing my use case in more detail will help guide decision making.
I have a small group of computer that acquire data from a scientific instrument. During a scan (data acquisition), additional data causes a file update on the computer anywhere from 1 to 30 seconds. The scan may take anywhere from 5 minutes to 50 hours. Data can be (partially) analyzed before the scan is complete. Each of the computers has maybe 100,000 old files, so I don't want to be triggering syncs too often. But I'd like the files as they are copied from the instrument computers to my laptop to be up to date within, say about 5 minutes, without generating excessive network traffice or disc scanning.
So for some of my scans, with the usual idle time setting, no sync happens until the scan is done. I do want near-real time syncs without the overhead - extra traffic and disc activity of true real time syncing.
As Zenju says, "Minimum time since last sync" may be the parameter of value to the user.
Maybe describing my use case in more detail will help guide decision making.
I have a small group of computer that acquire data from a scientific instrument. During a scan (data acquisition), additional data causes a file update on the computer anywhere from 1 to 30 seconds. The scan may take anywhere from 5 minutes to 50 hours. Data can be (partially) analyzed before the scan is complete. Each of the computers has maybe 100,000 old files, so I don't want to be triggering syncs too often. But I'd like the files as they are copied from the instrument computers to my laptop to be up to date within, say about 5 minutes, without generating excessive network traffice or disc scanning.
So for some of my scans, with the usual idle time setting, no sync happens until the scan is done. I do want near-real time syncs without the overhead - extra traffic and disc activity of true real time syncing.
-
- Posts: 2271
- Joined: 22 Aug 2012
On your use-case:
> The scan may take anywhere from 5 minutes to 50 hours
I assume you mean the duration of the FFS Compare phase?
> Each of the computers has maybe 100,000 old files
Which may be a reason why an FFS Compare phase would/could take that long.
Still, if my assumption is correct, 50 hours to compare 100,000 files seems excessive, unless you compare by Content and not by Time+Size.
You should consider if it is possible to store the data that needs to be synced quickly in a different or deeper folder than your 100,000 old files. And then have RTS just monitor said different or deeper folder, and FFS sync just that said folder. That would likely significantly reduce the duration of the FFS Compare phase.
If necessary, you could then use the Task Scheduler to (at low frequency) run another FFS sync to keep the 'old' files in sync.
> The scan may take anywhere from 5 minutes to 50 hours
I assume you mean the duration of the FFS Compare phase?
> Each of the computers has maybe 100,000 old files
Which may be a reason why an FFS Compare phase would/could take that long.
Still, if my assumption is correct, 50 hours to compare 100,000 files seems excessive, unless you compare by Content and not by Time+Size.
You should consider if it is possible to store the data that needs to be synced quickly in a different or deeper folder than your 100,000 old files. And then have RTS just monitor said different or deeper folder, and FFS sync just that said folder. That would likely significantly reduce the duration of the FFS Compare phase.
If necessary, you could then use the Task Scheduler to (at low frequency) run another FFS sync to keep the 'old' files in sync.
