sluggish recognition of mapped NAS drive

Get help for specific problems
Posts: 2
Joined: 11 Aug 2007

rarewolf

I'm trying to schedule a synchronization of local data with a mapped D-Link NAS drive. It isn't a problem with FFS, rather it takes ~10seconds for the NAS to be recognized even by Win8. This results in FFS fatal error (see log below). It's as if the remote folder is not found, FFS decides to create it, and by that time the remote location appears. If I try to run the scheduled task the next morning, it will work fine ...

FFS v.5.18. Please advise ...
Log file
[11:30:34 PM] Info: Starting comparison
[11:30:44 PM] Error: Cannot find the following folders:
                     Q:\HTPC\Music Libraries\JRiver\
                     You can ignore this error to consider each folder as empty. The folders then will be created automatically during synchronization.
[11:30:51 PM] Info: Synchronizing folder pair:
                        C:\Data\JRiver\
                        Q:\HTPC\Music Libraries\JRiver\
[11:30:52 PM] Fatal Error: Target folder 'Q:\HTPC\Music Libraries\JRiver' already existing.
[11:30:52 PM] Error: Synchronization completed with errors
User avatar
Site Admin
Posts: 7211
Joined: 9 Dec 2007

Zenju

FFS waits at most 10 seconds before it gives up. This should be enough to answer the question if a file location exists or not. (Or are there relevant scenarios where checking may take more time?).
Delays introduced due to network startup are a different kind. These should be covered by a general retry-feature:
[404, Invalid URL: https://sourceforge.net/p/freefilesync/feature-requests/257/]
Posts: 2
Joined: 11 Aug 2007

rarewolf

The FFS sync mode I was using was 'Mirror'. Maybe a more extensive "retry" should be implemented when a root directory can be assumed(?)
Posts: 28
Joined: 6 Aug 2013

petrovski80

I'm having the same or similar issue: when FFS batch is started via the Windows task scheduler to perform daily backup to NAS it regularly but not always fails to find the target folders on the NAS. Manual execution of the same batch after this failed scheduled attempt always works.

My NAS isn't being detected 'sluggishly' by Windows. Connections made via a file manager are almost instant. However, a possible explanation for the intermittent failures could be HDD hibernation of my NAS disk drives. Perhaps waking from hibernation takes just too long before FFS reaches its 10 second timeout.

I'll do some tests tomorrow.
User avatar
Posts: 2451
Joined: 22 Aug 2012

Plerry

In multi-disk NAS systems, the spin-up of the individual disks is almost always done sequentially.
Spin-up requires a supply current that is substantially higher than the steady-state current.
Simultaneous spin-up of all disks thus requires a more capable / bigger / more expensive / (steady-state) less efficient power supply.

Many single-disk NAS systems already require about 10 seconds to "restore" from hybernation.
In multi-disk systems, the sequential spin-up is often 2~4 seconds apart, and adds to the single-disk restore time.
Posts: 28
Joined: 6 Aug 2013

petrovski80

Sounds logical. I do have a multi-disk NAS by the way (Synology DS213j). In other words: what we want/need is a FFS option to customize the timeout to cope with NAS disks waking from hibernation.

I can think of several workarounds (disable hibernation, create a script that accesses the NAS just before the FFS batch is executed, etc) but ideally FFS should be able to handle this.
User avatar
Site Admin
Posts: 7211
Joined: 9 Dec 2007

Zenju

The default time-out should be large enough for most common scenarios to detect directory existence. 10 sec may be a little too low considering disk spin-up times (or worse CD-Rom insert which takes 15-20 secs). On the other hand it should not be too large: imagine a user knowing that some network share is not reachable and FFS endlessly waiting for some status info. Maybe 15 or 20 secs would be a more useful limit. Everything beyond "common scenarios" should be handled by the "retry" feature (todo) which will allow to enter a number of retries and a delay between each one.
Posts: 28
Joined: 6 Aug 2013

petrovski80

Agreed. Perhaps let it default to 10 seconds which covers the majority of scenarios and create a customizable timeout for users who need it?
Posts: 28
Joined: 6 Aug 2013

petrovski80

Currently I've implemented the following workaround: I've created a batch script that will access the SMB share on my NAS, causing the drives to wake from hibernation if they are hibernating. Then wait for 20 seconds (abusing the ping command for this) before executing the FFS batch.

Batch file:

@ECHO OFF
echo Running daily FFS batch, wait for result...
dir \\DiskStation\path\goes\here > nul
ping -n 21 127.0.0.1 > nul
"C:\Program Files\FreeFileSync\FreeFileSync.exe" "C:\path\to\my.ffs_batch"
IF ERRORLEVEL 0 SET ERRORLEV=0 - Synchronization completed successfully
IF ERRORLEVEL 1 SET ERRORLEV=1 - Synchronization completed with warnings
IF ERRORLEVEL 2 SET ERRORLEV=2 - Synchronization completed with errors
IF ERRORLEVEL 3 SET ERRORLEV=3 - Synchronization was aborted
echo Result: %ERRORLEV%
pause
exit
Posts: 28
Joined: 6 Aug 2013

petrovski80

Hm. After a few days without the error (FFS cannot find the target folders on my NAS when scheduled job is executed), this evening the error returned. Decided to have a look at /var/log/messages on my NAS and indeed no wake up event logged around 17:00 (when my scheduled FFS job is executed). But, as the log showed it was busy indexing at the time thus the drives weren't hibernating at all. Perhaps my earlier theory that spin up times caused by NAS drives waking up were the cause for the FFS errors is not true.

Perhaps my PC is somehow the cause then? I wasn't home in the afternoon, but my PC was on and I was logged on. My PC is set to not go to sleep. Only the display is turned off after 30 minutes. And even if my PC was asleep, the scheduled job is set to wake the computer if needed. I can't reason something is wrong here.

After some reading on the internet I'm trying something else but really it's trail and error:

- removed all NAS related credentials via Windows 7 control panel and re-created them based on IP instead of NETBIOS name.
- made the FFS batch job use IP instead of NETBIOS name for the target folder paths.

Will report back with results. :-)
Posts: 28
Joined: 6 Aug 2013

petrovski80

No dice. Running out of ideas here. I guess we have to wait until Zenju implemented the retry function. :-/
User avatar
Posts: 2451
Joined: 22 Aug 2012

Plerry

As you are proficient in creating batch files, you don’t need to wait
until the retry option is included in FFS.
By replacing your
> dir \DiskStation\path\goes\here > nul

by a real check on availability, e.g. via
IF [NOT] EXIST \DiskStation\path\existing_file_name
and only call your FFS batch job if that file is available.

You can add a loop around such check that includes a counter and a wait (via “ping”),
and exits after a given maximum number of failed attempts.
Posts: 28
Joined: 6 Aug 2013

petrovski80

Thanks for that idea Plerry, will try it this evening.
Posts: 28
Joined: 6 Aug 2013

petrovski80

Finally found the root cause of my issue. As stated earlier, it has nothing to do with waking NAS drives from hibernation. I created an AutoIt script that accessed the target drive on my NAS in every possible way it supports, then checked the drive status and if this would return 'ready' it would execute the FFS batch. But whether I used UNC drive paths or mapped drive letters, the script would always detect the target every time I ran it manually and would (almost always) fail when started as a scheduled task via Windows task scheduler.

The cause
It turned out to be caused by the task scheduler setting "Run with highest privileges", which effectively runs my .bat file or AutoIt script as Administrator (FFS needs to run with elevated privileges if it's set to use Windows volume shadow copy or copy NTFS metadata). And as Administrator, network resources that are perfectly accessible with my normal Windows user account, were unavailable as Administrator (check: issue 'net use' command with your normal user account, then repeat with a command prompt with elevated privileges).

The solution
Issue the 'net use' command in the .bat file before the FFS batch is executed:


@ECHO OFF
echo Attempting map of \\server\share to z: target drive:
echo.
net use z: /d
net use z: \\server\share
if exist "\\server\share" (
"C:\Program Files\FreeFileSync\FreeFileSync.exe" "C:\path\to\my.ffs_batch"
exit
) else (
echo Target drive not reachable. Sync not possible.
)
pause
exit


Apparently there is also a registry hack that makes mapped network shares 'stick' between accounts, including Administrator. I haven't tested this.

Links
http://social.technet.microsoft.com/Forums/windows/en-US/31c9eff2-ece3-4430-886d-19b54796e411/windows-7-run-as-administrator-loses-mapped-drives

http://support.microsoft.com/kb/937624

http://www.winability.com/how-to-make-elevated-programs-recognize-network-drives/
Posts: 8
Joined: 27 Aug 2013

dkloucks

Hello, I am getting this same error with 5.20 only. It occurs even if i leave the administrator account as the default logged on account. The ffs batch and the scheduled task have been created by the administrator.
This error is occurring on both WinXP Pro 32bit and Win7 Pro 64bit.
Below is the log from the XP Pro 32 bit machine. Both machines logs look identical.

_____________________________________________________________
|8/26/2013 - lei-4push: Synchronization completed with errors
|
| Items processed: 0
| Items remaining: 133,458 (61.2 GB)
| Total time: 00:01:32
|____________________________________________________________

[9:00:01 PM] Info: Starting comparison
[9:00:11 PM] Error: Cannot find the following folders:
L:\lei-4\
You can ignore this error to consider each folder as empty. The folders then will be created automatically during synchronization.
[9:01:33 PM] Info: Synchronizing folder pair:
C:\
L:\lei-4\
[9:01:33 PM] Fatal Error: Target folder 'L:\lei-4' already existing.
[9:01:34 PM] Error: Synchronization completed with errors
Posts: 8
Joined: 27 Aug 2013

dkloucks

Here's the log from a Win2k machine that had the backup drive connected directly to it. FFS version is 5.20 on this machine, as well.

______________________________________________________
|08/26/2013 - 7: Synchronization completed with errors
|
| Items processed: 0
| Items remaining: 372,130 (193 GB)
| Total time: 00:01:03
|_____________________________________________________

[06:56:27 PM] Info: Starting comparison
[06:56:37 PM] Error: Cannot find the following folders:
Z:\lei-7\c\
Z:\lei-7\e\
You can ignore this error to consider each folder as empty. The folders then will be created automatically during synchronization.
[06:57:28 PM] Info: Synchronizing folder pair:
C:\
Z:\lei-7\c\
[06:57:29 PM] Fatal Error: Target folder 'Z:\lei-7\c' already existing.
[06:57:29 PM] Info: Synchronizing folder pair:
E:\
Z:\lei-7\e\
[06:57:29 PM] Fatal Error: Target folder 'Z:\lei-7\e' already existing.
[06:57:31 PM] Error: Synchronization completed with errors
User avatar
Site Admin
Posts: 7211
Joined: 9 Dec 2007

Zenju

The next version FFS 5.21 will have 20 secs as maximum timeout value. That's probably more useful than 10 seconds while still being less than infinite.
Posts: 8
Joined: 27 Aug 2013

dkloucks

Zenju, if it helps, I tried the 5.21 beta that you posted in another thread on the XP Pro machine. It worked. I had only 2 errors relating to file permissions on 2 files.
Posts: 8
Joined: 27 Aug 2013

dkloucks

Zenju, I installed 5.21beta on my W2K machine and my Win7Pro64 yesterday. They both performed their backups properly.
Posts: 1
Joined: 2 Sep 2013

denis-bisson

The next version FFS 5.21 will have 20 secs as maximum timeout value. That's probably more useful than 10 seconds while still being less than infinite.Zenju
GREAT!!! Good news!!! Many thanks! Thank you very much! In my case, it will certainly solve me one of the only thing I dislike from FFS! Recently in my schedule tasks, fifteen minute prior FFS task, I scheduled a dummy compression including a folder of each NAS, just to "awake" the sleepy NAS harddisks. I now removed it and will retry the new FFS version this night and see. I think 20 seconds will be perfect for me.

I am not sure 20 seconds will be enough for everybody though. It could have been a settings, an option, to accomodate those people with special situations. Let the option to 10 seconds by default if you think it's "normal" or "what it should be". But for those special situation, poeple will just love to be able to set to 60 seconds if they need to. 60 seconds is not "endless". It's just 60 seconds and the person who set intentionnally the timeout option to 60 seconds will not complain your program is slow or whatever, don't worry. But no matter what, I am happy with this new 20 seconds timeout!
User avatar
Site Admin
Posts: 7211
Joined: 9 Dec 2007

Zenju

Automatic retry has been implemented for the next version. This should handle all remaining cases of startup failures as well as other intermittent network problems:
[404, Invalid URL: https://sourceforge.net/p/freefilesync/feature-requests/257/]