Can FreeFileSync do what I want?

[wus]

Hello,

I was using Resilio Sync for about 3 years to sync files between my PC, 2 laptops and a QNAP NAS. Until last year it was working well but then the problems started, and now I got to the same point that triggered Zenju to create FreeFileSync: One "unknown error" message too many.

Looking for alternatives I was pointed to FreeFileSync. https://www.computerbase.de/forum/threads/suche-alternative-zu-resilio-sync.2003911/post-25282898

From what I have seen so far it seems that FreeFileSync should be able to do what I want, but I am not sure, and I do have a couple specific questions.

Let me first describe what I want:
I and my partner who lives with me have 4 Windows machines, a tower and 3 laptops, all running Win 10, and a QNAP NAS (a TS-251+) running the latest version of QNAP's OS, currently QTS 4.5.2.1566. One of the laptops also runs Win 7 alternatively, occasionally. All PCs have at least 16 GB RAM and 500 GB SSDs. The NAS only has 4 GB RAM, I hope this is no problem. One folder on the NAS is mounted as a drive on the Windows PCs, this is my backup and synchronization folder.

My "data center" is the stationary PC which has enough HDD capacity (currently 8 + 10 TB HDDs) to hold all my data (currently about 7 TB, but bound to grow). The NAS serves as a file distributor within my home LAN. All data from the windows machines are continuously synchronized to the NAS. I and my partner access the NAS not only with the PCs but also with a couple streamers, smartphones, tablet and TVs.

Q1:
The 2 laptops obviously don't have such big storage capacity, so I can only sync some folders. I guess that shouldn't be a problem for FreeFileSync, but can the folders on the laptops have names different from the folders that they sync with on the NAS?

Q2:
Suppose I come home with 300 GB new data (photos, videos, soundfiles) on the laptop and place it in the docking station, which is connected by Gigabit LAN to the same switch where the NAS is also connected. Will FreeFileSync need much longer to sync the files to the NAS than if I copied them over manually?

Q3:
I haven't seen anything about where I need the software to run. Do I need a client on the NAS? If yes, do you have one for QNAP's OS, QTS?

Q4:
To save energy, I always shut down the computers when I'm not using them. When I go for a break I send them to "sleep", even if it's only for 10 minutes. The NAS runs from 7 am to 2 am, so it is shut down from 2 to 7 am. Will FreeFileSync be able to work properly under these circumstances? If not, can I set up FreeFileSync to shut the PCs down after it completed the sync? Or can FreeFileSync "delay" a manually invoked shutdown until it completed sync?

Q5:
I'm using an external USB 3.0 HDD to make additional backups once a month. Can I set up FreeFileSync to automatically sync data with the USB HDD when it gets plugged in? How long will it approx. need to check the whole 7 TB, and update, say, 500 GB of them?

[Plerry]

Did you watch the Tutorials?

Q1
No problem. You simply define one or more left and right sync base locations.
The path and path-length to those base locations may differ. The content of the base location will be synced, retaining the folder structure with in the base locations.

Q2
FFS always needs to run a Compare before performing a sync.
So, such first sync of a large amount of "new" data will take notably longer than directly copying over manually.

Q3
You run FFS on a laptop and/or a "server". The QNAP only needs to be accessible as a network resource.

Q4 Nothing goes really wrong if you shut down your PC while FFS may still be running. You just need to run an new FFS sync upon resuming your work to "finish" the sync.
But, you can also let FFS shut down (or hibernate) your PC upon completion of its sync. Use the "Run a command" part in the Sync Settings.
Or, you can define an FFS sync as a Scheduled Task, to be run on shutdown

Q5
You can use RealTimeSync (RTS) for that purpose. Upon detecting connection of the external drive, it can launch an FFS sync.

[wus]

This all sounds very promising!

Sorry I don't often watch video tutorials, I can't usually take a lot from videos. My English isn't quite good enough to listen to the voice and at the same time watch what's going on on the screen. I did watch a few of them now, and they were interesting / informative, but also hard to follow (had to repeat a few sections).

I will download and try FFS now.

Thank you, Plerry!

[wus]

Okay, I set up FFS to sync most of my folders on the local hard drives with their corresponding folders on the NAS which is mounted as an additional drive in Windows. This seems to work, I tried a few things to verify it. Very good!

But it requires manual activation.

Now I don't quite understand how I can make this a real time service with RTS. I read the manual but can't seem to find a possibility to import the folders (source and destination) from FFS. Do I have to do it all over again, from scratch? Is there no "bulk import" from FFS?

Also I don't know how to write the batch file that RTS calls in the command line. Are there any examples anywhere?

[xCSxXenon]

Drag and drop the ffs_batch configs you made onto a RTS setup window and it will auto fill the directories to monitor and the command line. You will only "have" to adjust idle time and then save the RTS config. You will have to configure the machine's startup to auto run these .ffs_real files so you don't have to start them manually. Whether you do that with some type of task scheduler or something else is up to you.

[wus]

Sorry, to drag and drop a file I first need to have this file. But - as written above - I don't know how to create this fffs_batch file. How do I do this? I googled for examples but didn't find any.

[wus]

I think I found it now, in one of the tutorials. This also seems to include the "bulk import" that I asked for above.

[wus]

I set RTS up now, but it doesn't quite work as expected.

After I saved the job that I previously created in FFS as batch job, I could open and, thus, import it in RTS. This also created the command line. When I clicked Start, RTS miminized to the notification area. I could hear the harddisks in my PC work for a couple minutes (as it did when I manually invoked the same job in FFS).

To test RTS function I then deleted a small subfolder in one of the monitored folders on the NAS. I did this using the Windows explorer, accessing the NAS's share mounted as a drive in Windows. I waited the idle time that I had entered in RTS (30 sec), but nothing happened. There was no other action going on, neither on the NAS nor on the PC.

Shouldn't it start to sync after 30 sec?

I waited another half hour or so, but the subfolder that I had deleted on the NAS was still present on the PC.

Am I missing something here?

[Plerry]

Look at the listed RTS limitations, specifically the 3rd bullet.
If the NAS OS does not support directory monitoring, changes on the NAS may not trigger RTS to launch FFS.

To verify the correct setup of your RTS job (assuming this a desktop or laptop to NAS sync) try making a change in part of the local filesystem that is monitored by RTS, e.g. by creating a small *.txt file.

[wus]

Thanks for your patience with me, Plerry.

I had seen these limitations, but thought as long as I stay in, and do everything, in Windows using the explorer, Windows should also generate the notification signal.

Did I expect too much?

[wus]

Importing that first batch file worked, and I also successfuly synchronised with an external USB HDD. However only when I used its drive letter. I tried to follow what is described here https://freefilesync.org/manual.php?topic=variable-drive-letters, but did not find the drop down menu that is mentioned in the blue Note box in the manual. So I used Wordpad's Find & Replace to replace all G: in the file by WD_14TB (that is how I call my external USB HDD) in the .ffs_gui file, and saved it back. When I tried to run this I got an error message "The following files do not yet exist:" with a long list of mtp:\WD_14TB\....

Did I misunderstand something here?

[Plerry]

When replacing "G:" by "WD_14TB", did you use square brackets (as in "[WD_14TB]" ), like shown in the manual section on Variable drive letters?
> ... WD_14TB (that is how I call my external USB HDD) ...
Obviously, you should also have changed its name accordingly. ;-)

mtp does not seem to make sense for a USB drive, but might be the result of (assumingly) forgetting the square brackets.

[wus]

I tried with and without square brackets, but none worked. And of course the drive was already named WD_14TB before I started to try.

Since you didn't suggest any other possible causes I had another look into the batch file with brackets and was surprized to see that there were double square brackets [[WD_14TB]].

I have no idea how they got there, but I corrected that and now it seems to work.

[wus]

Yesterday after I wrote the above, I verified that it works by adding files then checking that they were synced to the NAS. It took a while (I have over a million files in the folders scanned by RTS), but in the end I saw that it worked.

I then tried to set up RTS as a service on system startup, and that also seemed to work (although I'm not 100% sure now that I verified this, too, yesterday). Anyway, after starting up the computer again this morning, it didn't work.

So I went back and once more invoked RTS manually, the same way as I did yesterday. Like yesterday, I could see the RTS icon appear in the notification area, and after I again added a file in one of the local folders, I could see that "Scanning... 0" appeared in the notfication area (after the set delay of 10 sec).

Scanning.png (23.33 KiB) Viewed 2481 times

But unlike yesterday, where I could see a rapid progress next to "Scanning...", this time it remains at zero. Already since more than 30 min. Strangely, at the same time I can hear the local harddisk work all the time.

In the Task-Manager, I can see 3 RTS instances:

Task-Manager.png (17.38 KiB) Viewed 2481 times

What is going wrong here?

[Plerry]

If you have the Task Scheduler start an RTS task (at startup), the task should run, but does normally not show up in your task bar (upon logging-in).
Still, if you open the task manager, you should see the RTS task.
If you now also manually start the same RTS task, you would see two instances of RTS in the task manager.

When you now add a file to a local (RTS monitored) folder, after the idle time both identical RTS tasks may want to launch the specified FFS sync. I suppose the Task Scheduler started RTS instance launched its FFS sync just a bit earlier, and this FFS instance wrote the sync.ffs_lock file to the directories to be synced.
Just like this RTS task is not visible in your task bar, neither will this FFS instance, so you will not see its Compare and Sync progress.
Meanwhile, the FFS instance launched by your manually started RTS instance (both of which should be visible in your task bar) has started up but notices there is a sync.ffs_lock file in the locations it is trying to sync and waits (with progress 0) until the other FFS instance has ended and has removed the sync.ffs_lock files, to only then start a Compare and sync identical to one that has just run.
Very likely, in the condition you describe above (FFS progress 0), when you check your Task Manager you will see (at least) two RTS jobs, but also two FFS jobs, one running and the other one waiting ...

If you want to be able to see/monitor progress of any RTS launched FFS instances, you can best avoid using the Task Scheduler to launch RTS (at startup), but instead put the proper command-line instruction in your (current user) startup folder ( C:\Users\[User Name]\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup ) or add the RTS task to your (user) startup via some other suitable way.

[wus]

Thank you for this comprehensive answer! You were right, there were 2 instances of FreeFileSync shown in the Task Manager. I killed these now, and tried to also kill the 3 instances of RealTimeSync that the Task Manager showed (visible in my screenshot above).

But I could not instantly kill the last one. When I marked it and clicked the Task beenden button (Kill task - not sure how it is called in English, I only have a German Windows), nothing happened. I tried several times because I couldn't believe that the Task Manager would not kill this task, but it just didn't.

Anyway now that I wrote till here - 5 to 10 min later -, I tried once more, and now it killed it.

I will now try to put the proper command-line instruction in my (current user) startup folder ( C:\Users\[My User Name]\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup ), then do a restart.

Thank you again for your patience with me, and your continued support!

[Plerry]

As you killed the FFS tasks with the Task Manger, check if there are no left-over sync.ffs_lock files left in your left and right locations. If so, delete those before having a new FFS sync launched.

[wus]

I checked this before I tried, there were sync.ffs_lock files in all the synced folders. Since I didn't want to remove these files from all the many folders that I sync, I tried anyway, and it did not prevent FFS to to copy a new file to the NAS!

Now I'm curious how you will explain this.

[wus]

I also have some more questions. After I confirmed that FFS had synced the file that I had added to a folder on the local HDD to the NAS, I removed it from there, using the Windows Explorer. After the explanation that you had given yesterday,

Look at the listed RTS limitations, specifically the 3rd bullet.
If the NAS OS does not support directory monitoring, changes on the NAS may not trigger RTS to launch FFS.

I did not expect it to react to this - but it did, FFS deleted the file from the local HDD.

Question is, why did it do it this time, but not yesterday?

[wus]

And then I want to ask if I can add a 2nd command-line instruction in my (current user) startup folder. The one that I set up now syncs with the NAS, the 2nd one that I also want should sync to an external USB HDD when I connect it. I am afraid to cause the same lock-up situation that I had when I started RTS from the Task Scheduler.

[Plerry]

I don't know the details of how FFS handles locking/unlocking, but locking "expires" after some (?) time even if the sync.ffs_lock files were not deleted at the end of the previous FFS run.

> I did not expect it to react to this - but it did, FFS deleted the file from the local HDD.
Apparently your NAS does support directory monitoring after all.
No clue why it did not work before (unless you are now using a different protocol, e.g. FTP first and SMB now)

Having multiple instances of RTS active simultaneously is quite OK, provided there is no overlap between the RTS monitored folders.

[wus]

Of course I did not change protocols.

And the monitored local folders are exactly the same for the 2 sync jobs to the NAS and the USB HDD.

What can I do to solve this conflict and get both jobs done?

[Plerry]

> What can I do to solve this conflict and get both jobs done?
Which conflict and which two jobs?
There is no previous mention of a/this "conflict", only a not (yet?) understood apparent difference in ability of directory monitoring at the NAS side.
The only "both jobs" seem to be the two simultaneous RTS instances (and associated FFS syncs), for which it was already indicated that that is fine as long as there is no overlap between the RTS monitored folders (and the FFS synced locations).

[wus]

The 2 jobs are:

The one that I set up now syncs with the NAS, the 2nd one that I also want should sync to an external USB HDD when I connect it.

The conflict between the 2 jobs:

the monitored local folders are exactly the same for the 2 sync jobs to the NAS and the USB HDD.