Performance freefile sync

[akleine]

After working quite a long time with freefile sync, I find two problems:

1. After Synchronise a folder with more than 130.000 files, it still takes a long time even though all files have already been checked.

2. Why does it take so long to transfer 1-2 files? It feels like a transfer of 100 files is significantly faster.

For example, this folder with 130,000 files takes 7-8 minutes to sync. Then the transfer of 1-2 files takes another 2 minutes.

How can we speed up the process?

My customer needs to snyc this folders on a regular bases.

[xCSxXenon]

1. Every FFS run will scan the entire set of data to determine what actions to take
2. 130k files that are the same total size as 1-2 files will take way longer to transfer. This is unavoidable and is due to the overhead required for each file that gets transferred. What you can do is make sure the two locations are on SSDs. Mechanical hard drives are only 'fast' when they are working with large files, but they are quite bad when they have to search around constantly.

[therube]

What type of Compare?
Content? -> Back up update taking ages.

[LeoW]

What OS are you running?

With that massive amount of files you might want to test a utility program that runs at the command line and can probably do what you need it to a lot faster than you're getting it done now.

For many years Linux users would do this using a program called rsync. It's not very user friendly and only works on Linux.

However, there is a newer program that is similar in use but also works with cloud providers and has lots of flags to control what it does with large amounts of data.

You can for example set a flag that says only look at and sync files that have been modified in the last one hour and skip everything else.

The program is called rclone.

As an example, you could run this command every hour and only touch those files that have a modified time no more than 1 hour old. Everything older is completely ignored.

There is no GUI, so reading the docs is vital.

rclone sync source:path destination:path --max-age 1h


On Windows, you can automate with Task Scheduler and on Linux you would use cron or some other task manager like Gnome Schedule.

You'll need to spend a number of hours learning how to get it all working but when it's done you should have what you need.

I can assist, and there is also a forum.

[Zenju]

The program is called rclone.

LeoW, did you just register on this forum to be a shill for rclone?
This underhanded kind of advertisement will not be tolerated for too long and should better stop fast.

[xCSxXenon]

The program is called rclone.

LeoW, did you just register on this forum to be a shill for rclone?
This underhanded kind of advertisement will not be tolerated for too long and should better stop fast.

I noticed the same... It's almost copy-pasted
It started out as offering a decent solution for something FFS isn't designed to do, but it isn't really a direct replacement

[LeoW]

LeoW, did you just register on this forum to be a shill for rclone?

NO!

My apologies if you are upset. That was not my intention. Let me explain.

I am not intending to "steal" anyone, or steer people away. I am a retired volunteer with no financial incentives regarding any software whatsoever.

I spend some of my free time helping people with a variety of software products.

I spent my whole career working for software companies, sometimes in QA, and other times in IT.

When your software was first conceived, 99.999 % of your customers were backing up their computer to a local external hard drive.

That is simply not case anymore. Today, most everyone is backing up to the cloud. Your software provides 2 cloud services, and you have no interest in investing time creating api connections to dozens of cloud providers.

Rclone specializes and excels in this area. They support over 40 cloud providers and counting. They do not however like you, provide a complete solution that has an installation program and a graphical user interface, and is easy to use to say the least.

Rclone can be difficult to use for most folks. It requires time, reading, and often times help from others. It does not provide features like Real Time Sync, and file versioning. There is no GUI user interface, and there won't be.

For those who want a nice, simple, GUI backup program, you have a great product. It just doesn't support cloud providers other than Google. Many people have Microsoft Onedrive accounts and if they choose not to use Microsoft Onedrive software they have no option other than to go out and find something like Rclone. Microsoft's onedrive synchronization solution doesn't meet everybody's needs.

Folks who find their way to your website looking for backup software are looking for free solutions not paid solutions and to connect to the cloud Rclone is a near perfect fit.

Your software, paired with Rclone offers a LOT of people a solution to their cloud backup woes.

I recently spent many hours over several days helping one of your paying customers configure MS Onedrive to work WITH FFS. They are very happily using YOUR software to back up to MS Onedrive, while loading Rclone as a Windows Service. That person is backing up with your software they are merely using Rclone to read and write data to the cloud.

99.9 % of the time I having been helping your customers use your software in the cloud not replacing it.

Another feature that Rclone can add to FFS is encrypted backups.

More and more people when backing up to the cloud would like for their backups to be encrypted and FFS cannot do this. Rclone can. Even Google Drive.
Your customers will still be using your software to do their backups.

When files get synced to the cloud they get client side encrypted to the Cloud Server and when they get synced back down to the PC they will become unencrypted.

This would be a great feature for you to advertise.

Rclone has an Android version too, so both unencrypted and encrypted cloud files are accessible on Android phones and tablets.
FFS doesn't have an Android client.

Soon, nearly everybody will be backing up their data to the cloud. That is one of the reasons that it is particularly useful to bundle these two software products so they work together not one replacing the other.

People are going to start looking for alternate solutions and even though the alternate solutions require that they buy something they will do it because they deem it necessary at that point and your software will become less and less used over time.

Every once in a while there will be a person that requires one product over another product. No backup software product is a solution to absolutely everybody's needs and that includes BOTH Rclone and FFS. There's really no reason to get all bent out of shape because one person may need a different solution.

If you look at all of my postings carefully you will see that almost all of them are pairing the two software products together so they work with each other as a solution.

There are now multiple people BEGGING for you to support both pcloud and Mega. Rclone can mount both of these so they can be used with FFS!!! When your software first came out most everybody was backing up to an external hard drive locally now most everybody wants to back up to the cloud so it would be to your advantage to support the use of Rclone in CONCERT with FFS!

It puzzles me why you would not want to allow your customers to use software that complements your software and makes it better and more compatible with cloud services

In conclusion, FFS & Rclone working together make a formidable pair and overall a complete solution that greatly enhances the functionality and usefulness of FFS!!

[Plerry]

@LeoW
Suggesting to add or integrate Rclone or Rclone-like features (in)to FFS is something that may be considered suitable for this FFS forum.
However, you should understand that, in "his" FFS forum, Zenju does not appreciate your (or anyone else's for that matter) not FFS related repetitive extensive instructions on how to use Rclone.
Perhaps Zenju's use of the word "shill" is a bit sharp, but nevertheless meanwhile quite understandable to me.

[Zenju]

My apologies if you are upset. That was not my intention. Let me explain.

The only problem is the developer of free file sync
[...]
He keeps bitching at me every time I tell someone how to solve the problem
[...]
It begins with the letter r and ends in

.org.

I don't have time for drama, so I'll make this short: This repeated advertizement for software that is not FreeFileSync stops now. Your future posts in this regard will not be on this forum anymore.

[boris]

It has to be said that most of the people I know are like me, we won't use the cloud. I have a 12 portable drives all for different purposes and I love my system. They all last about 8 years before I get rid, they all are quite fast through USB C, and I saved clicky backup links to my desktop for each individual drive. Doesn't get any simpler tbh.
You can keep your Arsclone, FreeFileSync has been looking after me for over a decade.

[Zenju]

@LeoW: I've just deleted your sockpuppet account "sueQ".

There are a few ways:

First, if your client is Windows you can try this:
[...]
3. Rclone - Free

I've never had to remove a user from this forum yet. You are about to be the first.