I am building a new NAS and have over 20TB of files to copy from the old one. After copying them all over, just to be paranoid, I wanted to do a byte-by-byte compare before retiring the old NAS. FFS is great for this: I simply do a Compare operation using "File contents". This takes several days, with my 1Gbps ethernet, and it worked fine.
However, it would have been very nice to see some intermediate progress in terms of the number of files that have miscompared thus far in this very long process. Then, for example, after 10+ TB compared, if I knew that there were no miscompares, I could decide that all is well and cancel, saving a day or two.
Not a high priority, but seems an easy enough thing to add. Please consider it.
Thanks!
Feature suggestion: miscompare count in progress stats
- Posts: 21
- Joined: 14 Feb 2020
databoxer,
Here are two ways for you to compare the source folder to the destination folder for consistency.
First, is a program that I use in conjunction with FFS.
It's called rclone.
There is a "check" command built in.
after downloading the program you run this:
rclone check /path/to/source /path/to/destination
If the "destination" is your NAS, and it is remote, you'll need to add a "config" for it by running:
"rclone config"
Second,
There is a command available for linux OS called "rsync". It can also do a quicker comparison of two directories to make sure they match.
You can use the rsync -vnc command to compare two directories and check if they are the same. The -n flag tells rsync to do a dry run and not change anything. The -c flag tells rsync to compare the files using a checksum instead of just comparing the file sizes and modification times.
Here is an example of how to use it:
rsync -vnc /path/to/source/ /path/to/destination/
I hope this helps! Let me know if you have any other questions.
If you only have Windows, use rclone.exe or you can load rsync via WSL (Windows Subsystem for Linux)
Here are two ways for you to compare the source folder to the destination folder for consistency.
First, is a program that I use in conjunction with FFS.
It's called rclone.
There is a "check" command built in.
after downloading the program you run this:
rclone check /path/to/source /path/to/destination
If the "destination" is your NAS, and it is remote, you'll need to add a "config" for it by running:
"rclone config"
Second,
There is a command available for linux OS called "rsync". It can also do a quicker comparison of two directories to make sure they match.
You can use the rsync -vnc command to compare two directories and check if they are the same. The -n flag tells rsync to do a dry run and not change anything. The -c flag tells rsync to compare the files using a checksum instead of just comparing the file sizes and modification times.
Here is an example of how to use it:
rsync -vnc /path/to/source/ /path/to/destination/
I hope this helps! Let me know if you have any other questions.
If you only have Windows, use rclone.exe or you can load rsync via WSL (Windows Subsystem for Linux)
-
- Posts: 4056
- Joined: 11 Jun 2019
Also, you could have set the locations and/or filters to only compare 10TB in the first place
Where is that feature specifically? I can't find a filter option to limit the comparison or sync to be limited by quantity, gigabytes.Also, you could have set the locations and/or filters to only compare 10TB in the first place xCSxXenon, 20 Aug 2023, 14:11
- Posts: 21
- Joined: 14 Feb 2020
Sorry, but I don't see how this addresses my issue in any way. Maybe I'm missing something. I have more than 20TB to compare, and I want to compare all of it. What is the point of saying "stop compare after xxx TB total?" I am suggesting a feature that gives in-progress feedback on the how the comparison is going.Also, you could have set the locations and/or filters to only compare 10TB in the first place xCSxXenon, 20 Aug 2023, 14:11
In any case, the limits I see are for a single file size, not for the size of then entire comparison.
That's all I see too.In any case, the limits I see are for a single file size, not for the size of then entire comparison. databoxer, 20 Aug 2023, 14:57Also, you could have set the locations and/or filters to only compare 10TB in the first place xCSxXenon, 20 Aug 2023, 14:11
My next suggestion would be to break the sync into multiple jobs. Create a new profile and select enough folders that covers 5 Terabytes. Then save it with an appropriate job name and then create another one that does the next five terabytes and so on until you have all the jobs you need to synchronize all of the terabytes that you have.
-
- Posts: 4056
- Joined: 11 Jun 2019
There is no filter for size, I just meant use filters to leave an estimated 10TB leftWhere is that feature specifically? I can't find a filter option to limit the comparison or sync to be limited by quantity, gigabytes.
You specifically said "..after 10+ TB compared, if I knew that there were no miscompares, I could decide that all is well and cancel...", but now you're saying you do want to compare all of it? If the presence of errors within 10TB of the data is enough for you to conclude data integrity, as you said in your original post, then how does my suggestion of only comparing 10TB not address what you are trying to achieve?Sorry, but I don't see how this addresses my issue in any way. Maybe I'm missing something. I have more than 20TB to compare, and I want to compare all of it.
xCSxXenon,
Is there a way to tell free file sync to do a checksum after each file is transferred?
Is there a way to tell free file sync to do a checksum after each file is transferred?
-
- Posts: 4056
- Joined: 11 Jun 2019
https://freefilesync.org/manual.php?topic=expert-settings
"Verify Copied Files"
"Verify Copied Files"
Thank you!
