How does compare content work?

[banger696]

Without too much detail how does compare content work. Does it compare hashes of files or some other way. Just so I can be confident my files are the same for backups.

And when you mirror files or update is there a way to just compare the files just copied instead of the whole disk. As you can imagine compare 4tb of files takes a while.

[Plerry]

FreeFileSync (FFS) does not use hashes.
If you Compare by file content, FFS simply compares the full content of the left- and right-side file; the best guarantee that both files are identical.
Even more so if you make sure data buffers have been flushed since your last sync, e.g. by restarting your computer..

> ... is there a way to just compare the files just copied instead of the whole disk.
You can edit the GlobalSettings.xml file and set the VerifyCopiedFiles flag to "true". (Make sure to read the limitations mentioned in the description)
You can then set your Compare variant to "File time and size", which is much faster than "File content", particularly if one or more of your file locations is a network or remote location.
Compare by "File content" prior to the sync compares (the content of) all files in scope, not just the ones that were just synced.

[banger696]

Thanks for that the VerifyCopiedFiles option will be handy when just copying a few big files instead of verifying 4tb again.

[banger696]

Does the Update option verify files ? The log doesn't seem to be populated and I didn't monitor it.

[therube]

Do you mean with an Update Sync?
If so, then yes.
.

[11:13:50 AM]  Info:  Synchronizing folder pair: Update >
                          C:\My Documents
                          \\Nas\back\RUBENXP43\My Documents
[11:13:50 AM]  Info:  Creating file "\\Nas\back\RUBENXP43\My Documents\IRS\f941-2026-03.pdf"
[11:13:50 AM]  Info:  Verifying file "\\Nas\back\RUBENXP43\My Documents\IRS\f941-2026-03.pdf"
[11:13:50 AM]  Info:  Creating file "\\Nas\back\RUBENXP43\My Documents\IRS\_Forms\f941-2026.pdf"
[11:13:50 AM]  Info:  Verifying file "\\Nas\back\RUBENXP43\My Documents\IRS\_Forms\f941-2026.pdf"


[banger696]

Yes is this in Linux or Windows?

[banger696]

So looking at the log folder in Linux I can see that the Update sync did in fact verify the copied files.

[banger696]

Tried again this week and the GUI log was populated so must have been a one off.

[banger696]

Ah I see where I went wrong there was a warning about the drives being significantly different. If you click the 'i' information button on the left hand side all is revealed. :)

[bitwyse]

FreeFileSync (FFS) does not use hashes.
If you Compare by file content, FFS simply compares the full content of the left- and right-side file; the best guarantee that both files are identical.

It is not a guarantee at all: FFS frequently reports files as different, but a SHA256 comparison shows that they are identical. This could not happen if there were ANY differences.
I have confirmed this repeatedly. Just now it reported 9 files different: all but 2 are in fact identical.
It nearly always happens with large files (> 80 GB) which are complete system backups.

Comparing data on two USB disks: one permanently connected for daily backups, the other 3 in rotation (kept elsewhere).
All these disks have been checked with Crystal DiskInfo and chkdsk - with no errors at all.
I have also tried it with freshly formatted disks (full format), checked the same way.

So FFS is useful for a quick check for new and deleted files (on date and size) but completely unreliable using content comparison.

If you let it run anyway then compare again, it may say that the files that it just copied are different.

[Plerry]

The SHA256 algorithm only reads and evaluates the actual binary content/payload of a file.
It does not compare file meta-data.

[bitwyse]

If w want to restore a backup of a file what we are interested in is indeed the actual binary content.
It doesn't matter when the file was written, by whom, when, its permissions - or any other properties.
If needed we are going to want to restore its content to use.

If you compare the content of a file block by block then that won't include the meta-data.
FFS sometimes thinks files are the same on one run but not on the next. But the source file hasn't changed (including its meta-data). So it's not doing it well.
It's a waste of time over-writing a file with another whose content is identical.
There will always be some difference (like the time stamps) - but that's of no interest.

[therube]

FFS frequently reports files as different, but a SHA256 comparison shows that they are identical. This could not happen if there were ANY differences.

That is very strange, & unexpected.
Is there anything, something a wee bit smaller then 80 GB, that this can be duplicated on?

[therube]

The SHA256 algorithm only reads and evaluates the actual binary content/payload of a file.
It does not compare file meta-data.

And a content compare (VerifyCopiedFiles) does?

I would be very surprised if it did.
(I have never heard of a comparison program that does such?)


I would expect any file that hash compares to also content compare & vice-versa.

[bitwyse]

FFS frequently reports files as different, but a SHA256 comparison shows that they are identical. This could not happen if there were ANY differences.

That is very strange, & unexpected.
Is there anything, something a wee bit smaller then 80 GB, that this can be duplicated on?

Well in the last batch there was one of 300 MB. Still a bit big to send - and it's private !

[bitwyse]

And a content compare (VerifyCopiedFiles) does?

I would be very surprised if it did.
(I have never heard of a comparison program that does such?)

No way.
It would be a separate operation - and it wouldn't know what to compare: the meta-data depends on the file type, the program that wrote it, etc.
And the creation date is bound to be different (unless the copy program sets it).

I would expect any file that hash compares to also content compare & vice-versa.

Yeah, that's why I'm not happy with it.
But I have no doubt that if SHA256 produces the same hashes then the file contents are identical.

So I've started a program to copy and verify without having to wait to the end to find it didn't work:
1. read block from source
2. write block to destination
3. compare src block to dest block read back with SHA256
4. if fail retry (2) to (3) chosen number of times (3 ?)
5. if it fails 3 times, report error and exit, if success go to next block.

(step 3 covers any possible r/w errors)

[therube]

It does not compare file meta-data.

Were you speaking of ADS (Alternate Data Streams) here, aka, "meta-data"?
If so, then you would be correct that a hash compares a "file" (& not ADS).

Now what a content compare (VerifyCopiedFiles) does?
Does that verify both the "file" & the ADS?

(I may get around to checking... at some point...)


Given that ADS is a "stream" that piggy-backs (if you will) on a file (name)... oh, what a mess.
(ADS, IMO, have always been a mess.)

[Plerry]

Were you speaking of ADS (Alternate Data Streams) here, aka, "meta-data"?
If so, then you would be correct that a hash compares a "file" (& not ADS).

I had to check, but according to Google SHA256 does not include/compare ADS-data.