Comparing contents takes days

[Tio]

I am syncing 2 big folders from one hard drive to another. Say around 1.5TB and 700.000 files maybe. I did a first run and took around 3 days to compare and copy. I thought the first run is slower and the rest would be faster. Now the sync kicked in and I see it compares again the two folders and it takes around 2 days to complete. Is this normal? Thanks

[Plerry]

You should distinguish between the Compare phase and the Sync phase.
A Compare phase should last about equally long every time.
The Sync phase of the initial sync normally takes much longer than any consecutive sync, as in that initial sync all data must be copied over, and in consecutive syncs only any changes in data.

If you don't have a lot of changes, during consecutive runs your Compare phase will normally last much longer than your sync phase.
Which Compare variant (F6) did you select?
Obviously, if you select the File Content variant, your comparison will be slow; even more so if one or more of the locations involved in your sync is a network/remote location.
Selecting the File Time and Size Compare variant should be much faster, and should be quite OK for sync purposes.

[Tio]

Ah yes, if I switch to Compare Time and Size and it is hundreds of times faster. Thanks a lot. Instead of 3 days it takes some 1 hour which is reasonable.

So comparing the Time and Size should be enough to catch any changed file if I'm correct. Right?

[Plerry]

Comparing File Time and Size should be sufficient to determine if files need to be synced or not.

Comparing File Content makes more sense for data verification, and is e.g. suitable for incidentally checking mutual data consistency between sync locations (not necessarily data integrity).

If you want to make as sure as practically possible that a synced file is copied over exactly as intended, you can set the VerifyCopiedFiles flag to True VerifyCopiedFiles flag. But also read the warnings there.

[Tio]

Thanks a lot for the explanation! Got it!

[Vrish25]

Feature request: Please add an option to set the comparison like this:

If both files have same timestamp and size, regard them as equal
Else compare them by content.

This would be very useful for cases where file timestamps have changed but not their content.
This can happen eg. when applying temporary changes to files and then reverting these changes.

You could change the comparison settings like this:

Compare Timestamp = On - Off
Compare Size = On - Off
Compare Content = On - Off - Only when Timestamp/Size differs

[xCSxXenon]

Seems like too much of a specific case to matter. It isn't often that changes are made, saved, reverted, then saved again, on s file large enough to be worth to effort. Files that are typically worked with like that are usually small, like documents, spreadsheets, etc

[bod]

Seems like too much of a specific case to matter.

I disagree. I have exactly this Use Case when comparing local files on Windows (NTFS) and a remote FTP server running Linux. The file sizes can differ by several bytes on small files due to block size differences.

So being able to compare by a quick Time & Size, and then fall back to Contents, would be beneficial.

Or, alternatively, be able to specify an allowable file size difference like you can specify an allowable time difference, would also work.

[therube]

file sizes can differ by several bytes

In which case, they will never compare by Content.

The file sizes can differ by several bytes on small files due to block size differences.

I'd think that due to different EOL characters.
Block size should be irrelevant to a file's size.
(Block size matters in different ways, "slack space" & whatnot.)


Now, if you wanted to do something like "ignore spaces" or compare ANSI vs Unicode as equals, or ... but things like that are totally separate (& I'd think outside of the realm of FFS).