Best way to use FFS to detect and mitigate bit rot?

[Wrinkle4709]

I am new to the backup life style. I heard about bit rot being a really low possibility (like extremely low) for files. However, I would like to just try to mitigate that if possible at all. Would using the "Compare File Content" comparing the files bit-by-bit be a good way to detect bit rot between same folders?

If so, would I run a "Compare File Content" and then a "Compare File time and size" to make sure everything is properly backed up?

[Plerry]

If you Compare identically named, sized and timestamped files by File Content and a difference would be detected, how would you know which of the two (left or right) is suffering from bit-rot?

[Wrinkle4709]

That is a great point. My current process is to test both files to see which one is working properly and keep that one. I have had one experience like this with FFS, where two files were different in the File Content initial "scan" where both of the directories both had a ".Trash-1000" folders in them. Once I deleted the ".Trash-1000" from both the scan came back clean.

However, I am more than happy to hear about a better way to think about this. My current workflow is where I backup files in a flow: Drive 1 -> Drive 2 -> Drive 3 -> Drive X. As such, if there was a rotted file, my worry is that it would pass down to all of my other drives.

[harryytm]

The best practice to detect bit rot is to create a checksum file for the known good copy of files, and verify the files to detect bit rot, or create parity files for known good copy of files to check and repair corrupted data in files.

Since FreeFileSync main purpose is for files sync, I think it is unlikely any hashing function will be implemented in the near future, but there are other existing tools such as HashCheck or MultiPar, to create checksum file for files or create parity of files to repair corrupted data in files.

Another tool dedicated to fight bit rot is SnapRAID, it is a file-level RAID system creating parity file on another disk to protect files from bit rot.

HashCheck
https://github.com/idrassi/HashCheck

MultiPar
https://github.com/Yutaka-Sawada/MultiPar

SnapRAID
https://www.snapraid.it/

[harryytm]

That is a great point. My current process is to test both files to see which one is working properly and keep that one. I have had one experience like this with FFS, where two files were different in the File Content initial "scan" where both of the directories both had a ".Trash-1000" folders in them. Once I deleted the ".Trash-1000" from both the scan came back clean.

However, I am more than happy to hear about a better way to think about this. My current workflow is where I backup files in a flow: Drive 1 -> Drive 2 -> Drive 3 -> Drive X. As such, if there was a rotted file, my worry is that it would pass down to all of my other drives.

You can make use of the file versioning function in FreeFIleSync, to keep old version of files to a chosen path with timestamp folder of sync time during file sync, this way you can still recover the uncorrupted files in the revision folder in case of files unknowly corrupted.

Manual: File Versioning
https://freefilesync.org/manual.php?topic=versioning

[Wrinkle4709]

The best practice to detect bit rot is to create a checksum file for the known good copy of files, and verify the files to detect bit rot, or create parity files for known good copy of files to check and repair corrupted data in files.

Since FreeFileSync main purpose is for files sync, I think it is unlikely any hashing function will be implemented in the near future, but there are other existing tools such as HashCheck or MultiPar, to create checksum file for files or create parity of files to repair corrupted data in files.

Another tool dedicated to fight bit rot is SnapRAID, it is a file-level RAID system creating parity file on another disk to protect files from bit rot.

HashCheck
https://github.com/idrassi/HashCheck

MultiPar
https://github.com/Yutaka-Sawada/MultiPar

SnapRAID
https://www.snapraid.it/

Thanks for sharing, I'll look into getting checksums and trying to leverage parity files!

[Wrinkle4709]

You can make use of the file versioning function in FreeFIleSync, to keep old version of files to a chosen path with timestamp folder of sync time during file sync, this way you can still recover the uncorrupted files in the revision folder in case of files unknowly corrupted.

Manual: File Versioning
https://freefilesync.org/manual.php?topic=versioning

My only worry about versioning is that the additional storage for versioning terabytes of data will end up taking a lot more space on my drives.

[harryytm]

You can make use of the file versioning function in FreeFIleSync, to keep old version of files to a chosen path with timestamp folder of sync time during file sync, this way you can still recover the uncorrupted files in the revision folder in case of files unknowly corrupted.

Manual: File Versioning
https://freefilesync.org/manual.php?topic=versioning

My only worry about versioning is that the additional storage for versioning terabytes of data will end up taking a lot more space on my drives.

In the case of files unknowly corrupted on one side, "Compare File time and size" would not trigger copying the bad files to the other side in a sync, becasue the modify timestamp are the same on both sides, and Versioning only store the old version of updated files and deleted files, it do not make copies of every files during a sync, so usually it don't take a lot of space.

[Wrinkle4709]

In the case of files unknowly corrupted on one side, "Compare File time and size" would not trigger copying the bad files to the other side in a sync, becasue the modify timestamp are the same on both sides, and Versioning only store the old version of updated files and deleted files, it do not make copies of every files during a sync, so usually it don't take a lot of space.

I think "File Content" might be able to pick this up. I'll have to test both sides to see which location has the proper data (still have to figure this part out myself). Thanks for letting me know about versioning not taking up too much space! I think I'll either do that, or the options you mentioned above with parity. Thanks again!

[harryytm]

In the case of files unknowly corrupted on one side, "Compare File time and size" would not trigger copying the bad files to the other side in a sync, becasue the modify timestamp are the same on both sides, and Versioning only store the old version of updated files and deleted files, it do not make copies of every files during a sync, so usually it don't take a lot of space.

I think "File Content" might be able to pick this up. I'll have to test both sides to see which location has the proper data (still have to figure this part out myself). Thanks for letting me know about versioning not taking up too much space! I think I'll either do that, or the options you mentioned above with parity. Thanks again!

Yes, compare with "File Content" does pick up silent data corruption in files, but you should do a compare with "File time and size" to make sure the the files on both sides are in sync at first.

If the files you tried to preotect from silent data corruption that don't update the content from time to time, creating a checksum file for those files or parity is what you should do.

Keep in mind checksum file can only verify the integrity of the file, it can't do repair, it can only help you identify which side of that mismatch content file is the good copy, and you have to maually copy it to the other side to "recover" the damange when pairing with mirror or two way sync using FreeFileSync.

MultiPar on the other hand create checksum of "protected" files before creating the parity for repair altereted data in files, but the parity file does take up spaces, given that you sould already have a backup copy on another disk.

[Wrinkle4709]

In the case of files unknowly corrupted on one side, "Compare File time and size" would not trigger copying the bad files to the other side in a sync, becasue the modify timestamp are the same on both sides, and Versioning only store the old version of updated files and deleted files, it do not make copies of every files during a sync, so usually it don't take a lot of space.

I think "File Content" might be able to pick this up. I'll have to test both sides to see which location has the proper data (still have to figure this part out myself). Thanks for letting me know about versioning not taking up too much space! I think I'll either do that, or the options you mentioned above with parity. Thanks again!

Yes, compare with "File Content" does pick up silent data corruption in files, but you should do a compare with "File time and size" to make sure the the files on both sides are in sync at first.

If the files you tried to preotect from silent data corruption that don't update the content from time to time, creating a checksum file for those files or parity is what you should do.

Keep in mind checksum file can only verify the integrity of the file, it can't do repair, it can only help you identify which side of that mismatch content file is the good copy, and you have to maually copy it to the other side to "recover" the damange when pairing with mirror or two way sync using FreeFileSync.

MultiPar on the other hand create checksum of "protected" files before creating the parity for repair altereted data in files, but the parity file does take up spaces, given that you sould already have a backup copy on another disk.

Understood. That makes sense. I'll keep that in mind going forward. Thanks for all your help! Much appreciated!

[Plerry]

If you need to use a quote at all, please only quote the relevant part of the post you respond to, not the entire post.