xxhash & hash-sync mode

[+ffsuser+]

please excuse me, if this topic has been mentioned before - my search did not yield any hits.

i would like to know if the following scenario is conceivable in ffs:

hash source files - sync - hash destination files - compare hashes

as this is quite time consuming, it would be advantageous to integrate xxhash into ffs: https://github.com/Cyan4973/xxHash

the speed of xxhash is quite impressive, in my opinion.

xxHash is an Extremely fast Hash algorithm, running at RAM speed limits. It successfully completes the SMHasher test suite which evaluates collision, dispersion and randomness qualities of hash functions. Code is highly portable, and hashes are identical across all platforms (little / big endian).

Benchmarks

Hash Name     Width     Bandwidth (GB/s)     Small Data Velocity     Quality
XXH3 (SSE2)     64     31.5 GB/s         133.1             10    
XXH128 (SSE2)     128     29.6 GB/s         118.1             10    
XXH64         64     19.4 GB/s         71.0             10
XXH32         32     9.7 GB/s         71.9             10


[Plerry]

See
viewtopic.php?t=6709&p=22256#p22256
viewtopic.php?t=6296&p=20695#p20695

[+ffsuser+]

hi pierry,

thanks a lot for the links.

first off, i just realize that my wording is a bit vague. of course i was referring to local disk-disk syncronization only plus a verify process of the hashes of both sides after synchronization finished.

anyway, seems like this option already exists and my suggestion comes far too late? :)

i was just so impressed by the benchmarks that xxhash offers, so i believed ffs (file content variant also) could greatly benefit from its speed.

do you have more details on how VerifyCopiedFiles process works right now? from what i understood, it does not make use of hash mechanism?

[Plerry]

See the FFS Manual section on VerifyCopiedFiles and the further info it refers to.
And also that verification does not use hashes, as FFS does not use hashes, for the reasons explained in the earlier references.

[+ffsuser+]

thanks for replying.

i understood that ffs does not make use of hash mechanisms.

the intention to mention xxhash was also rather meant to introduce a possibly new mode similar to the (quite slow) file-content-mode, but only for mirroring locally connected devices, e.g. hhd -> external-hdd - based on xxhash (including optional verification afterwards).

i am not sure if it can be implemented without major hurdles - theoretically, however, it should result in a safety and speed advantage due to the performance of xxhash.

[Plerry]

As should be clear by now: it simply does not make sense for FFS to make use of hashes/checksums for the reasons referred to earlier. So, don't expect this feature to be introduced in FFS any time soon (if ever).

[+ffsuser+]

don't expect this feature to be introduced in FFS any time soon (if ever).

no worries. be assured that i do not expect anything mandatory.

it was merely a suggestion and i was interested in your assessment.

so nothing to be afraid of.