James Truitt (he/him)

Hey, #digipres folks, what algorithms do you use for periodic fixity checks on data at rest (esp. if stored in the cloud)?

CRC32 is tempting, because it's much faster than MD5 (and thus presumably cheaper in terms of compute, emissions, and money). However, I get the impression that it's not much used for this purpose. Is that because of collision worries? Not being cryptographically secure? Something else?

Mar 20 at 1:10pmWeb
Show threadDenis Warburton3d ago
@linguistory Xxhash is a modern non-cryptographic checksum used extensively in the film/tv industries thanks to its speed. I highly recommend it.
Show threadJames Truitt (he/him)3d ago
@artandtechnic Thanks! I should have specified, limitations of our cloud provider restrict us to the most widespread algos—CRC, MD5, or a SHA.
Show threadPeter B.1d ago

@linguistory Unless you're actually handling sensitive data, which would attract hashcode-matching forged data, I'm not worried about cryptographic-requirements on the hash.

For integrity checking.
According to Reto K., mac seems to have hardware support for SHA - outspeeding MD5. Not so on PCs.

On my notebook, MD5 is faster than CRC32, btw. (0,08 vs 0,136s)

Show threadPeter B.1d ago

@linguistory periodicity of checks of data "on shelves" (#LTO?):

Good question!

Professionally, I'd say "it depends": Find a sweetspot which doesn't disrupt daily work, and a method which has a reasonable runtime (days regular, weeks is normal, months uncommon)

Are you talking about data on shelves, or on a "nearline" backup copy on servers/HDDs?

With LTO there's also the issue with Generations and drives...