Graham Sutherland / Polynomialreposting for the day crowd: I ran into a memcmp implementation that only compared 25% of the bytes, and the issue wasn't caught in the build because the vendor toolchain failed to emit a warning.
to be fair it should also have been unit tested but I'm gonna cut the devs some slack here because the toolchain vendor rugpulling a whole warning category is a significantly worse offense.
Dag Ågren ↙︎↙︎↙︎@gsuberland Pretty sure this would have passed the unit tests that anyone would have been likely to write anyway.
@WAHa_06x36 this is why fuzz testing is a thing!
@gsuberland Hmm, would even fuzz testing find it? That seems tricky to set up in a way that a) would actually find the bug and b) would occur to you before seeing the bug.
I guess for very short inputs you might find it more easily by chance...
@WAHa_06x36 of course. fuzz testing would quickly find memcmp("aaaa", "Aaaa") == 0 or memcmp("aaaa", "aaaA") == 0 as a violation of the contract (depending on endianness)
@gsuberland I mean, if you set up a special test harness against a known-good implementation and used something like afl that actually instruments the code itself, maybe, but, who would ever do that?
@WAHa_06x36 quite a few people! there are even coverage tools specifically for doing this.
@gsuberland Hmm, interesting, haven't seen those!
halcy 🔜 
@WAHa_06x36 @gsuberland i think „only one byte differs“ kind of tests would probably find it, right? And these seem like something you’d write to test that