I have a test that generates random numbers as input for an algorithm. I can verify some general properties that always hold (“all the input is accounted for”) but I can also test some conditions that are probabilistically true (this should be at least 10, to 5σ). Thoughts?
(To be clear, the goal here is that there are some outputs that are very very unlikely and probably indicate a bug in my code rather than something that occurred naturally. But they *can* happen. Think “this sort algorithm never moved any elements”–fails if input was sorted.)
@saagar at some point the probability of failure is dominated by the probability of CPU misbehaviour or data corruption. "No flake" is a physical impossibility.
Saagar Jha
Paul Khuong