Saagar JhaAug 16, 2024
I have a test that generates random numbers as input for an algorithm. I can verify some general properties that always hold (“all the input is accounted for”) but I can also test some conditions that are probabilistically true (this should be at least 10, to 5σ). Thoughts?
Add this to CI
46.7%
“Flaky” tests are not OK
53.3%
Poll ended at .
Show threadWowfunhappy
@saagar Would it make sense to try rerunning the test a few times on getting the unlikely result? I figure something which is merely unlikely to happen once becomes practically impossible to occur several times in a row.
Aug 16, 2024 at 8:55amWeb
Show threadSaagar JhaAug 16, 2024
@Wowfunhappy That’s just driving the probability down. I can already do that in the test itself by picking an appropriate problem size
Show threadWowfunhappyAug 16, 2024

@saagar But there's a point at which the probability is so infinitesimally small that it's irrelevant, right? I mean the test could also be messed up by cosmic rays flipping bits (perhaps so many in the wrong places that ECC doesn't save you), but you don't worry about that either.

If the chance of anyone ever running into this is smaller than, say, you winning the lottery jackpot for several weeks in a row, then stop worrying about it.

Show threadSaagar JhaAug 16, 2024
@Wowfunhappy This would be the second option in the poll