Saagar JhaAug 16, 2024
I have a test that generates random numbers as input for an algorithm. I can verify some general properties that always hold (“all the input is accounted for”) but I can also test some conditions that are probabilistically true (this should be at least 10, to 5σ). Thoughts?
Add this to CI
46.7%
“Flaky” tests are not OK
53.3%
Poll ended at .
Show threadGeorge Nachman
@saagar seed the prng so it’s a deterministic test. If you change the code it might give a false fail but that’s better than no test and it won’t be flaky
Aug 16, 2024 at 5:46amWeb
Show threadSaagar JhaAug 16, 2024
@gnachman I guess this could work. I was against seeding the PRNG because it’s always testing the same thing, which kind of negates the point of testing against novel random data. But if the alternative is not testing at all this is probably better.
Show threadGeorge NachmanAug 16, 2024
@saagar if you give it enough iterations it’s still testing “a lot”. This is probably more for emotional support than correctness but it’s harmless