Lars WirzeniusApr 12, 2024

I published the wumpus hunter, a tool I wrote to help #Radicle find flaky tests in its test suite. Can your test suite run 1000 times in a row without failures? Have you tried?

https://crates.io/crates/wumpus-hunter
https://wumpus.liw.fi/counts.html
https://blog.liw.fi/posts/2022/kilotest/

crates.io: Rust Package Registry

Show threadDavid BremnerApr 12, 2024
@liw Interesting. This is one case where a CI-skeptic like myself can see the benefits. Let's see if the #notmuch test suite can run 1000 times before crashing or me getting annoyed by the extra load on my desktop. I think it should be able to finish about 200 repetitions an hour on this machine.
Show threadLars WirzeniusApr 12, 2024

@bremner The following Bash loop can suffice for a first try:

for x in $(seq 10); do echo -n "$x "; if ! chronic cargo test; then break; fi; done; echo

(for a Rust program: replace the "cargo test" bit with whatever notmuch needs.)

Show threadDavid Bremner

@liw Am at run 226. My usual incantation is

count=1; while make test ; do count=$((count + 1)); echo $count | tee -a runs.log ; done

I'm sure there are improvements possible. The suite internally uses "/usr/bin/parallel", so there's probably no point in trying to run many instances of it.

Apr 12, 2024 at 1:08pmWeb
Show threadDavid BremnerApr 12, 2024
@liw OK, you win, I found a bug after 291 runs. It seems something (probably #emacs #NativeCompilation) is leaving 16M of libgccjit-* junk in /tmp per run, so that kindof places a hard limit on the number of runs I can do at the moment.
Show threadLars WirzeniusApr 12, 2024
@bremner There's always something, I find.
Show threadBrett Sheffield (he/him)Apr 12, 2024

@bremner @liw Same problem here - filled the disk with test data files. These are presentl only cleaned up after a `make clean`.

Have raised a ticket for myself to delete the files iff the test passes. It doesn't autodelete them at present because they may need inspecting, but only on failure.