Bryan CantrillThroughout my career, I have believed that it is essential to dig into mysterious pathological systems behavior even if it seems somewhat tangential, for it can often reveal problems deep in the system that can have much more damaging presentation.
For a vivid example of pathological behavior, a deep underlying problem, and (especially) the methodology to connect one to the other, read this extraordinary analysis from @oxidecomputer engineer Dave Pacheco:
cockroachdb crashed in Go runtime during test run: `s.allocCount != s.nelems` · Issue #1146 · oxidecomputer/omicron
There's a lot of detail in this report. For a summary of this problem, the root cause, and a workaround, see this comment below. Again trying to reproduce #1130, I found a different issue that ...