Bryan CantrillDec 16, 2022

Throughout my career, I have believed that it is essential to dig into mysterious pathological systems behavior even if it seems somewhat tangential, for it can often reveal problems deep in the system that can have much more damaging presentation.

For a vivid example of pathological behavior, a deep underlying problem, and (especially) the methodology to connect one to the other, read this extraordinary analysis from @oxidecomputer engineer Dave Pacheco:

https://github.com/oxidecomputer/omicron/issues/1146

cockroachdb crashed in Go runtime during test run: `s.allocCount != s.nelems` · Issue #1146 · oxidecomputer/omicron

There's a lot of detail in this report. For a summary of this problem, the root cause, and a workaround, see this comment below. Again trying to reproduce #1130, I found a different issue that ...

GitHub
Show threadJauntyWunderKind

@bcantrill @oxidecomputer i look for some kind of blanket term to advocate fearlessness, the drive to keep going lower and lower through the machine until it starts making sense. that willingness to roll up your sleeves & dive in & find out is an essential mindset to have available when working with computers, is how we attain real knowledge & make informed choices.

this is an incredible tour de force going so very deep into the machine. but even just breaking open the source of the library we might be using is a critical step a lot of devs bounce off of.

trying to use Maven plugins was perhaps the greatest teacher of this lesson to me, so so long ago. most ended up being fairly direct & simple, but the documentation & interfaces, as an external user, had always left vast mystery & uncertainty & confusion.

getting lower in the machine needs stronger advocacy! yes we can! we can find out!

Dec 17, 2022 at 5:35pmWeb