mhoyeJul 29, 2023

This is a great writeup of a DB corruption bug and its detection and resolution. Much respect to Claire, these "things that should always happen in the right order have happened in the wrong order because of some particular set of extreme conditions, with surprise downstream consequences" bugs are absolutely the worst. "Let's reason backwards from effects to causes, with the caveat that causality maybe sometimes doesn't exist" is so hard.

https://thomasp.vivaldi.net/2023/07/28/what-happened-to-vivaldi-social/

What happened to Vivaldi Social? | Thomas Pike’s other blog

A deep dive into the events of Saturday 8 July 2023, when user accounts started disappearing from the Vivaldi Social Mastodon instance.

Thomas Pike’s other blog
Show threadmhoye
A few things jumped out at me that were barely mentioned though, particularly around practices at the margins. Editing a whole-DB SQL file in Vim when you're exhausted? Using the edited file without tool validation first? This is a good incident report, but IMO they could stand to do a proper retrospective.
Jul 29, 2023 at 1:58amWeb
Show threadDevil Doll, Esq.Jul 29, 2023

@mhoye

“I know of no case study in history that describes an organization that has been managed out of a crisis. Every single one of them was led.” - Simon Sinek, "Leaders Eat Last"

Show threadmirabilosJul 29, 2023
@mhoye also wtf, splitting by characters not lines?