Mastodawn

mhoye Jul 29, 2023

This is a great writeup of a DB corruption bug and its detection and resolution. Much respect to Claire, these "things that should always happen in the right order have happened in the wrong order because of some particular set of extreme conditions, with surprise downstream consequences" bugs are absolutely the worst. "Let's reason backwards from effects to causes, with the caveat that causality maybe sometimes doesn't exist" is so hard.

https://thomasp.vivaldi.net/2023/07/28/what-happened-to-vivaldi-social/

What happened to Vivaldi Social? | Thomas Pike’s other blog

A deep dive into the events of Saturday 8 July 2023, when user accounts started disappearing from the Vivaldi Social Mastodon instance.

Thomas Pike’s other blog

Show thread

Daniel Schwarz

Jul 29, 2023

@mhoye What jumped out at my was the fact that a significant percentage of the Mastodon dev team were engaged over the weekend on this issue. Thanks to them for pitching in, but it doesn’t seem sustainable long term. They need to grow their team or risk burnout and stalling the project. Not sure what plans they have in the works to do that?

Show thread

mhoye

@dschwarz that struck me as well, but to understand that we’d need more insight into the frequency of that kind of event than I have. Black swan events are actually fine if you learn from them, but they should be relatively rare.