Matthew GarrettNov 18, 2022
I feel like Elon stans asserting that Twitter will be just fine with no meaningful engineering left are not going to respond well to "Facts don't care about your feelings" as it all gradually falls apart
Show threadMatthew GarrettNov 18, 2022
What's obvious to basically everyone in the industry but is probably surprising to everyone else is that twitter is a harder problem than getting a rocket to the moon. One of these problems only involves physics, the other involves humans.
Show threadMatthew Garrett
What's depressing is that this will inevitably turn into criticism of Twitter's dev team for not building software that was resilient against the entire SRE team quitting, as if maybe if they'd just been more competent all the scaling and maintenance issues would just go away
Nov 18, 2022 at 10:46amWeb
Show threadMatthew GarrettNov 18, 2022
Saying Twitter should work fine without SREs and if it doesn't the devs were incompetent is like saying Apollo 13 should have been fine without ground control because Rockwell should have designed the oxygen tanks better
Show threadWonkyNov 18, 2022

@mjg59 I have seen this attitude time and time again during cost cutting.

"We don't need this ops team, its too expensive"

"Why do we need dev and staging environments when we can just apply changes to prod"

Zero percent chance of ending well, even with a mature platform.

Show thread🦄 I Make Robots 🦄Nov 20, 2022
@wonky @mjg59 I haaaaaaate working with teams that have no integ stage. Their jank level is invariably off the charts.
Show threadTrismegistusMxNov 20, 2022

@wonky @mjg59 remember when engineers told NASA that the o-rings were going to fail on the Challenger and the Bean counters said it was worth the risk?

Elon only listens to Bean counters.

Show threadWonkyNov 20, 2022
@mjg59 @TrismegistusMx this is a great analogy.
Show threadWonkyNov 18, 2022

@mjg59

Inevitably what is going to happen is that there will be failures that require tribal knowledge to fix. There will be failed releases that also fall rollback. There will be core infrastructure and dependent service incompatibilities as the configuration drift expands.

Show threadgaytabaseNov 18, 2022
@mjg59 if they were just a bit smarter they would have designed a system that could deal with the threat of being bought out by a moron
Show threadgaytabaseNov 18, 2022
@mjg59 i'm wondering if he'll decide the server bill is too big next and start limiting compute resources.
Show threadAlan BellNov 18, 2022
@dysfun @mjg59 there probably are compute time and performance benefits to be had if he rips out lots of user tracking stuff. That would work great as a strategy until he has to sell that data to corporate partners and he doesn't have it.
Show threadStephan GarlandNov 18, 2022

@dysfun @mjg59
"I found out we aren't using CPU limits on our pods! Now that they're all constrained, we can spin down a bunch of these nodes."

-Musk, probably.

Show threadHenry Andrews (he/they)Nov 20, 2022

@sgarland @dysfun @mjg59

Would those be Tide Pods? 🌀😜

Show threadLambdaDuckNov 20, 2022
@mjg59 @dysfun he’s already expressed plans to shut down one of the three data centers. who needs redundancy?
Show threadAnnNov 20, 2022

@LambdaDuck @mjg59 @dysfun

Of course he has 🤦‍♀️

Show threadAndrew TimmesNov 20, 2022
@dysfun @mjg59 he already has
Show threadAriadne Conill 🐰Nov 18, 2022

@dysfun @mjg59

i think back to 2007 when @blaine wanted Twttr to interoperate with StatusNet, and how he built Webfinger to help realize that goal.

Show thread64kbNov 18, 2022
@dysfun @mjg59 I don't see any privately owned company prioritising this.
OTOH, when I've worked in govt departments, it feels like the whole system is organised around this principle.
Show threadBen A L JemmettNov 18, 2022
@dysfun @mjg59 The problem with designing things to be rich-moron-proof is that the universe promptly builds a bigger and richer moron...
Show threadKevin LydaNov 18, 2022
@mjg59 hey, as someone who has been an SRE I'm mad impressed its stayed up this long. And you're right, clueless people won't get that but then we're all used to that by this point.
Show threadjacquelines 🌟Nov 18, 2022
@mjg59 i would easily make my software resilient to this, using a little known tool i like to call ‘rm’
Show threadPJ "Vote Green" CoffeyNov 20, 2022
@jacqueline @mjg59 -rm Elon.exe perhaps?
Show threadLars Marowsky-Brée 😷Nov 18, 2022
@mjg59 Twitter's scaling and maintenance issues are in the process of going away though
Show threadRupert ReynoldsNov 18, 2022
@mjg59 I think (hope) there is an understanding that large systems that change frequently will always need some hand-holding, even when changes are stopped for a while.
Show threadGreg TaylorNov 18, 2022
@RupertReynolds @mjg59 especially when you run your own data centers like we did/do at Twitter.
Show threadMonsieur Valentine 🇺🇦Nov 18, 2022
@mjg59 Twitter proves that Scala rules. It was the biggest Scala shop in SV. Maybe still is
Show threadpotatoofdestinyNov 20, 2022
@mjg59 the fact that anything is running at all at this point is a testament to just how good at it they were