In response to criticism of the Fediverse I've read today, and to some very uninformed comments I've read on Twitter from Musk supporters about firing most of Twitter's employees, I want to talk about what I know about the technical side of running a social service.

The short answer is: it ain't easy and it ain't cheap.

First, a crude analogy for non-technical people.

Imagine you have a car, the Twittermobile. If it's not working, a mechanic opens the hood and looks at the engine. But unlike real cars, with standardized internal combustion engines, this engine is entirely custom.

You're lucky if there's sparkplugs and a carburetor in here. Almost every piece is unique, built by a team of automotive techs, and fitted together in a complex arrangement nobody fully understands.

Why the hell is every piece unique? Because the Twittermobile is unique. It's a racing car, meant to go at ultra-high speeds, all day long. Other high-performance cars like the Youtube Coupe will have solved similar challenges, but under the hood will be very different.

If you see people talking about "microservices" and databases and all that, each of those things is a component in this engine.

At these scales, the way you work with your software changes. I can get my tires rotated every 5000 miles on my commuter car. Formula 1 cars replace their tires every 20-50 laps around the track.

If one of the servers that runs a big service fails, you don't typically log onto it and find out why. You reboot it, and if that fails, you re-image it with a fresh installation, and move on.

Tremendous amounts of engineering go into making that possible.

When each of your components is either custom made, or at least a heavily modified version of something off the shelf, you need technical expertise to keep them running - and improving them, which you must do all the time, because your business side demands it.

Much of the knowledge of those systems is locked in the heads of your employees, because we've stopped caring about documentation and technical writers. But that's another story.

I have never seen any of Twitter's internals, but Mastodon's are freely available (https://github.com/mastodon/mastodon) and I've been a Ruby guy for years, so I feel comfortable reading it. I've read the criticism of ActivityPub from @atax1a and I have nothing to add there except that the bigger a service gets, the more of that Actual Work is required. Oh, and Kerberos everywhere would have been great.
GitHub - mastodon/mastodon: Your self-hosted, globally interconnected microblogging community

Your self-hosted, globally interconnected microblogging community - mastodon/mastodon

GitHub

Second, an important note about content moderation from someone who has not worked as a content moderator, but has supported them.

Mods are underappreciated and undervalued. They wade into waist-deep muck every day, and if they're lucky they get paid for it. But mods absolutely, definitely need good tools to do their jobs.

Content moderation is inherently a human job. You're looking for things that are hurtful to human beings, and there's no algorithm that can come close to doing that well.

But engineers can write tools to help that search. Word searches. Blacklists. Flexible ban systems. Tools to flag content, and tools to review flags.

It's vital that as Mastodon and the Fediverse figure out a moderation strategy that engineers be ready to implement the tools that those moderators (individually or as a collective) ask for. The people who have done this for a long time know what they need. There's no engineering solution to toxic content, but there is engineering work nonetheless.

I have hope for Mastodon as a platform. I want to help make it succeed. But it has to succeed for EVERYBODY - the marginalized, the endangered, the vulnerable. People must absolutely be safe.

That's hard to do. But it must be done.

(End thread, thanks for reading)

@astralfrontier

By this analogy, EMu has fired not just the pit crew but also the supply chain for custom tires and engine parts

@trochee That's absolutely the case, yeah. The Musk fans who are saying "yeah you only need 250-300 devs, it's just an app" feel like contractors or juniors who've never had to deal with the logistics of software. Testing, deployment, log consolidation, etc. don't just magically happen.
@astralfrontier @trochee 250-300 people would be the skeleton crew just to keep their data centres helmed 24/7. Big apps make it look easy because they've learned from their many failures and their multitudes of staff to stick their collective fingers in the leaky dam. Too few fingers and the dam eventually fails.
@astralfrontier @trochee Oh, you give them way more credence than I do. Most of these people have never worked in a technical role in their lives.