lionel16Nov 20, 2022
Justin Hendrix
Elon Musk tweeted an image on a whiteboard from a "Twitter HQ code review" that wrapped up early Saturday. The image has attracted nearly 12k upvotes and over 1,000 comments over at the subreddit r/ProgrammerHumor, under the title "Twitter for Dummies". The image itself is an architecture schematic, not code. I put it on a Miro Board and Luke Dubois and Mark Hansen helped annotate it. If you're curious how Twitter's bits work, check it out here: https://miro.com/app/board/uXjVPBnTJmM=/?share_link_id=748653261357
Twitter Architecture

miro.com
Nov 19, 2022 at 6:25pmWeb
Show threadDavid KampmannNov 19, 2022
@justinhendrix so three weeks in, and after firing a vast majority of the company, he finally decides to see how Twitter works. A+
Show threadgenevieveNov 19, 2022
@mrkampmann @justinhendrix and after shutting down 80% of the services!! did they just throw darts blindfolded to figure out what to turn off?
Show threadcolleenNov 19, 2022
@justinhendrix @mrkampmann my observation exactly. Isn’t this the first thing you do? Then win the hearts and souls of the devs. Then cull ruthlessly?
Show threadScott Hanselman ✅👸🏽🐝🌮Nov 19, 2022
@justinhendrix this…looks like a pretty normal distributed system. Guess I’m not sure what I’d expect. It’s a thing like the others things we’ve worked on.
Show threadMatthew WaltonNov 19, 2022
@shanselman @justinhendrix it's kind of reassuring to me. I feel like this is an architecture I might have been involved in.
Show threadShriram NatarajanNov 19, 2022
@justinhendrix this feels pretty standard - possibly better architected than most distributed systems out there 😂
Show threadmusicologymanNov 19, 2022

The fact the Musk called this a"code review" itself is bit odd, if not downright ludicrous. This is more like the outcome of an architecture review, and a very high level one at that. That Musk would call this a "code review" leads me to wonder how much he actually knows about building secure, reliable, and maintainable software.

In short, I don't think Musk has a clue about what a code review is, let alone how is supposed to work.

Show threadFrancis ☑️Nov 19, 2022
@musicologyman complete lack of code in the code review for a start
Show threadAllen HolubNov 19, 2022
@justinhendrix This picture is only a small part of the overall system, ofc. Unless I'm missing something, It doesn't look at admin, moderation, history generation, support, &c. Your annotations are great!
Show threadAshtonNov 20, 2022
@allenholub @justinhendrix for whatever reason, it seems like it’s the front end that he has a bee in his bonnet over.
Show threadrfvgyhnNov 19, 2022

@justinhendrix Anytime I see an architecture diagram, I can't help but think of the classic microservices video.

https://www.youtube.com/watch?v=y8OnoxKotPQ

Microservices

it's because of the way our backend works // merch: https://merch.krazam.tv// https://www.instagram.com/krazam.tv // https://twitter.com/krazamtv

YouTube
Show threadLeonardraNov 19, 2022
@justinhendrix This is a clear redesign of the architecture by bytebytego
Show threadAnthony PiscitelliNov 19, 2022
@justinhendrix Thank you, what I understood from you notes was very interesting.
Show threadSimeon NedkovNov 19, 2022
@justinhendrix Cool, thanks for doing that. The whole thing looks like a polite “you don’t know what you are talking about” Mr. Musk/boss.
Show threadillepicNov 19, 2022
@justinhendrix "why read the docs when the remains of the team can come in Saturday and show me"
Show threadThe MentorNov 19, 2022
@justinhendrix thanks for sharing. Seems fairly standard with the exception of the TLS API being deprecated - presumably they’re going to flip people across to the GraphQL implementation albeit that feels quite aggressive.
Show threadMichael SokolovNov 19, 2022
@justinhendrix no search?
Show threadDaniel FeldmanNov 20, 2022
@msokolov @justinhendrix earlybird is search
Show threadSörenNov 19, 2022
@justinhendrix @shanselman https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale has some more details as well
The Infrastructure Behind Twitter: Scale

The Infrastructure Behind Twitter: Scale

Show threadcolleenNov 19, 2022
@justinhendrix I laugh at “hydration” every time. Been in tech for 23 years and I’ve heard all thesaurus euphemisms for “load the data” at this point.
Show threadDj PadzenskyNov 19, 2022
@justinhendrix A popular interview question for system architects is, “design Twitter.” As ASP apps go, the architecture of the thing is relatively simple (assuming you blackbox things like timeline algorithms and advertising), but the question goes more toward handling chokepoints and managing performance over millions of hits per second.
Show threadChristie DudleyNov 20, 2022
@justinhendrix This is great and all, but don't you think this is something that should have been done first before anyone was fired?
Show threadKeith SoltysNov 20, 2022
@longobord @justinhendrix There should already have been documentation of all of this. But of course all the technical writers/architects are probably gone.
Show threadChristie DudleyNov 20, 2022
@ksoltys @justinhendrix I don't doubt that there was never a "big picture" view like this back when they had people to do documentation. It is rare that anyone not intimately familiar with one or more systems would need to know it.
Show threadKeith SoltysNov 20, 2022
@longobord @justinhendrix I worked on architectural docs in my last job at the Toronto Stock Exchange. They were mainly for knowledge transfer and to satisfy regulatory oversight requirments. They were also helpful for QA and troubleshooting.
Show threadBlake C. StaceyNov 20, 2022
@justinhendrix 🌟 for the comments about the lighting
Show threadAmormzeeNov 20, 2022
@justinhendrix Loved the miro board addition
Show threadSaagar JhaNov 20, 2022
@justinhendrix Twitter Front End is their network connection multiplexer, not something that users interact with directly
Show threadJeroen BaertNov 20, 2022
@justinhendrix this was an interesting read, thanks!
Show threadMarcus Griep Nov 20, 2022
@justinhendrix I found the commentary on signage building codes and lighting environmental effects to be most illuminating.
Show threadAshtonNov 20, 2022
@justinhendrix it’s kinda funny how this undermines his “android makes thousands of requests” spat that he fired a dude over.
Show threadZeeNov 20, 2022
@justinhendrix @PadreSJ might be interested in this
Show threadEric D. SchabellNov 20, 2022
@justinhendrix @rafelbev why keep bringing that storyline over here? 🐘😆
Show threada1Nov 20, 2022
@justinhendrix this is quite enlightening. Thanks for sharing "Twitter for Dummies" with me (I admit being a Dummie).
Show threadMark Blumberg, Charity LawyerNov 20, 2022
@justinhendrix this is a great tool for reviewing other peoples work. Every government or major corporation that puts out a press release should be reviewed in this way. Adds a lot of context.
Show threadSparkle TeaNov 22, 2022

@justinhendrix @canadiancharity
Reading through this thread, this post exemplifies what I used to use the Quote feature for. It was perfect to say an "aside"; that is, a comment relevant to the main thread but not precisely about the heart of the thread.

When I found people who used the tool to disparage, I'd usually just scroll on by, or block.

And, I agree about the micro board! It reminds me of mind-mapping to take notes.

Show threadJeff ♨️ DarcyNov 20, 2022
@justinhendrix This looks like it encompasses only the top-level components in the main request path. As usual, only a few bits of deeper infra (network, storage, OS, provisioning, name service, auth*n) even get passing mention. Monitoring, alerting, tracing, performance analysis tools, dev tools are omitted entirely. Ditto for "offline" processing, which AIUI is substantial. What percentage of Twitter code does this represent?
Show threadAmit Bahree 🌎💾Nov 20, 2022
@justinhendrix and hey its RPC free 😂
Show threadDavid WeinbergerNov 20, 2022
@justinhendrix Very interesting to a tech hanger-on like me. Thanks!
Show threadGuySoftNov 20, 2022
Wonderful
Show threadJimMcLMetaNov 20, 2022
@justinhendrix Think they are on a path to open-sourcing Twitter?
Show threadLouis MamakosNov 22, 2022

@justinhendrix What's missing from this high-level overview are the infrastructure components. What's doing the orchestration and launching/restarting/deploying all of the dozens/hundreds of application instances? Simple stuff like DHCP servers, DNS servers, load balancers. How does the horizontal scalability work for each of these components? It's the interdependencies that are often the most interesting and brittle.

"Doors and corners kid. That's where they get you." #TheExpanse