this seems different in that there are big communities that are into doing a lot of (1).

There are people who are famous for complaining that people do X stupidly but have never fixed any of the stupid things they complain about in major domains or even suggested a remotely plausible fix, there are forums for communities of people who are really "good at" coming up with obviously incorrect reasons things are done stupidly, etc.

IMO, this is much stranger than p95 just not being that good.

With Twitter being in the news so much, there are examples from Twitter, e.g., https://twitter.com/GergelyOrosz/status/1606354058280779779. The direction this guy was going was silly enough that a number of folks, unsolicited, messaged me in the early days saying that, based on the public tweets, this guy had no chance (h/t https://sigmoid.social/@swyx for images).

That example is only notable because the person actually tried and failed. There are plenty of more absurd claims from people who didn't try, like https://mastodon.social/@danluu/109374630690202023.

A popular take on George Hotz's failure to ship anything useful at or really make any difference at all has been that you can't really make a difference at a big company which, per (2), seems wrong.

E.g., I shipped https://danluu.com/metrics-analytics/ basically immediately at Twitter and, once it was shipped, there was a clear path to scaling it up to $100M/yr/yr cost savings and, off the top of my head, I can think of multiple domains where an outsider could pick from multiple problems with similar impact.

I don't think Hotz's failure at Twitter is particularly interesting but, at a meta level, I wonder why nearly everyone who's railing against industries for doing silly things picks the wrong things and fails, causing big wins to generally come from insiders, when there are so many easy wins around.

See also, Alan Kay on how computers could be 1000x faster (https://www.patreon.com/posts/54329188), Forth supremacists, performance engineers who don't know a domain claiming people should make X faster, etc.

And BTW, I think I have a decent claim to being an outsider at Twitter in that, except for 1 year writing C++ for Bing, my prior professional experience was all in hardware jobs, and C++ is basically unused at Twitter (there's one important piece of software that Twitter maintains that's in C++, but I didn't touch it at all).

Anyway, even when I look at wins that were simple enough they could've come from outsiders, they almost always come from insiders because outsiders tend to be cranks, e.g.

, for a simple example, if you look at the bent-shaft paddle (https://twitter.com/danluu/status/1476260570370232323), an extremely obvious-in-retrospect invention, it came from Jensen, the most insider-y insider w.r.t. canoe racing there ever was.

Or to pick a more involved example, there are a handful of "standard" naive ideas that freshman (and Alan Kay) like to throw around to explain why hardware engineers are doing hardware wrong and hardware would be 1000x faster if engineers only did X.

Yossi Kreinin wrote a nice series of posts explaining a set of naive misconceptions in https://yosefk.com/blog/the-high-level-cpu-challenge.html, https://yosefk.com/blog/high-level-cpu-follow-up.html, and https://yosefk.com/blog/its-done-in-hardware-so-its-cheap.html.

One person followed up with "neural nets in hardware worked out, so weren't the high-level CPU people right?". I can see how someone would say that, but if you look at the early successes, those all came from insiders.

Yossi himself (VP and Fellow at Intel/Mobileye) was involved in one of the first at-scale production successes.

The other big early success was the TPU, which was Andy Phelps's 20%-time project. Before Google, he was a distinguished hardware engineer at Sun. I was the 2nd engineer at the project and had previously only worked in hardware. The 3rd engineer, Greg Thorson, was a long-time Cray hardware engineer, etc.

And if you look at the approaches software folks proposed, they were as hopeless as Hotz, "J" and "O" in https://mastodon.social/@danluu/109374630690202023, etc., just absolute nonsense.

I was chatting with Yossi about a failed attempt that was contemporary with the TPU but earlier; his comment on the paper was "from just [redacted] in the title, I could tell you they were going to fail".

One of the responses to https://twitter.com/altluu/status/1594372539568250881 was that outsiders can see all sorts of inefficiencies because things are so bad. While I agree that's true in principle, people who make a big deal about being smart outsiders seem to generally have the same 6 boring ideas every freshman has.

My guess for one reason there's a discrepancy between (1) and (2) is that the people who are famous for talking about how stupidly things are done have totally the wrong meta approach and that they end up with these communities that grow up around them that copy the same doomed meta approaches.

The people who are famous for this stuff are bombastic, dogmatic, and generally do not attempt to learn much about the domain before coming up with solutions, which seems like a poor approach.

*now to explain this asterisk:

before joining Twitter, I gave myself something like a 30% chance of succeeding. It's not that I think any outsider should be able to succeed anywhere, it's just that there are enough problems that can be solved that I'd expect to see a lot more successes given out many people loudly proclaim that they can fix company/org/industry X, if only people would listen to them.

When I joined Twitter, I had one problem in mind that I thought would be low-impact and I had a ~50% chance of being able to ship a solution to: https://twitter.com/danluu/status/705815510479302656

It's obvious that this is inefficient and that fixing it would reduce cost as well as increase engagement/revenue, more than enough to cover my comp, but not enough to really move the needle.

But I figured it made sense to try for an easy win and then look for other ones before doing anything big/risky.

When I joined, it turned out that there was already a major effort to fix that and there were plenty of bigger wins that would be easier for someone with my skillset to fix, so I ended up elsewhere.

But, in terms of what to look for, the people who are proposing that Twitter do X aren't picking obvious wins; they often pick areas where, if you spend 2 minutes talking to someone, you'd understand are fairly complex with lots of subtleties. Exactly the wrong thing for an outsider to touch.

E.g., "Twitter's cache should be one of the simplest caches in the world to build".

Right, caching, famously one of the easiest problems in computer science, especially at scale for the site wikipedia says is the 4st highest traffic website in the world (probably not actually correct, but still suggestive of not-low traffic).

For non-Twitter examples, also see the "obviously good ideas" in https://twitter.com/danluu/status/1266522850908762112, etc.

People are pretty clearly shooting from the hip in areas they didn't understand at all.

Conversely, when I joined Twitter, I made it a point to understand the landscape, do background reading, etc., e.g., because Twitter runs on the JVM and I'd never really used the JVM, I asked Nitsan Wakart (a JVM expert) for a list of books and papers to read if I wanted to understand the JVM and then read them; since Twitter uses Scala, I learned Scala, etc.

It turns out, if you go through the reading list Nitsan suggested and then look at Twitter metrics, you can find 8-9 figures a year of profit just lying on the floor waiting to be picked up. That was lucky, but it's a target rich environment, so getting lucky isn't too surprising.

People who assume their smarts means that they can make up for not understanding the domain at all aren't likely to get lucky but are likely to stumble into the same naive, boring, and wrong, ideas every freshman has.

Another "trick" is just asking people who'd been around for a long time what they thought the problems were.

There's a sense in which the hit rate is low because most people don't connect their ideas to actual impact but, at least with the data infra Twitter has, that's easy to do yourself, and enough people do understand how to evaluate total impact that, if you spend a week chatting with folks, you'll talk to multiple people who can credibly back up the impact of their suggestions.

@danluu

I think basically the dynamic you're describing (new folks on a team can and do make big improvements quickly, people making big statements about "X is stupid" from the outside are always wrong and fail come down to two factors.

@danluu (1) The main reason that folks don't make the easy obvious improvements isn't that they don't *see* them, it's that they've gotten used to them-- or are busy with other things.

So there's always stuff that someone who's new (and therefore isn't already working on stuff) can notice and start improving pretty quickly.

@danluu (2) Out of the range of all possible interventions that new person could pick, they'll have dramatically more success if they pick something that the existing team sees as a problem. And that takes some time to detect!

Not necessarily a lot of time but you need to spend at least a few days talking to folks and getting a sense of what they value before you can make the distinction. And you absolutely can't do it from the outside.