Mastodawn

Even though they're technically not contradictory, I'm not sure how to reconcile the following two things:

1. Nearly every time an outsider says that people are doing X stupidly and could do way better, they're wrong for boring reasons that are obvious to any insider.

2. It seems fairly easy* to find huge wins as an outsider.

At some level, maybe this is like https://danluu.com/p95-skill/, where the median player in an objective-based game regularly loses because they don't touch the objective, but

95%-ile isn't that good

Dan Luu Jan 6, 2023

this seems different in that there are big communities that are into doing a lot of (1).

There are people who are famous for complaining that people do X stupidly but have never fixed any of the stupid things they complain about in major domains or even suggested a remotely plausible fix, there are forums for communities of people who are really "good at" coming up with obviously incorrect reasons things are done stupidly, etc.

IMO, this is much stranger than p95 just not being that good.

Dan Luu Jan 6, 2023

With Twitter being in the news so much, there are examples from Twitter, e.g., https://twitter.com/GergelyOrosz/status/1606354058280779779. The direction this guy was going was silly enough that a number of folks, unsolicited, messaged me in the early days saying that, based on the public tweets, this guy had no chance (h/t https://sigmoid.social/@swyx for images).

That example is only notable because the person actually tried and failed. There are plenty of more absurd claims from people who didn't try, like https://mastodon.social/@danluu/109374630690202023.

Gergely Orosz on Twitter

“This is getting out of hand. Hotz joins Twitter with high aspirations, criticises everything and doesn’t ship a single non-trivial code change in 4 weeks (shipped 4 feature toggles). Hosts a space saying Twitter eng is trash. And now calls a Twitter VPE a “senior” engineer.”

Twitter

Dan Luu Jan 6, 2023

A popular take on George Hotz's failure to ship anything useful at or really make any difference at all has been that you can't really make a difference at a big company which, per (2), seems wrong.

E.g., I shipped https://danluu.com/metrics-analytics/ basically immediately at Twitter and, once it was shipped, there was a clear path to scaling it up to $100M/yr/yr cost savings and, off the top of my head, I can think of multiple domains where an outsider could pick from multiple problems with similar impact.

A simple way to get more value from metrics

Dan Luu Jan 6, 2023

I don't think Hotz's failure at Twitter is particularly interesting but, at a meta level, I wonder why nearly everyone who's railing against industries for doing silly things picks the wrong things and fails, causing big wins to generally come from insiders, when there are so many easy wins around.

See also, Alan Kay on how computers could be 1000x faster (https://www.patreon.com/posts/54329188), Forth supremacists, performance engineers who don't know a domain claiming people should make X faster, etc.

Dan Luu Jan 6, 2023

And BTW, I think I have a decent claim to being an outsider at Twitter in that, except for 1 year writing C++ for Bing, my prior professional experience was all in hardware jobs, and C++ is basically unused at Twitter (there's one important piece of software that Twitter maintains that's in C++, but I didn't touch it at all).

Anyway, even when I look at wins that were simple enough they could've come from outsiders, they almost always come from insiders because outsiders tend to be cranks, e.g.

Dan Luu Jan 6, 2023

, for a simple example, if you look at the bent-shaft paddle (https://twitter.com/danluu/status/1476260570370232323), an extremely obvious-in-retrospect invention, it came from Jensen, the most insider-y insider w.r.t. canoe racing there ever was.

Or to pick a more involved example, there are a handful of "standard" naive ideas that freshman (and Alan Kay) like to throw around to explain why hardware engineers are doing hardware wrong and hardware would be 1000x faster if engineers only did X.

Dan Luu on Twitter

“Some examples of questions: Why weren't canoe paddle innovations made sooner? The bent shaft paddle is attributed to Jensen in the 70s, but we know the canoe is at least 8000 years old. Maybe it's unlikely the bent shaft would've been invented before mechanics?”

Twitter

Dan Luu Jan 6, 2023

Yossi Kreinin wrote a nice series of posts explaining a set of naive misconceptions in https://yosefk.com/blog/the-high-level-cpu-challenge.html, https://yosefk.com/blog/high-level-cpu-follow-up.html, and https://yosefk.com/blog/its-done-in-hardware-so-its-cheap.html.

One person followed up with "neural nets in hardware worked out, so weren't the high-level CPU people right?". I can see how someone would say that, but if you look at the early successes, those all came from insiders.

Yossi himself (VP and Fellow at Intel/Mobileye) was involved in one of the first at-scale production successes.

The "high-level CPU" challenge

Dan Luu Jan 6, 2023

The other big early success was the TPU, which was Andy Phelps's 20%-time project. Before Google, he was a distinguished hardware engineer at Sun. I was the 2nd engineer at the project and had previously only worked in hardware. The 3rd engineer, Greg Thorson, was a long-time Cray hardware engineer, etc.

And if you look at the approaches software folks proposed, they were as hopeless as Hotz, "J" and "O" in https://mastodon.social/@danluu/109374630690202023, etc., just absolute nonsense.

Dan Luu Jan 6, 2023

I was chatting with Yossi about a failed attempt that was contemporary with the TPU but earlier; his comment on the paper was "from just [redacted] in the title, I could tell you they were going to fail".

One of the responses to https://twitter.com/altluu/status/1594372539568250881 was that outsiders can see all sorts of inefficiencies because things are so bad. While I agree that's true in principle, people who make a big deal about being smart outsiders seem to generally have the same 6 boring ideas every freshman has.

Dan Luu on Twitter

“@ohabryka @danluu What is it about the LW/rat/etc. community that causes so many people to have so much confidence in their opinions on topics they clearly know nothing about? Per capita, LW famously punches above its weight. Is there a community norm that causes this? Why would that be?”

Twitter

Dan Luu Jan 6, 2023

My guess for one reason there's a discrepancy between (1) and (2) is that the people who are famous for talking about how stupidly things are done have totally the wrong meta approach and that they end up with these communities that grow up around them that copy the same doomed meta approaches.

The people who are famous for this stuff are bombastic, dogmatic, and generally do not attempt to learn much about the domain before coming up with solutions, which seems like a poor approach.

Dan Luu Jan 6, 2023

*now to explain this asterisk:

before joining Twitter, I gave myself something like a 30% chance of succeeding. It's not that I think any outsider should be able to succeed anywhere, it's just that there are enough problems that can be solved that I'd expect to see a lot more successes given out many people loudly proclaim that they can fix company/org/industry X, if only people would listen to them.

Dan Luu Jan 6, 2023

When I joined Twitter, I had one problem in mind that I thought would be low-impact and I had a ~50% chance of being able to ship a solution to: https://twitter.com/danluu/status/705815510479302656

It's obvious that this is inefficient and that fixing it would reduce cost as well as increase engagement/revenue, more than enough to cover my comp, but not enough to really move the needle.

But I figured it made sense to try for an easy win and then look for other ones before doing anything big/risky.

Dan Luu on Twitter

“OMG twitter made my image larger while reducing the quality. PNG after trimage: 59K. Twitter's post-processed JPEG: 72K.”

Twitter

Dan Luu Jan 6, 2023

When I joined, it turned out that there was already a major effort to fix that and there were plenty of bigger wins that would be easier for someone with my skillset to fix, so I ended up elsewhere.

But, in terms of what to look for, the people who are proposing that Twitter do X aren't picking obvious wins; they often pick areas where, if you spend 2 minutes talking to someone, you'd understand are fairly complex with lots of subtleties. Exactly the wrong thing for an outsider to touch.

Dan Luu Jan 6, 2023

E.g., "Twitter's cache should be one of the simplest caches in the world to build".

Right, caching, famously one of the easiest problems in computer science, especially at scale for the site wikipedia says is the 4st highest traffic website in the world (probably not actually correct, but still suggestive of not-low traffic).

For non-Twitter examples, also see the "obviously good ideas" in https://twitter.com/danluu/status/1266522850908762112, etc.

Dan Luu on Twitter

“@nitsanw @hillelogram His next item is an interpretation of physics that I think many physicists would say is not obvious at all. AFAICT, the top three items are cultural shibboleths more than anything else?”

Twitter

Dan Luu Jan 6, 2023

People are pretty clearly shooting from the hip in areas they didn't understand at all.

Conversely, when I joined Twitter, I made it a point to understand the landscape, do background reading, etc., e.g., because Twitter runs on the JVM and I'd never really used the JVM, I asked Nitsan Wakart (a JVM expert) for a list of books and papers to read if I wanted to understand the JVM and then read them; since Twitter uses Scala, I learned Scala, etc.

Dan Luu Jan 6, 2023

It turns out, if you go through the reading list Nitsan suggested and then look at Twitter metrics, you can find 8-9 figures a year of profit just lying on the floor waiting to be picked up. That was lucky, but it's a target rich environment, so getting lucky isn't too surprising.

People who assume their smarts means that they can make up for not understanding the domain at all aren't likely to get lucky but are likely to stumble into the same naive, boring, and wrong, ideas every freshman has.

Dan Luu Jan 6, 2023

Another "trick" is just asking people who'd been around for a long time what they thought the problems were.

There's a sense in which the hit rate is low because most people don't connect their ideas to actual impact but, at least with the data infra Twitter has, that's easy to do yourself, and enough people do understand how to evaluate total impact that, if you spend a week chatting with folks, you'll talk to multiple people who can credibly back up the impact of their suggestions.

Dan Luu Jan 6, 2023

Taking a step back, the "heroes" of communities that are into pointing out "the world is mad", "X is stupid", etc., seem really into "smarts" and clever solutions and they're explicitly into IQ being important, as in https://www.patreon.com/posts/62933244.

I think this is totally wrong. IMO, having a respect for domain knowledge, domain complexity, meticulously looking at data, etc., is a much more reliable way to find wins as an outsider than being super smart and reasoning from little to no knowledge.

Joshka Jan 6, 2023

@danluu sounds like you experienced a lot of psychological safety in making some of those wins. That’s not the sort of thing that’s evenly distributed (and perhaps is non existent at the moment in a particular company)

@danluu Wait, if I may, you're saying it looks simple to the people who can't see all of the problem 😁

Gabriel Pettier Jan 6, 2023

@danluu regarding the effect of being an outsider, i think it simply allow less conventional thinking, the more you know about the problem, the more you tend to think like other people working on the problem. Of course, thinking outside the box often leads to dead ends, but if one is careful about understanding enough not to run into obvious walls, chances to contribute something meaningful become much higher.

But yeah, it's a balance between coming with a fresh look, and doing the homework.

Chris Hopkins Jan 6, 2023

@danluu this all sounds very much like the old adage, that 6 months in the lab, can save you 2 hours in the library.

s-ol Jan 6, 2023

@danluu
I guess the question is, how do you "understand the landscape, do background reading, etc" without losing the outsider advantage?

Sasha Jan 6, 2023

@s_ol not to speak for Dan but my experience has very consistently been that the background reading is only a small part of why insiders are insiders. there's a lot of interpersonal cultural stuff and years of work experience that typically accounts for way more of it.

Lars Lehtonen Jan 18, 2023

@danluu is there an extant theory that describes how aggregate smart people are stupid, and aggregate average people are genius?

Adrian Cockcroft Jan 6, 2023

@danluu from the outside you can see the product features that you want to fix. From the inside you can also see the existing implementation, which is probably more complex than you expected. However I think the thing that really slows implementations down is all the other people, the management and ownership structure, processes and culture. It takes much longer to get up to speed and find all the ways people get things done. You can get lucky sometimes.

David Jan 6, 2023

@danluu related thought: my most impactful "obvious" outsider ideas have never gone like:

"Me: let's try X. Insider: that'll never work! Me: hah I have proved you wrong!"

Instead it's always been more like:

"Me: let's try X. Insider: Yeah, X keeps getting bumped down the priority list but maybe you could make a case with data / oh X failed when we tried two years ago but maybe Y has changed / hmm I don't see how you'd get around Z but if you can sounds great. Me: cool solved the last bit!"

Nat Bennett Jan 8, 2023

I think basically the dynamic you're describing (new folks on a team can and do make big improvements quickly, people making big statements about "X is stupid" from the outside are always wrong and fail come down to two factors.

Nat Bennett Jan 8, 2023

@danluu (1) The main reason that folks don't make the easy obvious improvements isn't that they don't *see* them, it's that they've gotten used to them-- or are busy with other things.

So there's always stuff that someone who's new (and therefore isn't already working on stuff) can notice and start improving pretty quickly.

Nat Bennett Jan 8, 2023

@danluu (2) Out of the range of all possible interventions that new person could pick, they'll have dramatically more success if they pick something that the existing team sees as a problem. And that takes some time to detect!

Not necessarily a lot of time but you need to spend at least a few days talking to folks and getting a sense of what they value before you can make the distinction. And you absolutely can't do it from the outside.

@danluu yeah, this is honestly my take on the problem you pose in this thread. people are bad at taking the time to look & listen to the problem, and instead go straight to the behaviours that are socially rewarded.

@danluu thinking a little more : the problem appears worse because those behaviours *are* socially rewarded. ask a random engineer for what they'd do to try to optimize Twitter & you'll likely enough get a decent response. but when you look at the self selecting pool of people loudly talking about what they'd fix at an organisation they're not in & a situation they have little context for... the hitrate is going to be lower.

@danluu but it looks like being loud & wrong is working out for those people, so it's not clear what the incentives are for them to stop.

Julien Kirch Jan 6, 2023

@danluu do you have any opinion about how the JVM is when trying to work on performance analysis as you did?

Sam Minnée Jan 6, 2023

@danluu the LessWrong community seems to have as a foundational principle “everyone’s cognition is trash, except ours” and despite any other benefits of their approach it basically makes them operate much like conspiracy theorists.

The idea that, even with careful work by intelligent people, you might still get bad outcomes, is anathema to them. It’s far easier to assume that either the people weren’t intelligent or the work wasn’t careful.

Raunak Jan 6, 2023

@danluu I am interested in learning more about the JVM (beyond the docs on Oracle). Can you share the list of papers and books for understanding the JVM better?

Tim Pote Jan 6, 2023

@danluu Well out with it. What's the list?

Zeke Arany-Lucas Jan 6, 2023

ultracrepidarian - noun, adjective
 “person who offers opinions beyond their own knowledge. It can also be used as an adjective describing such a person. This word is used in situations when someone is speaking as an authority on a subject that they have only limited knowledge of.”

A. R. Younce Jan 6, 2023

@danluu “Smart” as an identity is basically paralyzing even for very smart people. Humility isn’t fun and few want to admit they’re bored by the details. Hot takes that reinforce one’s identity are basically low-hanging fruit for those that can’t handle failure.
Not to be overly pessimistic but I do wonder if most of the people who fall into the identity trap are “smart” enough to realize they’ve hamstrung themselves.

Maybe some of them are the wrong things. But there have been many things that took Twitter many years to implement. Many of which other large tech companies did in way less time.

Why did it take years to increase the character limit?

Why did it take years for an edit button to be introduced?

Why can people still not disable QTs?

Some of this is being careful to consider harms, but some of seems like fear of change.

@danluu where's the business model in that?

Jack Mott Jan 6, 2023

@danluu Seeing George turn into an Elon simp was really sad. :(

Hazel Weakly Jan 6, 2023

@danluu I read your thread and I've been thinking about this and I think there's two separate things going on (that are probably related)

- outsiders have no internal context, and insiders have no external context, so neither have a complete picture of the system and both are essentially aimlessly poking at the system

- effective and efficient information discovery isn't even close to solved. I think this is the more clearer instance of the problem

Hazel Weakly Jan 6, 2023

@danluu one example of the information sharing thing that I think about a *lot* is the following paradox:

- it's very difficult to hire talented people and retain them
- it's very difficult to be hired at an engaging company as a talented person

These sound almost identical to your first comment to me, which is why I think that the information discovery thing is what's happening.

Hazel Weakly Jan 6, 2023

@danluu it feels like the difficulty is deceptive. Low hanging fruit is there still precisely because it hasn't been found and picked yet. Being able to methodically scale up a system continuously *and* continually improve the ability to look at it from novel angles feels akin to discovering a way to efficiently search proof spaces.

But perhaps I'm wrong. I'd kind of like to be, to be honest. Would supportive and encouraging environments magically fix things? Something else?

OpenDNA⚙️Jan 6, 2023

@danluu The slip, I suspect, is outsiders expecting that a specialty is exactly like something they know well, instead of marginally related at best.

I spent time in the trucking industry, as a driver and a manager. Because any accidents or moving violations are effectively disqualifying, every commercial truck driver is in the top 95% of drivers. What distinguishes productivity among drivers isn't skill at driving - or hours spent (because HOS) - but geography and market niche.

OpenDNA⚙️Jan 6, 2023

@danluu The belief that driving a truck long distance is just like commuting in a car, but longer, leads outsiders to come up with obvious solutions which insiders understand are silly.

"AI electric trucks will replace diesel and drivers within 10 years" is one of my favorites, because it presumes that self-driving vehicles or electric vehickes are the Hard Problem, so self-driving cars means self-driving trucks. To outsiders, the replacement is obvious.

OpenDNA⚙️Jan 6, 2023

@danluu To insiders, there are a host of other, greater, barriers which are obvious. They range from the capital cost to replace equipment decades ahead of schedule to the mechanical challenges of winter (and summer) weather across America to the geographical proximity of service techs.

i.e. I had a driver make the "stupid" mistake of turning off the engine to sleep, causing the engine to freeze. How will computers and batteries perform at <10°C without diesel's "waste" heat?

hyperpape Jan 6, 2023

@danluu Without snark, I think it’s easy if you have good ideas, and hard if you don’t.

Structurally, good ideas are disproportionately clustered in places people aren’t looking or where there’s an obstacle to implementation.

Mostly Harmless (Cam Johnson)Jan 6, 2023

@danluu It was analyzed by Karl Popper in The Open Society and its Enemies, and Scott in Seeing like a State. Every field has insane complexity, because the real world is complex, but there’s some abstractions we can draw to simplify the complexity. Outsiders like Hotz are great at knowing the general rules, but always massively underestimate the unique complexities of a given case. It takes time to understand those, and insiders have spent the time.

John Feminella 🌠Jan 6, 2023

@danluu The reconciliation for me is pretty straightforward: outsiders have low context and institutional memory/bias (leading to 1) but are also free of sticky assumptions (leading to 2).

Jeremy Fitzhardinge Jan 7, 2023

People who are very close to a system and not thinking broadly will ask "How can I make this system faster?" If the system has already undergone multiple passes of optimization there won't be much left to win. Not many degrees of freedom.

Jeremy Fitzhardinge Jan 7, 2023

@danluu
Someone familiar with the domain but more removed will ask "how do I solve this problem better?" which focues on the higher-level - not just one system, but likely systems of systems. They're less beholden to history, and there's a lot more degrees of freedom to make improvements.

They're an "outsider", but not necessarily *that* outside.

Jeremy Fitzhardinge Jan 7, 2023

Someone who is a complete outsider doesn't even know what the problems are, so their "solutions" are nonsense. We see this with all the re-engineer Twitter reply guys who have literally no idea what Twitter is as a business, let alone what technical mechanisms are needed to implement it.

Similarly with the math and physics cranks who are trying to redefine pi or design perpetual motion machines.

chozu v1.2.0 Jan 14, 2023

@danluu it's hard to get to the point where people trust your judgement as an outsider, and lots of people absorb culture and implicit limitations and so on as they become an insider, even if they cared about something else

I remember someone (idiosyncratic pro-gamer turned coach due to bad eyesight who repeatedly broke the meta or innovated a strategy in some way) in a game, talking about something like "Yeah, it doesn't matter if these coaches say something that's true or mathematically correct by the rules of the game, because the players don't trust them, the coaches can't prove it to the players because they aren't good at the game. The difference is when *I* say some build order or strategy is OP is that I can play against the players and show it to them, even if I'm not quite at their level, I can still trade blows with them"

which solves the "oh, no one (good) does that" problem -- you just do it yourself, then it's no longer true that no one good does it

lots of people with feasible ideas aren't in a position to implement them, and it's probably not the case that the path to having that power is especially easy going for those kinds of people (possibly it's harder, don't see any reason why it would be easier, since the premise is that the ideas aren't "mainstream" enough that people have already done it)

chozu v1.2.0 Jan 14, 2023

@danluu also https://danluu.com/corrections/ has "effective kills the site", presumably this should be "effectively"

Major errors on this blog (and their corrections)