Pan! ⚙️ Jul 12
Asahi Lina (朝日リナ) // nullptr::live

METR: "We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers.

The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't."

https://bsky.app/profile/metr.org/post/3ltn3t3amms2x

(emphasis mine)

METR (@metr.org)

We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers. The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.

Bluesky Social
Jul 12 at 3:48amWeb
Show threadAsahi Lina (朝日リナ) // nullptr::liveJul 12
This is really important work!! If there's one thing AI is good at, it's getting people to drink the kool-aid. Randomized studies don't lie!
Show thread🩷💜 PYNK 💜🩷Jul 12
@lina
it's also very good at getting corporate to lie about its usage ​​
Show threadCorpomancerJul 12

@pynk @lina We just kept adding functionality like keyloggers, audio taps, nonsensical popups, stealth policy changes... Nobody seemed to bat an eye, so we also included secondhand Al exposure as dedicated user base and then doubled that number for shareholder optics.

What could possibility go wrong?

Show threadanemoneJul 12
@lina @grok is this true?
Show threadaburka 🫣Jul 12
@lina tried posting this in the AI discussion channel at work. Got dismissed because it only looked at "experienced developers working on their own projects". The Kool-Aid is strong!
Show threadKrupoJul 12
@aburka @lina lol so did they out themselves as inexperienced developers?
Show threadsanpanJul 12
@aburka @lina They should have tried it against AI prompt engineers with prior block-chain experience and of course double-blinded it with chimpanzees on typewriters, right?
Show thread😀🚲Jul 12
@lina are you sure this is more scientific than the endless flood of AI-generated press releases from AI companies claiming you can pay them $20/month to replace $200k/yr skilled professionals and every anyone will easily become *the* next zuckerberg? FOMO hype train automation as a service!
Show threadmeduz'Jul 24
@lina Really interesting. Another learning highlight that people seem to have forgotten: developers are bad at estimates, what do you think would happen? 
Show threadyoung man yells at the cloudJul 12
@lina god I love it when research aligns with my pre-existing worldview and I can wave it around in everyone's face
Show threadyoung man yells at the cloudJul 12
@lina here is a link to the full paper if anyone is looking: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
Show threadrfJul 12
@bamboombibbitybop @lina man, the paper is really good and i love the digging into the data and reasons for the results!
Show threadRejZoRJul 12
@lina I can see how Ai for coding can be useful if you're learning and we assume Ai gives correct code. So you learn from it only when you need to instead of waiting for hours on social networks to get a reply. But we all know Ai outputs bullshit constantly and people always entirely rely on it to do shit and not just when there is specific need.
Show threadSilicon DryadJul 12

@lina the graph of completion vs time spent is really interesting (https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:dll3hepzq76nymel5c3yt6nk/bafkreifh2o4t47ofomweyal42ed7sffeuchkmlxknybbrwyuse3ayb4hnq@jpeg)

The "AI" tool users show a somewhat steady linear completion over time, whereas the "non-AI" users follow more of an s-curve (I think they're called sigmoids?).

Show threadNtropicJul 12
@lina the breakdown is really interesting. I mostly use it via ollama for code completions, so I don't have a prompting overhead. I ask the chat assistants only when I really don't know how to approach a problem at all.
Show threadAsahi Lina (朝日リナ) // nullptr::liveJul 12
@Ntropic If you are using it as smarter autocomplete then it probably does save time, but only to the extent that it's faster than typing...
Show threaddoragasuJul 12
@lina Also, even if it could maybe speed up unexperienced developers, they will hardly become experienced developers if they rely on the AI and do not learn to solve problems by themselves.
Show threadBeady Belle FanchannelJul 12
@lina overfetishization of productivity
Show threadThe TurtleJul 12
@lina is that like how you feel more interesting when you've done a bunch of coke?
Show threadさよなら皆さんJul 12
@lina boosted bookmarked and starred. Also actually bookmarked as well. This is an important finding!
Show threadMany_sunJul 12
@lina I feel the same way, but it seems there are some issues with the methodology https://mastodon.social/@grimalkina/114837000070700830
Show threadRed LucyJul 12
@lina A BIT LOUDER FOR THE PEOPLE IN THE BACK
Show threadIdahoJul 13
@lina AI can expand your capabilities if used properly, I know it for a fact, my scripts as a sysadmin got a whole lot more complex and better the day I started using chatGPT...