Drew GingerichJul 16, 2025
0xDE

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

METR ran a controlled experiment with 16 experienced open-source developers on 246 programming tasks. The developers predicted that the use of AI would speed up their work by 25%, and after the experiment judged that it had sped it up by 20%. But when actually measured, the use of AI slowed down their work by 19%.

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Jul 10, 2025 at 7:49pmWeb
Show threadpenguin42Jul 10, 2025
@11011110 My guess on this, since it's about 'experienced' devs - is that they really know the detailed innards of the codebase they work with and mostly work with it; they know any boiler plate needed well, and most of their work is actually dealing with odd corner cases or new stuff they've already investigated.
Show threadAntonio PatriarcaJul 10, 2025
@penguin42 @11011110 Possibly, but it is interesting that the developers' estimates, both before and after the study, show a significant positive bias toward this technology, even if the measured performance was negative.
Show threadpenguin42Jul 10, 2025
@antopatriarca @11011110 Yeh true - although that seems odd to me; all the experienced devs I know are very negative about AI (often having received a lot of false postivie reports)
Show threadAntonio PatriarcaJul 10, 2025
@penguin42 @11011110 I know a few of them. But more importantly, while you are somewhat forced to experiment with AI nowadays in the corporate world, this is open-source, so they have probably chosen to try these tools.
Show threadWulfy—Speaker to the machinesJul 10, 2025

@11011110

Git Gud!

Seriously: The findings are likely true.
#Vibecoding (presently) may well slow down experienced code monkeys by ~19%

Now, measure BREADTH of coding.
I can #vibecode way faster, in frameworks I don't know.
I design aviation systems.
Have a safe flight 😉

Show threadIstar EldritchJul 11, 2025

@11011110 I gave it a good rest recently. Mainly to Gemini, both flash and pro. I find it ok to generate tests based on specs. Write documentation drafs and so on. It does increase productivity there. But when it comes to code, I found I'm usually faster doing it myself than having to review the generated code and correct the AI at every turn. Reviews take a lot of time. More than writing the code itself. It also leads you down a rabbit hole that is usually not the right solution in the first place. So I feel like it's probably true that it slows you down overall on its current form.

Also, using it with Rust I found it struggles with stuff like lifetimes.