0xDEJul 10, 2025

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

METR ran a controlled experiment with 16 experienced open-source developers on 246 programming tasks. The developers predicted that the use of AI would speed up their work by 25%, and after the experiment judged that it had sped it up by 20%. But when actually measured, the use of AI slowed down their work by 19%.

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Show threadpenguin42Jul 10, 2025
@11011110 My guess on this, since it's about 'experienced' devs - is that they really know the detailed innards of the codebase they work with and mostly work with it; they know any boiler plate needed well, and most of their work is actually dealing with odd corner cases or new stuff they've already investigated.
Show threadAntonio Patriarca
@penguin42 @11011110 Possibly, but it is interesting that the developers' estimates, both before and after the study, show a significant positive bias toward this technology, even if the measured performance was negative.
Jul 10, 2025 at 8:51pmWeb
Show threadpenguin42Jul 10, 2025
@antopatriarca @11011110 Yeh true - although that seems odd to me; all the experienced devs I know are very negative about AI (often having received a lot of false postivie reports)
Show threadAntonio PatriarcaJul 10, 2025
@penguin42 @11011110 I know a few of them. But more importantly, while you are somewhat forced to experiment with AI nowadays in the corporate world, this is open-source, so they have probably chosen to try these tools.