Very thoughtful analysis by @grimalkina of the experimental design and results from the recent METR study on “the impact of early-2025 AI on experience open-source developer productivity”.

Are developers slowed down by AI? Evaluating an RCT (?) and what it tells us about developer productivity
Seven different people texted or otherwise messaged me about this study which claims to measure “the impact of early-2025 AI on experience open-source developer productivity.” You know, when I decided to become a psychological scientist I never imagined that “teaching research methods so we can actually evaluate evidence about developers”