n o ren
AI & Technology

The Speedup That Wasn't There

Experienced developers using AI assistants finished tasks nineteen percent slower, and still believed the tools had sped them up.

2026-08-312 min read

Self-report is the instrument almost every organization uses to evaluate AI tooling. Someone tries the assistant, says it helps, and the rollout follows. A trial published by METR in July 2025 is worth attention because it put that instrument on trial too, and the instrument failed.

The design was unusually honest. Sixteen experienced open-source developers took on 246 real tasks drawn from their own mature repositories — code they knew well — randomly assigned, task by task, to allow AI assistance or forbid it. Beforehand the developers expected the AI condition to make them roughly a quarter faster. It made them about nineteen percent slower. The part that should unsettle anyone making a tooling decision is what happened next: having just lived through the slowdown, the developers still reported that the AI had made them faster.

The gap is not stupidity, it is what the two conditions feel like from the inside. Without the assistant you sit and think, and thinking registers as effort with nothing to show for it. With the assistant you are always doing something — reading a suggestion, accepting it, correcting it, prompting again. Time spent reviewing generated code that turns out to be wrong feels like progress in a way that time spent staring at a function does not. Fluency gets read as speed, and the measurement most teams actually take is a survey, which is taken in precisely the units the illusion corrupts.

None of this means the tools do not work. It means the population studied was narrow and specific: experts, inside codebases they already knew deeply, where the model's context was thinner than their own. That is close to the worst case for AI assistance — and it is also the population most likely to be asked when a company decides what to buy.

Key insights
Sixteen experienced developers were about nineteen percent slower with AI assistance in their own repositories.
They predicted a speedup beforehand and still reported one afterward.
Self-reported productivity is the single measure this effect corrupts most reliably.
Why it matters

If you choose AI tooling by asking people whether it helped, you are using the one instrument this effect reliably distorts.

The work where assistance genuinely wins is not the work your most senior engineers are reporting on.

Use this tomorrow
1
Pick two tickets of similar size this week, do one with the assistant switched off, and write down the start and finish timestamps for both before you form an opinion.
2
Ask three engineers what percentage the assistant speeds them up, then pull the actual cycle times on their last ten closed tickets and put the two numbers side by side.
Go deeper

METR published the trial in July 2025 as "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." Randomization ran per task rather than per developer, which removes the usual objection that faster people simply chose the AI condition. The authors were careful about scope and framed the result as a finding about this population, not a verdict on AI coding tools generally. It stands out because most evidence in this area is either benchmark performance or survey data, and this was neither.

The result has a much older sibling. Lisanne Bainbridge's 1983 paper "Ironies of Automation" argued that automating the easy parts of a job leaves the operator holding the hard parts, with less practice and worse conditions for handling them. Her setting was process control rather than software, and her concern was failure handling rather than throughput. What the two share is the observation that the costs of automation land somewhere other than where its benefits get measured.