Do AI Coding Tools Actually Make Developers Faster? A Review of Every Major Study, 2023 to 2026
Short answer: the evidence is mixed, and it depends heavily on who is coding and what is being measured. Controlled studies on simple or well-scoped tasks found developers finished 21% to 56% faster with AI. A widely cited 2025 trial with experienced open-source developers on their own large codebases found them 19% slower. A follow-up in late 2025 pointed to a speedup, but its authors called the data "very weak evidence".
At the team level, more code is not automatically turning into more delivered software. Google's DORA research and CircleCI's delivery data both found that AI increases activity while instability rises, and developers' trust in AI-generated code fell in 2025 even as usage grew. For India, which GitHub says now has the second-largest developer community in the world, the question of whether AI tools actually help is directly tied to jobs and team productivity.
We reviewed 11 public sources, including randomised controlled trials by METR, Google and Microsoft-affiliated economists, Google's DORA reports, the Stack Overflow 2025 developer survey and CircleCI delivery data, to answer three questions: do AI coding tools make developers faster, why do studies disagree, and what does this mean for Indian developers and teams.
Key findings
- Short, well-defined tasks show large gains. In a 2023 experiment, developers using GitHub Copilot built an HTTP server 55.8% faster than those without it.
- Experienced developers on complex, familiar code saw no gain in 2025. METR's trial found experienced open-source developers took 19% longer with AI, even though they believed AI had made them 20% faster.
- Newer tools probably help more, but it is hard to measure. METR's late-2025 data pointed to 4% to 18% speedups, with wide confidence intervals and serious selection problems.
- Junior developers gain the most in field studies. Three field experiments with 4,867 developers found 26% more completed pull requests per week, with larger benefits for less experienced developers.
- Teams are not yet shipping proportionally more. DORA 2024 linked AI adoption to 1.5% lower throughput and 7.2% lower stability. CircleCI's 2026 data showed main branch throughput down 7% for the average team.
How we reviewed the evidence
We grouped the evidence into three tiers: randomised controlled trials (the strongest evidence for cause and effect), large field experiments inside companies, and surveys or delivery telemetry (which show patterns but not causes). For each study, we note who participated, what task they did, which AI tool and model they used, and what was measured, because those factors explain most of the disagreement.
What the major studies found
| Study | Who and what | Result |
|---|---|---|
| Peng and others (2023) | Recruited developers, build an HTTP server, Copilot | 55.8% faster |
| Field experiments (2022 to 2023) | 4,867 developers at Microsoft, Accenture and a Fortune 100 firm | 26% more pull requests a week |
| Google RCT (2024) | 96 Google engineers, one complex enterprise task | About 21% faster |
| METR (early 2025) | 16 experienced open-source developers, their own repositories | 19% slower |
| METR (late 2025, returning) | 10 developers from the original study | 18% faster, very uncertain |
| METR (late 2025, new) | 47 new developers | 4% faster, very uncertain |
The METR trial: experienced developers were slower
METR's study, published in July 2025, is the most-discussed result. Sixteen experienced developers worked on 246 real issues from large open-source repositories they had contributed to for years, with each task randomly assigned to allow or disallow AI tools. When AI was allowed, they mainly used Cursor Pro with Claude 3.5 and 3.7 Sonnet. Tasks took about 19% longer with AI, with a 95% confidence interval of 2% to 39% slower.
The perception gap is the most striking part. Before starting, the developers expected AI to speed them up by 24%. After finishing, they still believed it had made them about 20% faster, even though the measured result was a slowdown.
The 2026 update: probably faster now, but hard to prove
METR repeated the experiment in late 2025 with 57 developers and more than 800 tasks, and published an update on 24 February 2026. Returning developers showed an estimated 18% speedup (confidence interval from 38% faster to 9% slower), and new developers a 4% speedup (15% faster to 9% slower). METR called this "very weak evidence" for the size of the effect and said the true speedup may be larger, because:
- Many developers declined to join because they did not want to work without AI.
- 30% to 50% of participants said they skipped submitting some tasks they did not want to do without AI.
- Pay fell from $150 to $50 an hour, which may have changed who took part.
- Time tracking broke down when developers ran several AI agents at once.
METR said it is redesigning its experiment. As Rob Bowley's summary of the update puts it, many developers now rely on AI so heavily that they will not work without it, which makes a no-AI comparison group hard to run.
Google and the field experiments: clearer gains
Google's randomised trial with 96 full-time engineers found AI features cut time on a complex enterprise task by about 21%, according to the paper's abstract, with a wide confidence interval. Engineers who spent more hours a day coding gained more.
The largest study, covering 4,867 developers in three field experiments at Microsoft, Accenture and a Fortune 100 electronics company in 2022 and 2023, found that Copilot users completed 26.08% more pull requests a week, according to InfoQ. Less experienced developers were more likely to adopt the tool, keep using it and benefit from it.
Why do the studies disagree?
Reading the studies side by side, four factors explain most of the gap:
- Experience and familiarity. AI helps most when the developer knows less than the model about the task. METR's developers knew their codebases deeply, so AI had less to add and more to get wrong.
- Task type. Building a small server from scratch suits AI well. Fixing issues in a million-line project with strict quality standards does not.
- What is measured. Pull requests and task time measure output. They do not capture review time, rework or bugs found later.
- Tool generation. The field experiments used GPT-3.5-based Copilot, METR's first study used early-2025 tools, and its second used late-2025 agents. Results from one generation may not hold for the next.
What happens at the team level?
| Source | Finding |
|---|---|
| DORA 2024 | AI adoption linked to 1.5% lower throughput and 7.2% lower stability |
| DORA 2025 | Nearly 90% use AI, throughput now positive, instability still rising |
| CircleCI 2026 | Average main branch throughput down 7%, build success at 70.8% |
Google's DORA research, which surveys thousands of technology professionals each year, found in 2024 that delivery throughput was 1.5% lower and delivery stability 7.2% lower where AI had been adopted, and 39% of respondents had little or no trust in AI-generated code, according to InfoQ. In 2025, with nearly 90% of respondents using AI, DORA found a positive link with throughput but continued rising instability, and described AI as "an amplifier" that "magnifies the strengths of high-performing organisations and the dysfunctions of struggling ones".
CircleCI's 2026 State of Software Delivery report points in the same direction. As summarised by Rob Bowley, average teams saw a 4% rise in total pipeline runs, but main branch throughput, the code actually heading to production, fell 7%, and main branch build success dropped to 70.8%, the lowest in over five years. Bowley questions some of the report's data on top-performing teams but finds the core finding credible: teams are generating code faster than they can safely review and integrate it.
Do developers trust AI code?
The Stack Overflow 2025 Developer Survey, with nearly 50,000 respondents, found that 84% use or plan to use AI tools, up from 76%, and 47.1% use them daily. But only 33% trust the accuracy of AI output, down from 43% a year earlier, while 46% distrust it and just 3% highly trust it, according to The Decoder. The most common frustration, cited by 66%, is AI answers that are "almost right, but not quite", and 45% said debugging AI-generated code takes more time than expected.
What this means for Indian developers
India is now central to this question. GitHub's Octoverse 2025, reported by Business Standard, says India added 5.2 million developers in a year, grew 31% and now has more than 21.9 million developers, second only to the US, with a projected 57.5 million by 2030. GitHub also reports that about 80% of new developers worldwide use Copilot in their first week.
Based on the evidence reviewed, a practical approach is:
- Freshers and juniors: AI likely speeds you up the most, according to the field experiments, but you still need to understand and review what it writes. We covered what this means for hiring in AI coding agents and fresher jobs.
- Senior developers on familiar code: measure before assuming a speedup. The METR results suggest feeling faster is not the same as being faster.
- Team leads: track delivered changes and failures, not lines of code or pull requests. DORA's finding that AI amplifies existing practices means testing and review need to scale with output.
- Cost-conscious teams: measure cost per solved task, as we did in the real cost of AI coding in India, and see our research review of coding agents for ways to get more from them.
Frequently asked questions
Do AI coding tools make developers faster?
Often, but not always. Controlled studies found 21% to 56% faster completion on well-defined tasks, while METR's 2025 trial found experienced developers working on their own large codebases were 19% slower with AI.
What did the METR study on AI and developer productivity find?
METR's July 2025 trial found 16 experienced open-source developers took 19% longer on real tasks when allowed to use AI tools, though they believed AI had made them 20% faster. A February 2026 update suggested a speedup with newer tools, but METR called that data very weak evidence.
Why did developers feel faster but measure slower?
The METR study did not settle the cause. Possible explanations discussed in the sources include time spent prompting, waiting for and reviewing AI output, which can feel less effortful than writing code even when it takes longer.
Does GitHub Copilot increase productivity?
In field experiments with 4,867 developers at Microsoft, Accenture and a Fortune 100 company, Copilot users completed 26% more pull requests a week, with larger gains for less experienced developers.
Do junior or senior developers benefit more from AI?
The field experiments found less experienced developers benefited more. METR's study of highly experienced developers on familiar code found no speedup in early 2025.
How many developers trust AI-generated code?
Only 33% of developers trust the accuracy of AI tools, according to the Stack Overflow 2025 survey, down from 43% in 2024. 46% actively distrust it.
How many developers does India have?
GitHub's Octoverse 2025 reports more than 21.9 million developers in India, second only to the United States, with 5.2 million added in a year and a projection of 57.5 million by 2030.
Limits of this review
- Tools change faster than studies. Most trials tested tools that are now one or two generations old.
- Small samples. METR's studies used 16 and 57 developers, with wide confidence intervals.
- Different measures. Task time, pull requests and pipeline runs are not the same as delivered value.
- Few Indian studies. None of the trials focused on Indian developers or companies, so applying them to India is our inference.
- Secondary summaries. Several findings come from reputable summaries (InfoQ, The Decoder, Rob Bowley) rather than the full reports.
References
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, 10 July 2025
- We are Changing our Developer Productivity Experiment Design, METR, 24 February 2026
- METR's developer productivity research: 2026 update, Rob Bowley, 4 April 2026
- How much does AI impact development speed? An enterprise-based randomized controlled trial, Paradis and others, Google, arXiv, 2024
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot, Peng, Kalliamvakou, Cihon and Demirer, arXiv, 2023
- Copilot and developer productivity: three field experiments, InfoQ, September 2024
- 2024 DORA report, InfoQ, November 2024
- DORA 2025 State of AI-assisted Software Development, InfoQ, September 2025
- Developers rely on AI tools more than ever, but trust is slipping, The Decoder, 2025
- More code, less delivery: does the CircleCI 2026 report show 1 in 20 teams are benefiting?, Rob Bowley, 2 April 2026
- GitHub projects India to have over 57.5 million developers by 2030, Business Standard, 28 October 2025