Who Codes Best? Benchmark leaders, developers’ favorite and the cheapest models are not the same
Top of the test, most popular, best value: in coding AI, are those the same name? We lined up 739,000 blind votes, a 15,000-developer survey, trillions of tokens and real company case studies. On September 23, every test had a different winner. By October 6, Claude Opus 5.5 had taken first place on all three leaderboards we track, though only by error-bar margins. The most-used tool and the cheapest models are still different names.
Public-source analysis · Edited by Dandan Dad · Editorial policy
$ agent "fix the flaky checkout test"› read 38 files · plan: 4 steps› edit src/checkout/retry.ts +42 −9✓ 128 tests passed · PR opened$ On Sep 23, each event had a different leader. By Oct 6, Opus 5.5 led Code Arena and both Terminal-Bench boards (official and Vals.ai), but the gaps are too small to call a general winner.
JetBrains 2026 survey. More than double January’s 18% in six months.
The Pragmatic Engineer’s “most loved tool.” Cursor is second at 19%.
On Sep 23, OpenRouter’s top two by tokens were cheap open models. That ranks usage, not revenue or customer count.
01. Performance: each benchmark is a different event
“Good at coding” covers a lot of ground. Fixing one bug, finishing a multi-hour job in a terminal and building a good-looking web page are different events. Just as the 100-meter champion rarely wins the marathon, each event has its own leader.
Fix 500 real GitHub issues and pass hidden tests. On Vals.ai, 7 of 86 models clear 95%, and with scores bunched near the ceiling, Vals.ai has stopped testing new models on it.
Scale AI’s harder sequel, built to cut the risk that test problems leaked into training data (contamination). On 731 public tasks the leader scores just 61.5%. The board has not added the newest models yet.
66 long, real terminal jobs: building a service, training an ML model, running a forensic analysis and more. Median expert estimate 4 hours, 8-hour limit, no partial credit.
People compare two models’ web builds for the same prompt without seeing the names, then vote. By Sep 23, 132 models had drawn 739,375 votes.
SWE-bench, the most famous exam, went from nearly impossible to nearly perfect in three years. When an exam gets easy, it stops telling the top students apart. In 2026 the real contest happens on longer, harder tests and in blind votes by humans.
02. Leaders by event
We collected the leaders of six events as of Sep 22–23. Anthropic models led three; OpenAI, Meta and Moonshot models led one each. There is no agreed way to merge different evaluations into one ranking, so the medal count is a summary of separate events, not an overall ranking. Notice that even on the same Terminal-Bench 4.0, the winner changed depending on who ran it and with which harness. This medal table did not last long. For how the boards moved after Claude Sonnet 5.5 (Sep 28) and GPT-6.1 Sol (Sep 29) arrived, see section 03.
- 🥈 GPT-6 Astra 1792
- 🥉 Claude Fable 5.1 1755
- 🥈 Fable 5.1 + Claude Code 57.9%
- 🥉 Opus 5 + Claude Code 53.9%
- 🥈 GPT-6 Astra 57.07%
- 🥉 Claude Fable 5.1 49.49%
- 🥈 GPT-5.4 (xHigh) 59.1%
- 🥉 Muse Spark 55.0%
- 🥈 DeepSeek V4 Pro 96.4%
- #3 GPT-5.6 Sol 96.2%
- 🥈 Qwen3.8 Flash Next 1636
- 🥉 Hy4 preview 1631
Blind human votes · 2026-09-23 · 739,375 votes · CI: Opus 5.5 ±21, Astra ±12, so the intervals overlap
Model + agent harness · tbench.ai snapshot 2026-09-03 · ±2.8–3.8 pts · on Oct 6, Opus 5.5 + Claude Code leads at 64.8%
Independent run, same harness (Terminus 2) · avg of 3 runs · 2026-09-22 · the other 27 models all score under 30% · Oct 6 rerun in section 03
Public set, 731 tasks · Scale AI · ±3.1–3.6 pts · equal numbers are ties · still no GPT-6 or Claude 5.x entries on Oct 6
Sources: Code Arena [1] · Terminal-Bench [2] · Vals.ai [3][4] · Scale AI [5]
03. The fine print on the top score, and October’s rerun
In Vals.ai’s independent run on September 22, Claude Opus 5.5 topped Terminal-Bench 4.0 at 61.62%. But Vals.ai spelled out how that score was calculated, right under the table.
Of 198 task attempts, 30 were answered not by the requested Opus 5.5 but by provider-side fallbacks, Opus 5 or Opus 4.8. Think of a pinch hitter’s hits landing in the starter’s stats. Vals.ai’s score with fallback responses counted as failures is 53.54%, below GPT-6 Astra’s 57.07%. It is not a fresh run of every task without fallbacks. Fable 5.1 also had 28 fallback attempts out of 198; its score with fallback responses counted as failures is 42.42%. [3]
When we checked on September 23, the official Terminal-Bench board also had GPT-6 Astra (58.2%) and Claude Fable 5.1 (57.9%) closer than the error bars (±2.8–3.8 points), and Code Arena’s 1818 (±21) and 1792 (±12) overlapped. [2][1]
Two weeks later the boards had moved. On the official Terminal-Bench board on October 6, Opus 5.5 with Claude Code leads at 64.8%, ahead of Sonnet 5.5 with Claude Code at 61.8%; GPT-6 Astra and GPT-6.1 Sol, each with Codex, tie for third at 58.2%. The same day, Vals.ai published a rerun with a different harness, mini-SWE-agent. Opus 5.5 again leads at 65.15%, but 22 of its 198 attempts were fallbacks. Count those as failures and it drops to 58.08%, behind Sonnet 5.5 (62.63%, only 4 fallbacks) and GPT-6 Astra (59.60%, none). Code Arena on October 1 reads Opus 5.5 1815, GPT-6 Astra 1788, Sonnet 5.5 1786. [2][3][1]
So one name now sits on top, but the gaps are still small. The top two on the official board have overlapping error bars, and Code Arena itself gives Opus 5.5 a rank range of 1 to 2. Public scores alone cannot call a general winner. Compare models with the tools and tasks you will actually use. For what it is worth, Opus 5.5 took the top spot on Artificial Analysis’s overall Intelligence Index on September 22 (58) and still held it on October 6. [24]
04. Usage: the tools developers use at work
Now step off the scoreboard and into the office. In JetBrains’ 2026 survey of 15,000+ developers worldwide (May–July), 39% used Claude Code at work, more than double January’s 18%. For 31% it is the single most-used tool. [6]
GitHub Copilot slipped from 29% to 21%, though 79% of developers still know it. Cursor fell from 18% to 12%, most sharply in China (28% to 16%). The steepest line belongs to Codex: roughly 5x, from 3% to 16%, while awareness jumped from 27% to 65%. The open-source agent OpenCode reached 7%. [6] Stack Overflow’s 2026 Developer Survey, published October 6 with 30,000+ respondents, puts them in the same order: among coding agents used in the past year, Claude Code 66%, GitHub Copilot 59%, Codex 30% and Cursor 26% (multiple answers allowed).
Use AI agents at work
Use them daily
In that survey, tool usage among full-stack developer respondents was GitHub Copilot 65% and Claude Code 50%. These are usage figures, not single-choice preferences or mutually exclusive shares. [12]
Small teams try new tools right away; big companies use Copilot, which already cleared procurement and security review. Survey author Gergely Orosz attributes the gap to enterprise procurement more than preference. [9]
05. Developers’ favorite tools, in their words
What you use and what you love can differ. The Pragmatic Engineer asked 906 developers in January–February 2026 which AI tool they love most. Keep in mind the sample skews senior (median 11–15 years of experience) and toward the US and Europe. [9]
In JetBrains’ January survey Claude Code also posted the market’s best loyalty numbers: 91% CSAT and an NPS of 54. [9][8]
Cursor kept the #2 “most loved” spot even as its adoption fell. Regular agent users were far more excited about AI (61%) than non-users (36%). Here are real voices from the studies, the survey and a company case study. [9]
My head’s going to explode if I try to do too much the old fashioned way because it’s like trying to get across the city walking when all of a sudden I was more used to taking an Uber.
Developer in METR’s productivity study, 2026 [15]
I didn’t write any code during those seven hours. I just provided occasional guidance.
Kenta Naruse, machine learning engineer, Rakuten [20]
I use agents for pretty much all coding work… Almost all of my AI-written code is still reviewed and ‘crafted’… I am still very much in the loop.
Pragmatic Engineer survey respondent, smaller company (excerpt) [9]
I’m torn. I’d like to help provide updated data on this question but also I really like using AI!
Developer asked to rejoin METR’s study [15]
06. Cost: token volume and cost per task
Now open the bill. OpenRouter, which puts many models behind one API, paints yet another picture. Its ranking counts tokens processed, not revenue or customers, and cheaper models process more tokens for the same budget. On September 23, GLM 5.3 Flash (19T) and DeepSeek V4.1 Flash (18.4T) led, both low-cost models. Among apps, Claude Code was #2, with open-source agents Kilo Code and Cline at #3 and #4.
OpenRouter overall rankings as of Sep 23, 2026 (not coding-only). Tools that call models through their own servers, and private enterprise contracts, barely show up here. The ranking moves a lot week to week: in the week to October 5, an unnamed ‘stealth’ model, Space Bunny Alpha (33.3T), led, followed by DeepSeek V4.1 Flash (29.6T) and GLM 5.3 Flash (10.2T). [13]
Math: tasks × Vals.ai’s Sep 22 measured cost per task; expected wins = tasks × success rate. Real costs vary widely with task length, caching and plans. For subscription limits see our AI Limits & Pricing Tracker.
07. Productivity studies: does AI make you faster?
The best-known test is METR’s randomized controlled trial: METR decided at random, task by task, whether AI was allowed. Experienced developers expected AI to make them 24% faster; with early-2025 tools they actually took 19% longer. Stranger still, afterwards they believed they had been 20% faster. [14]
METR 2025 randomized controlled trial: 16 experienced open-source developers, 246 real tasks. [14]
Developers got too used to AI to measure
Starting Aug 2025, METR reran the study with 57 developers, 143 repos and 800+ tasks. Returning developers were estimated 18% faster (CI −38% to +9%), new ones 4% faster (−15% to +9%). But METR calls the data unreliable: more developers refused to work without AI, and 30–50% withheld tasks they did not want to do without it. METR thinks the true speedup is likely higher. [15]
AI adoption now lines up with higher delivery throughput, but still with lower delivery stability. [16]
🤖 agentic coders (84% of their code agent-written on average) · 🤝 AI-assisted · ✍️ manual. On average about 47% of code is fully agent-written (self-reported).
Share of developers generating 80%+ of their code with agents. China, Japan and South Korea run at twice Europe’s rate. [7]
08. Numbers companies report themselves
The numbers companies report themselves are far bolder. Most are self-reported, and methods are not disclosed, so we label them.
Announced by CEO Sundar Pichai in April 2026, up from 50% last fall. [17]
Self-reportedIts CFO said “90%-plus” in May 2026. [18]
Self-reportedMicrosoft earnings, July 2026. GitHub Copilot has 50 million users. [21]
Self-reportedSemiAnalysis estimate, Feb 2026, forecasting 20%+ by year-end. [19]
Third-partyA 12.5M-line vLLM refactor at 99.9% accuracy; feature time-to-market cut 79%, from 24 days to 5. [20]
Case studyCompanies don’t disclose how they count “AI-generated code” (e.g., whether accepted autocomplete counts). Read them as trends.
Caveat09. Incidents: handing an agent permissions
Faster work has also produced incidents. Four numbers to remember before handing a coding agent the keys.
During a code freeze in July 2025 it wiped a production database, losing 1,206 executive records and 1,196+ company profiles, then tried to hide it and wrongly claimed recovery was impossible. [22]
Tracking 100+ models, Veracode finds the average security pass rate stuck at 56% for a year; without security guidance, models produce vulnerable code about 44% of the time. [23]
Stack Overflow 2025. More developers distrust AI accuracy (46%) than trust it (33%); only 3% highly trust it. [10][11]
Stack Overflow’s April 2026 pulse. 68% prefer one predictable agent over multi-agent setups: agents on a leash. [12]
The lesson is simple: never give an agent write access to production data, and let the pipeline, with tests, security scans and code review, stop bad changes automatically. There is a reason developers keep their agents on a leash.
10. Testing on your own work: what to try first
Answer three questions and we will suggest a setup to try first, based on the data in this article. Treat it as a starting point and make the final call by comparing tools on real tasks from your own repository.
You want a terminal agent you can hand whole tasks to. You get models that rank near the top of several evaluations (Claude Opus 5.5 and Sonnet 5.5, GPT-6 Astra and GPT-6.1 Sol). Claude Code is the most-used tool at work (39%), and Codex adoption rose from 3% to 16% in six months. Put tests and code review in front of long runs as automatic gates.
You like to pair inside the editor, mixing autocomplete with chat edits. Cursor’s adoption dipped but it is still the #2 most-loved tool (19%); Copilot is the familiar choice with 79% awareness.
It is the most-used tool at companies with 10,000+ employees (56%), and Microsoft says over 90% of the Fortune 500 use GitHub. It has likely already cleared procurement and security review.
Pair an open-source agent with a cheap open model. On Sep 23, OpenRouter’s top two models by tokens were GLM 5.3 Flash and DeepSeek V4.1 Flash. But both solved only 19.7% on Vals.ai’s Terminal-Bench 4.0 (runs of Sep 22 and Oct 6), so hand the big jobs to a stronger model.
How to compare on your own work
- Pick 3–5 issues you actually handled recently. Mix a bug fix, a feature and a refactor.
- Give two candidate tools the same instructions and the same permissions.
- Record whether tests pass, time taken, cost, and how much a person had to fix.
- If results are close, pick the cheaper tool or the one your team already uses.
The short version. One new model can reshuffle a benchmark, and the top gaps sit within the error bars. At work, Claude Code is the most used; at companies with 10,000+ employees, GitHub Copilot leads. By tokens processed on OpenRouter, cheap open models lead. The better question now is not who is best, but who fits your work.
🔎 Fact-check notes: common claims vs the data
| COMMON CLAIM | WHAT WE FOUND |
|---|---|
| A 97% SWE-bench score means coding is solved | Verified has lost its separating power (7 of 86 models at 95%+), and Vals.ai has stopped testing new models on it. The harder SWE-bench Pro public set is led at 61.5%. |
| According to the 2026 Stack Overflow survey… | The 2026 annual survey results came out on October 6: among coding agents used in the past year, Claude Code 66%, GitHub Copilot 59% (multiple answers). When quoting, check which one you mean: the 2025 survey, the April 2026 pulse or the 2026 annual survey. |
| Opus 5.5 crushed Terminal-Bench | Vals.ai’s scores include fallbacks (30 on Sep 22, 22 on Oct 6). Count them as failures and Opus 5.5 falls below GPT-6 Astra in September, and below Sonnet 5.5 and Astra in October. On the official board, its lead over #2 is within the error bars. |
| Opus 5.5 and GPT-6 Astra are tied for first | There is no agreed way to merge different evaluations, and overlapping confidence intervals do not prove equal performance either. So this article lists each event’s leader instead of declaring a tie. |
| The top OpenRouter model is the best seller | OpenRouter ranks tokens processed. Revenue and customer counts are not published, and cheaper models use more tokens per dollar. |
| GitHub Copilot is finished | Adoption fell from 29% to 21%, but it leads at 10,000+ employee companies (56%) and has 50M users (per Microsoft). It is also #2 among coding agents in Stack Overflow’s 2026 survey (59%). |
| AI always makes you faster | METR’s 2025 trial measured a 19% slowdown. The 2026 follow-up saw signs of speedup but calls its numbers unreliable due to selection bias. |
| 75% or 90% of a company’s code is AI | All self-reported, with undisclosed definitions of “AI-generated.” |
Quick questions
Which AI model is best at coding?
As of October 6, 2026, Claude Opus 5.5 leads both Code Arena WebDev (1815) and the official Terminal-Bench 4.0 board (64.8% with Claude Code). The order changes often: on September 23, GPT-6 Astra led the official Terminal-Bench board. Claude Sonnet 5.5 and GPT-6 Astra are close behind, too close to call a general winner from public scores alone, so compare them on the tools and tasks you will actually use.
Which AI coding tool do developers use most?
In JetBrains’ 2026 survey of 15,000+ developers, adoption at work was Claude Code 39%, GitHub Copilot 21%, Codex 16% and Cursor 12%. At companies with 10,000+ employees, Copilot leads. Stack Overflow’s 2026 annual survey shows the same top two among coding agents used in the past year: Claude Code (66%) and GitHub Copilot (59%).
What is the cheapest setup?
An open-source agent such as OpenCode, Kilo Code or Cline paired with a low-cost model like GLM or DeepSeek. Expect lower success on long tasks (GLM 5.3 Flash solved 19.7% on Terminal-Bench 4.0). On cost per task, GPT-6.1 Sol was the cheapest model above 50% in Vals.ai’s October 6 run ($1.72, 55.05%).
Does AI actually make coding faster?
In METR’s 2025 trial, experienced developers took 19% longer. The 2026 follow-up saw signs of speedup but could not pin down a number because more participants refused to work without AI.
Keep exploring
The Vote Winner vs the Designer’s Pick
⚖️Claude Opus 5.5 vs GPT-6 SolThe cost of getting it done
🦞The Future of AI AgentsWhy Did the Lobster Grab the Keyboard?
Sources
- [1]Arena · Code Arena WebDev (checked Sep 23 and Oct 1, 2026)
- [2]Terminal-Bench · Terminal-Bench 4.0 official site (checked Oct 6, 2026)
- [3]Vals.ai · Terminal-Bench 4.0 (run Sep 22, 2026; rerun Oct 6, 2026)
- [4]Vals.ai · SWE-bench Verified (last updated Sep 1, 2026, now archived)
- [5]Scale AI · SWE-Bench Pro (public set, checked Oct 6, 2026)
- [6]JetBrains · AI Coding Agents: Adoption Trends (DevEco 2026)
- [7]JetBrains · How Much Code Do Developers Really Let Agents Write?
- [8]JetBrains · Which AI Coding Tools Do Developers Actually Use at Work? (AI Pulse, Jan 2026)
- [9]The Pragmatic Engineer · AI Tooling for Software Engineers in 2026
- [10]Stack Overflow 2025 Survey · AI
- [11]Stack Overflow · 2025 Developer Survey press release
- [12]Stack Overflow blog · Agents on a leash (April 2026 Pulse Survey)
- [13]OpenRouter · Rankings & Apps (checked Sep 23 and Oct 5, 2026)
- [14]METR · Early-2025 AI and experienced open-source developers
- [15]METR · Uplift update (Feb 24, 2026)
- [16]Google Cloud · Announcing the 2025 DORA report
- [17]The Verge · Google says 75% of its new code is AI-generated
- [18]TechSpot · Anthropic: more than 90% of code now written by AI
- [19]SemiAnalysis · Claude Code is the inflection point (Feb 5, 2026)
- [20]Rakuten Today · Rakuten accelerates development with Claude Code
- [21]Microsoft · FY26 Q4 earnings (Jul 29, 2026)
- [22]The Register · Replit deleted production database (Jul 21, 2025)
- [23]AIThority · Veracode 2026 GenAI Code Security Report
- [24]Artificial Analysis · Claude Opus 5.5 takes the top spot (Sep 22, 2026)
- [25]arXiv · SWE-bench paper (2310.06770)
- [26]OpenAI · Introducing SWE-bench Verified
- [27]SWE-bench news · mini-SWE-agent 65%
Checked Sep 24, 2026; leaderboards rechecked Oct 6.
