16 AI Coding Models Compared: Accuracy, Speed, and the Cost of Being Wrong

That recommendation comes from a September 14, 2026 Is It Just Me daily benchmark, the publisher’s separate methodology, and current primary documentation from the model providers. Grow AI Skills did not run this 16-model test. We are using it as a well-documented case study in model selection—not as a universal ranking of intelligence.

The six decisions hiding inside one leaderboard

The original X post—“Opus 5 is the priciest, DeepSeek Pro is the slowest”—is a good hook because both results are counterintuitive. But the full data supports six different buying decisions:

If you care most about…	First model to test	Why	Important caution
Highest first-pass solve rate	GPT-6 Astra	It led this slate at 81.7%.	Its 6–3 head-to-head edge over Sol was not statistically significant in the source analysis.
Strong default economics	GPT-5.6 Sol	It reached 76.7% at $0.0462 per correct result.	Its 62.45-second median API time was almost three times Astra’s.
Fast, balanced interaction	Gemini 3.8 Flash	It combined 73.3% with a 23.33-second median.	One competitive-programming slate cannot predict a repository agent.
Low-cost validated automation	DeepSeek V4.1 Flash	It reached 70.0% at $0.0217 per correct result.	Cheap failures are still failures; require deterministic checks.
Open-weight deployment	NVIDIA Nemotron 3 Ultra	It led the open-weight entries at 67.3%.	API benchmark cost does not include your hosting, utilization, or operations.
Disposable candidate generation	Mercury 2.5	It was fastest at 5.30 seconds and cheapest at $0.0016 per correct result.	It solved only 30.0%; do not confuse generation speed with completion speed.

Notice that this is not a podium. Different constraints produce different shortlists. A team with reliable unit tests can exploit a cheaper model more aggressively than a team whose “validation” is a senior engineer reading every change. A support script that drafts a reversible reply can tolerate more misses than a database migration agent. The model does not carry the risk by itself; the surrounding system does.

What the test actually measured

The publisher generates new competitive-programming problems each morning and keeps the generator private. Models receive the same task structure through raw APIs, without browsing, external tools, an agent scaffold, or provider-side code execution. The score is pass@1: the first submitted solution either passes held-out tests or it does not. There is no partial credit.

Runs have a 20-minute ceiling. A model timeout counts as a miss and its consumed tokens still count toward cost. Provider errors are retried or excluded under the published rules. Reported speed is median API-call time, excluding the publisher’s queue, retries, and grading. “Cost per correct” divides all model spend—including failed attempts—by the number of solved problems.

Those choices make the test more informative than a screenshot of a vendor benchmark. Fresh private tasks reduce the chance that models have memorized the exact questions. Held-out execution catches solutions that sound plausible but fail. Failed-run spend prevents a verbose, unreliable model from looking artificially cheap.

They also define a strict evidence boundary. This is a test of one-shot competitive-programming transfer through APIs. It does not measure whether a model can navigate your monorepo, preserve a product requirement over 40 tool calls, inspect a screenshot, recover from a broken build, ask a useful clarifying question, or avoid a dangerous action. It also cannot establish stable model order from one day. The source itself describes daily results as noisy and uses confidence tests and a canary set to look for broader consistency.

The full 16-model result

The table below preserves the source ordering. A dash does not mean zero or unavailable; it means the absolute number was not stated in the visible daily commentary we reviewed. Relative labels such as “95% faster than the slowest” were not converted into invented absolute values.

Across the run, the source reports roughly 21 million tokens and $109.81 in spend. Its model ordering also repeated on 49 of 55 comparable calls, with a Kendall rank correlation of 0.7818. That is useful evidence that the slate was not pure randomness. It is still not evidence that the same order will hold on your work.

Four prices people accidentally treat as one

Most model comparisons put “price” in a single column. In practice, four different numbers matter.

1. List price per token is the provider’s unit rate. It may vary by input, cached input, output, context length, region, batch mode, or promotion.
2. Total benchmark spend is what one model consumed across all attempts in the test. It grows with output length, failures, and retries.
3. Cost per correct result divides total spend by successful tasks. It penalizes models that are cheap per token but rarely finish correctly.
4. Total economic cost adds validation, human review, rework, delay, incidents, and the cost of maintaining the route.

The source’s leaderboard image includes a relative “cost versus priciest” view based on the complete run. Its narrative and cost-capability chart discuss dollars per correct result. Those are both valid, but they answer different questions. A model can produce a lower total bill because it emits less text, yet still be expensive per correct result because it solves fewer tasks. Never carry a percentage from one denominator into the other.

Claude Opus 5 is the clearest example. Anthropic’s current list price for Opus is lower than Fable 5.1’s: $5 input and $25 output per million tokens for Opus, versus $10 and $50 for Fable. Yet Opus was the most expensive model per correct result in this test. The source reports 773,530 Opus output tokens, compared with 352,889 for Fable, plus three Opus clock-outs. Lower unit rates did not overcome much higher output volume and fewer successes.

The same logic protects you from a misleading “cheap model” claim. OpenAI currently lists GPT-5.6 Luna at $0.20 input and $1.20 output per million tokens, one-tenth Terra’s input price. Luna nevertheless solved 48.0% here, versus Terra’s 75.0%. If every missed job requires a person, the small API bill may be irrelevant.

The cost of being wrong: a transparent scenario

Suppose you have 100 independent tasks similar to this benchmark. To keep the arithmetic readable, multiply each published solve rate by the reported cost per correct result. Then assume every unsuccessful task creates $5 of review or repair work. The $5 is deliberately hypothetical; replace it with your own number.

Model	Expected correct out of 100	Approx. model spend	Misses × $5	Illustrative total
GPT-6 Astra	81.7	$11.36	$91.50	$102.86
GPT-5.6 Sol	76.7	$3.54	$116.50	$120.04
DeepSeek V4.1 Flash	70.0	$1.52	$150.00	$151.52
Claude Opus 5	66.7	$34.46	$166.50	$200.96
Mercury 2.5	30.0	$0.05	$350.00	$350.05

This does not predict your bill. It demonstrates a break-even principle: as the cost of failure rises, accuracy can dominate token price. In this scenario, Astra’s model bill is about $7.82 higher than Sol’s, but it avoids roughly five additional misses. If one miss costs more than about $1.56, Astra becomes cheaper overall on these assumptions. Against DeepSeek V4.1 Flash, the approximate break-even failure cost is $0.84 per miss.

The reverse is also true. If a deterministic compiler, schema validator, or retrieval check can reject a bad candidate for a few cents, a lower-cost model may win even with a lower first-pass rate. The economic decision belongs to the complete loop, not the model response.

What each model family is really offering

OpenAI: four price points, not one automatic winner

OpenAI’s current comparison positions GPT-6 Astra for the hardest end-to-end work, GPT-5.6 Sol for complex professional work, Terra as the balanced cost option, and Luna for cost-sensitive volume. Their standard input/output prices per million tokens are currently $10/$50, $4/$20, $2/$12, and $0.20/$1.20 respectively. All four list a 1.05M context window and up to 128K output.

Those are base rates, not one price across the full context window. OpenAI currently bills prompts above 272K input tokens at twice the input rate and 1.5 times the output rate for the full request. Sol’s listed $4/$20 price is also promotional through at least November 21, 2026. The provider prices and limits in this article were rechecked on September 14, 2026.

The daily board broadly supports a tiered lineup, but it does not prove four clean capability bands. Astra led, while Sol and Terra remained close enough to demand workload tests. In fact, the source’s direct Astra-versus-Sol tally was 6–3 with p = 0.5078, which is not statistically significant. “Astra ranked first” is supported. “Astra will beat Sol on your next task” is not.

If you are deciding among these models, our matched Astra, Sol, and Terra comparison shows the more useful pattern: keep task, prompt, and acceptance test fixed, then inspect where the outputs actually differ.

Gemini, Grok, and Qwen: latency belongs beside quality

Gemini 3.8 Flash is the quiet standout in this particular dataset: 73.3%, 23.33 seconds, and $0.0582 per correct result. Google currently lists an introductory price of $0.75 input and $3.75 output per million tokens, scheduled to change after December 31, 2026. That date matters; a promotion is not a permanent cost model.

Grok 4.6 reached a similar 71.9% but needed a 209.36-second median. Qwen 3.8 Max reached 71.7% at 138.96 seconds and recorded seven timeouts. xAI’s official page lists Grok 4.6 at $2/$6 for shorter-context input and output, while Alibaba’s Qwen pricing varies by region. Neither list price predicts the operational result by itself. For an interactive coding assistant, two minutes can be a product failure even when the code is eventually correct. For an overnight batch, it may be acceptable.

Claude: verbosity can reverse the price story

Claude Fable 5.1 and Opus 5 are positioned for demanding, long-horizon work. In this slate, Fable solved 71.7% and Opus 66.7%. More importantly, Opus generated more than twice Fable’s output tokens. The result is the article’s headline surprise: a model with a lower posted token rate became substantially more expensive per successful solution.

That is not a universal judgment on Opus. Competitive-programming prompts may reward concise implementation differently from exploratory engineering, review, or long-running agents. It is a warning to log output length and clock-outs instead of multiplying an average prompt by a price-table cell.

DeepSeek: low cost is useful only with current model identity

DeepSeek V4.1 Flash landed near the middle on solve rate and near the front on cost per correct result. V4 Pro was the slowest model in the comparison. The old V4 Flash 0731 looked even cheaper, but old aliases and retired routes are not durable procurement options. The current DeepSeek pricing page is the operational source of truth.

Our V4.1 Flash versus V4 Pro guide documents the alias changes, current prices, and the difference between the launch article and later service notice. The key practice is to record the requested model ID, resolved version, region, time band, reasoning effort, and response metadata with every evaluation.

Nemotron, Kimi, and GLM: open weights change the equation

NVIDIA Nemotron 3 Ultra led the open-weight entries at 67.3%. Its official model card describes a 550B-parameter model with 55B active parameters, a one-million-token context, and a hybrid architecture. It also describes substantial deployment hardware. Kimi K3 lists 2.8T total and 104B active parameters with open weights and native image and text input. GLM 5.3 Flash lists 320B total and 18B active, native multimodality, and open weights.

Open weights give you control over deployment, fine-tuning, data boundaries, and inference optimization. They do not make inference free. Add accelerators, replicas, utilization, batching, quantization loss, observability, upgrades, incident response, and the people who operate the system. A routed API benchmark measures the provider route used that day, not the economics of your own cluster.

Mercury: a genuinely different speed story

Inception Labs describes Mercury 2.5 as a diffusion language model and reports throughput above 1,100 tokens per second under its launch setup. Its standard list price is $0.20 input and $0.75 output per million tokens, with lower promotional rates at launch. In the third-party test it returned in 5.30 seconds and cost $0.0016 per correct result—then solved only 30.0%.

That combination is not useless. It suggests an architecture that can generate many disposable candidates quickly. Give it tasks with cheap validators: produce test cases, propose regexes, draft SQL that runs only against a disposable database, or generate alternative implementations that a stronger model and test suite can filter. It is a poor fit for a one-shot, unattended task where a plausible wrong answer escapes into production.

A model-routing policy you can actually operate

A mature system rarely needs one permanent winner. It needs an escalation policy.

Task shape	Suggested starting route	Acceptance gate	Escalate when…
Small code transformation with exhaustive tests	DeepSeek V4.1 Flash or Gemini 3.8 Flash	Formatter, compiler, unit tests, diff limits	Tests fail twice or forbidden files change
Hard repository issue with ambiguous requirements	GPT-5.6 Sol	Targeted tests plus human review	The model cannot localize the fault or risk is high
High-cost migration, security, or production change	GPT-6 Astra	Independent review, rollback, staged canary	Any acceptance rule is uncertain
Interactive pair-programming	Gemini 3.8 Flash or Astra	Developer judgment and live tests	Latency breaks flow or task scope expands
Private self-hosted workflow	Nemotron 3 Ultra shortlist	Same task suite plus infrastructure accounting	Utilization or quality makes hosted APIs cheaper
Mass candidate generation	Mercury 2.5	Cheap deterministic filter before use	No reliable validator exists

The exact names will age. The policy survives: use the cheapest route whose expected failures your validation layer can catch, then escalate the cases that exceed its confidence or risk boundary.

Run a canary before switching production traffic

A credible internal evaluation can start with 30–100 recent tasks. It does not need to imitate an academic benchmark, but it must preserve the conditions that make your work hard. If you need a fuller template, use our guide to building an AI regression set.

1. Sample real work. Include ordinary cases, expensive edge cases, long context, tool failures, ambiguous requests, and inputs the model should refuse. Remove sensitive data without sanding away the difficulty.
2. Write acceptance tests first. Define the required state, forbidden changes, evidence requirements, security constraints, and human review rule before seeing any model output.
3. Pin the configuration. Save provider, model ID and resolved version, date, region, prompt, reasoning effort, tools, context, token cap, timeout, and retry policy.
4. Use repeated, interleaved runs. Alternate models and repeat stochastic tasks. One lucky answer is a story, not a distribution.
5. Score the finished state. A correct-looking patch that fails tests is a failure. Separate correctness, safety, completeness, style, and reviewer confidence.
6. Log the whole loop. Capture API latency, time to accepted result, tokens, cache usage, tool turns, retries, timeouts, human minutes, and severity of each failure.
7. Calculate cost per accepted task. Include all failed attempts, validation compute, escalation calls, and review time. Also report the tail—not only the average.
8. Canary and keep rollback. Route a small, reversible share of traffic. Compare failure categories with the baseline and stop automatically when a safety or quality threshold breaks.

If you are building the surrounding workflow for the first time, the AI agent implementation guide explains why tools, state, verification, and stop conditions matter more than a clever system prompt.

When this comparison should be rerun

Run the evaluation before a model or provider change, after a major prompt or tool change, and on a regular cadence for critical workflows. Keep a small canary set running more frequently. A silent alias change, a new pricing tier, a worsening latency tail, or a rise in human review time is enough reason to retest; none of those changes has to move the public leaderboard before it affects your economics.

Choose the validator before the model

The original post is worth a deeper article because its apparent contradictions are the lesson. Opus shows why list price is not realized cost. DeepSeek Pro shows why model quality cannot excuse unusable latency. Mercury shows why raw generation speed is not task completion. Astra and Sol show why a leaderboard lead can be real without being decisive.

Before comparing token prices, write down the acceptance gate and the cost of a miss. That one step determines whether Mercury’s speed is an advantage, whether DeepSeek’s low API bill survives contact with review, or whether Astra’s higher model spend is insurance rather than waste.

Use the public ranking to build a shortlist, then replace it with your own evidence as quickly as possible. The best coding model is not the one that looks impressive in isolation. It is the route that produces an accepted result, within your latency budget, at the lowest total cost your users actually experience.