The 112× Finding 🧪

What do frontier LLMs actually cost to do the same job? A hands-on benchmark with real API spend — not list prices, actual invoices.

Key findings: On the same 6-turn agentic job, measured back-to-back under the same config, MiMo v2.6-flash completed the task for $0.0025 while Claude Opus 5 cost $0.28 — a 112× cost gap, with both passing functional quality checks. Even restricted to open-weight models (legacy run), GLM-5.3-flash was 15.5× cheaper than Qwen3-235B. The core insight: cheap-per-token does not mean cheap-per-task — verbosity and context re-reads, not token price, drove real costs. All numbers are real billed API spend (not published price sheets), with raw data and a reproducible preset-based benchmark runner included in this repository.

112×
cost gap, cheapest (MiMo v2.6-flash $0.0025) vs priciest (Opus 5 $0.28) on the same 6-turn agent job, same config
15.5×
open-weights-only legacy run: GLM flash vs Qwen3-235B — the gap persists without Big-AI tax
$0.0025
all-in cost of a full check-passing agentic run on MiMo v2.6-flash (Opus 5: $0.28)
UPDATE (Sep 22) — quality-graded head-to-head, new value champion: the same 6-turn task was re-run and each model's final code was executed in a real Windows PowerShell 5.1 child process (5 functional checks; transcripts + grader in the repo, scripts/quality_check.ps1).
ModelCostTimeQuality
xiaomi/mimo-v2.6-flash$0.002169s5/5
z-ai/glm-5.3-flash$0.0049197s5/5
xiaomi/mimo-v2.6-pro$0.0071179s2/5
MiMo v2.6-flash dethrones GLM flash: 2.3× cheaper, ~3× faster, equally correct. Two failure modes surfaced that cost-only benchmarks miss: reasoning models can return turns with empty visible content (everything stays in reasoning_content, poisoning multi-turn history; fix: reasoning: {exclude: true}, runner switch -ExcludeReasoning), and a small max_tokens budget silently burns on invisible thinking, truncating real output (800 → 2000 fixed it). The "pro" tier cost 3.4× the flash tier and was less reliable (4/5, 3/5, 2/5 across runs).
Methodology — how the numbers were produced:

0 · The headline (re-measured, same config): 112×

Same 6-turn agent task, same runner config (reasoning excluded, max_tokens 2000), measured back-to-back in one session on Sep 22, 2026. MiMo v2.6-flash completed the job for $0.0025; Opus 5 for $0.284 — a 112× gap, with both passing the functional quality check. GLM-5.3-flash came in at $0.0054 but failed the auto-check on this run. Notably, Opus got more expensive under the updated config ($0.14 → $0.28): at max_tokens 2000 the reasoning-excluded run simply spends more billed tokens per turn.

1 · Six-turn agent task — full field, Sep 22 original config (legacy snapshot)

The original Sep 22 snapshot: all six models, original runner config. Under that config GLM-5.3-flash led at $0.0018, making Opus 5 a 77× outlier. Kept unchanged for the legacy comparison; the current headline is chart 0 above.

2 · Six-turn agent task — open-weight models only

Even restricting to models whose weights you can self-host, GLM flash wins by a wide margin.

3 · Cost growth per turn (full field) — who ramps up fastest?

Qwen3-235B explodes past every other model from turn 1 — it writes thousands of tokens per turn. Opus 5 is a steady, expensive climb. GLM flash hugs the floor the whole way. Log scale: with costs spanning 0.0002 to 0.14, a linear axis flattens seven of eight lines into noise. † MiMo v2.6 lines use the updated config (reasoning excluded, max_tokens 2000); MiMo-flash tracks GLM's legacy curve until its context re-reads compound late.

4 · Simple Q&A — cost per answer

Even for a plain question, Opus 5 charges ~30–150× more than GLM flash. † Rows marked with the dagger were re-measured with the updated config (reasoning excluded): MiMo-flash $0.00011, MiMo-pro $0.00032, and GLM-5.3-flash now just $0.00005 per answer — reasoning tokens were the hidden cost driver.

5 · Live preset runs — run it yourself

Loading preset run data…

Presets are declarative specs (presets/*.json: task, turns, models, USD budget kill-switch). Run one with scripts/run_benchmark.ps1 -Preset general — results land in data/preset_runs.csv and appear here automatically. Budget exhaustion is a result, not an error.