What do frontier LLMs actually cost to do the same job? A hands-on benchmark with real API spend — not list prices, actual invoices.
Key findings: On the same 6-turn agentic job, measured back-to-back under the same config, MiMo v2.6-flash completed the task for $0.0025 while Claude Opus 5 cost $0.28 — a 112× cost gap, with both passing functional quality checks. Even restricted to open-weight models (legacy run), GLM-5.3-flash was 15.5× cheaper than Qwen3-235B. The core insight: cheap-per-token does not mean cheap-per-task — verbosity and context re-reads, not token price, drove real costs. All numbers are real billed API spend (not published price sheets), with raw data and a reproducible preset-based benchmark runner included in this repository.
scripts/quality_check.ps1).
| Model | Cost | Time | Quality |
|---|---|---|---|
| xiaomi/mimo-v2.6-flash | $0.0021 | 69s | 5/5 |
| z-ai/glm-5.3-flash | $0.0049 | 197s | 5/5 |
| xiaomi/mimo-v2.6-pro | $0.0071 | 179s | 2/5 |
reasoning_content, poisoning multi-turn history; fix: reasoning: {exclude: true}, runner switch -ExcludeReasoning), and a small max_tokens budget silently burns on invisible thinking, truncating real output (800 → 2000 fixed it). The "pro" tier cost 3.4× the flash tier and was less reliable (4/5, 3/5, 2/5 across runs).
Same 6-turn agent task, same runner config (reasoning excluded, max_tokens 2000), measured back-to-back in one session on Sep 22, 2026. MiMo v2.6-flash completed the job for $0.0025; Opus 5 for $0.284 — a 112× gap, with both passing the functional quality check. GLM-5.3-flash came in at $0.0054 but failed the auto-check on this run. Notably, Opus got more expensive under the updated config ($0.14 → $0.28): at max_tokens 2000 the reasoning-excluded run simply spends more billed tokens per turn.
The original Sep 22 snapshot: all six models, original runner config. Under that config GLM-5.3-flash led at $0.0018, making Opus 5 a 77× outlier. Kept unchanged for the legacy comparison; the current headline is chart 0 above.
Even restricting to models whose weights you can self-host, GLM flash wins by a wide margin.
Qwen3-235B explodes past every other model from turn 1 — it writes thousands of tokens per turn. Opus 5 is a steady, expensive climb. GLM flash hugs the floor the whole way. Log scale: with costs spanning 0.0002 to 0.14, a linear axis flattens seven of eight lines into noise. † MiMo v2.6 lines use the updated config (reasoning excluded, max_tokens 2000); MiMo-flash tracks GLM's legacy curve until its context re-reads compound late.
Even for a plain question, Opus 5 charges ~30–150× more than GLM flash. † Rows marked with the dagger were re-measured with the updated config (reasoning excluded): MiMo-flash $0.00011, MiMo-pro $0.00032, and GLM-5.3-flash now just $0.00005 per answer — reasoning tokens were the hidden cost driver.
Loading preset run data…
Presets are declarative specs (presets/*.json: task, turns, models, USD budget kill-switch). Run one with scripts/run_benchmark.ps1 -Preset general — results land in data/preset_runs.csv and appear here automatically. Budget exhaustion is a result, not an error.