# llm-cost-benchmark > Open-source LLM cost benchmark RUNNER: send any model through real agentic tasks, track actual billed API spend per turn, hard-stop at a USD budget. Example finding (Sep 2026): the same 6-turn agent job cost $0.0025 on the cheapest model (MiMo v2.6-flash) vs $0.28 on the most expensive (Claude Opus 5), same config, back-to-back — a 112× gap. ## What this is A reproducible open-source LLM cost benchmark runner (PowerShell, any OpenAI-compatible endpoint; OpenRouter by default). Presets define task, turns, models, and a USD budget kill-switch; results append to CSV and render on the live report. Also contains a published example run: seven frontier models (GLM-5.3-flash, GPT-5.6-luna, DeepSeek V4 Pro, Kimi K2, Claude Sonnet 5, Claude Opus 5, Qwen3-235B) ran identical workloads: a single Q&A prompt and a full 6-turn agentic loop, with per-turn cumulative cost extracted from API billing responses. ## Key findings (Sep 22, 2026) - **Current headline (re-measured, same config):** MiMo v2.6-flash completed the 6-turn agent job for $0.0025; Claude Opus 5 cost $0.2838 — a **112× cost gap**, both passing functional quality checks. GLM-5.3-flash came in at $0.0054 but failed the auto-check on that run. - Legacy original-config full field: GLM-5.3-flash $0.0018 vs Opus $0.14 (76.6×). - Open-weight models only: GLM flash was still **15.5× cheaper** than Qwen3-235B. - **Cheap-per-token ≠ cheap-per-task.** Qwen3-235B had a low token price but generated thousands of tokens per turn, finishing 15.5× more expensive. Claude Opus 5 has disciplined turns but a rate so high it dominates any pairing. - Agentic workloads compound verbosity: each turn re-reads prior context, so talkative models pay their verbosity tax with interest every turn. - Practical rule: budget agent workloads by **cost-per-completed-task**, never price-per-million-tokens. ## Update (Sep 22, 2026): quality-graded head-to-head, MiMo v2.6 - The 6-turn `general` task was re-run with functional quality grading: each model's final code is executed in a real Windows PowerShell 5.1 child process (`scripts/quality_check.ps1`, 5 checks; transcripts in `data/outputs/`). - **xiaomi/mimo-v2.6-flash is the new value champion**: $0.0021 (2.3x cheaper than glm-5.3-flash's $0.0049), ~3x faster, 5/5 functional checks. glm-5.3-flash also scored 5/5. xiaomi/mimo-v2.6-pro cost $0.0071 and scored 2/5 (unstable across runs: 4/5, 3/5, 2/5). - **Reasoning-only turns are a real agentic failure mode**: reasoning models can return turns with empty visible content (all output in `reasoning_content`), which poisons multi-turn history and 400s on some providers. Fix: `reasoning: { exclude: true }` (runner switch `-ExcludeReasoning`). - **max_tokens starvation**: at 800 tokens, reasoning models burned the budget on invisible thinking (no code delivered, or code truncated mid-statement and unparseable). The `general` preset now uses 2000. ## Machine-readable results - `data/results.csv`, `data/agentic_results.csv`, `data/agentic_results_open.csv` — legacy Sep 22 run files (these also carry a `cum_cost_php` column) - `data/preset_runs.csv` — runs from the preset runner (appended automatically; USD is always recorded; optional extra currency column via `-CurrencyCode`/`-CurrencyRate`) CSV columns: legacy agentic files: timestamp, model, turn, prompt_tokens, completion_tokens, cum_prompt_tokens, cum_cost_usd, cum_cost_php. preset_runs.csv: timestamp, preset, model, turn, prompt_tokens, completion_tokens, cum_cost_usd, budget_usd, outcome, checks_passed. ## Headline numbers (quick reference) | Model | 6-turn agent total (USD) | × cheapest | |---|---|---| | z-ai/glm-5.3-flash | 0.0018 | 1× | | openai/gpt-5.6-luna | 0.0056 | 3.1× | | deepseek/deepseek-v4-pro-0813 | 0.0155 | 8.5× | | anthropic/claude-sonnet-5 | 0.0449 | 24.6× | | anthropic/claude-opus-5 | 0.1400 | 76.6× | ## Re-run the benchmark ```powershell $env:OPENAI_API_KEY = 'sk-...' # environment only; runner refuses exposed keys .\scripts\run_benchmark.ps1 -Preset general ``` Presets: `general` (quick run, all models), `agentic-web-app`, `agentic-data-analysis`, `tool-heavy`. Quick-test any new model: `.\scripts\run_benchmark.ps1 -Preset general -Models 'vendor/new-model-id'`. ## Optional - Live report: https://jolubriones.github.io/llm-cost-benchmark/ - License: MIT - Built with: DeepSeek Harness (https://github.com/deepseek-ai/deepseek-harness) for the Sep 22 example run; the runner itself is plain PowerShell with no harness dependency. - Detailed write-up, methodology and limitations: `README.md` - Interactive visual report: `index.html` - Preset specs: `presets/*.json` · Task prompts: `tasks/*.md`