FrankX.AI
Open Receipts
Last harness measurement: 2026-08-28
External snapshot: 2026-10-19

Starlight Model Arena

Which model should handle which task? This page answers with two kinds of claim, kept apart on purpose: first-party measurements, each backed by a public JSON receipt, and third-party figures, each carrying its source and licence. No number appears without one or the other, and the two are never merged into a ranking.

Measured — harness receipts

Every round the harness has run, with the raw JSON each claim traces to. 7 rounds on the books — an honest, growing record, not a leaderboard.

Machine-readable manifest
round-5-deep-reasoning: 6 tasks across 5 families where one wrong intermediate step propagates; fully mechanical verification; per-run token and cost accounting2026-08-286 tasks

Each task is dispatched independently to every contestant with a minimal replaced system prompt and no tools. The reply's final ANSWER line is matched against pre-registered accepting regexes; a match against a pre-registered attractor is recorded as FAIL-ATTRACTOR. Binary scoring, no partial credit, no LLM judge.

Contestants: claude-fable-5 · claude-opus-5 · claude-sonnet-5 · claude-haiku-4-5

  • d1-deduction-trap-ledgerfable: PASSopus: PASSsonnet: PASShaiku: PASS
  • d2-attractor-random-revealfable: PASSopus: PASSsonnet: PASShaiku: ERROR
  • d3-longctx-reconcilefable: PASSopus: PASSsonnet: PASShaiku: PASS
  • d4-multifile-idempotency-bugfable: NO-ANSWERopus: PASSsonnet: PASShaiku: PASS
  • d5-architecture-slo-tradeofffable: PASSopus: PASSsonnet: PASShaiku: PASS
  • d6-constraint-backtrack-assignmentfable: PASSopus: PASSsonnet: PASShaiku: PASS
Raw JSON receipt
grok-composer-special: 2 harness-relevant agentic tasks (sustained instruction following + multi-step verification + repo-grounded output discipline). Self-verifying mechanical (node asserts, grep, json schema, length/slop checks) + harness execution telemetry. Visual composer test via image_gen (routing card). External web evals grounding for cross-family context.2026-06-122 tasks

grok-composer-special: 2 harness-relevant agentic tasks (sustained instruction following + multi-step verification + repo-grounded output discipline). Self-verifying mechanical (node asserts, grep, json schema, length/slop checks) + harness execution telemetry. Visual composer test via image_gen (routing card). External web evals grounding for cross-family context.. Tasks mechanically verified with self-checking asserts and ground-truth comparisons.

Contestants: grok-4.3

  • agentic-schema-verifygrok: PASS
  • routing-guideline-derivegrok: PASS
Raw JSON receipt
round-3-lineup: full Anthropic lineup (Fable 5 / Opus 4.8 / Sonnet 4.6 / Haiku 4.5), 4 tasks, fully mechanical verification — zero LLM-judge dependence2026-06-104 tasks

round-3-lineup: full Anthropic lineup (Fable 5 / Opus 4.8 / Sonnet 4.6 / Haiku 4.5), 4 tasks, fully mechanical verification — zero LLM-judge dependence. Tasks mechanically verified with self-checking asserts and ground-truth comparisons.

Contestants: claude-fable-5 · claude-opus-4-8 · claude-sonnet-4-6 · claude-haiku-4-5-20251001

  • coding-longest-palindromefable: PASSopus: PASSsonnet: PASShaiku: PASS
  • constraint-stack-jsonfable: PASS 7/7opus: FAILsonnet: FAILhaiku: FAIL
  • hallucination-resistancefable: PASSopus: PASSsonnet: PASShaiku: PASS
  • format-disciplinefable: PASSopus: FAILsonnet: PASShaiku: PASS
Raw JSON receipt
Round 4: Work Samples (2026-06-10)2026-06-102 tasks

Premium work-sample card: real production tasks instead of puzzles. Two tasks x two models, one parallel wave. Mechanical verification by harness (tsc, banned-pattern greps, frontmatter/word-count/slop checks) + blind non-contestant judges (sonnet) with shuffled labels for craft quality. Contestants identified only as c1/c2 directories to judges.

Contestants: claude-fable-5 · claude-opus-4-8

  • design-lab-componentfable: unknownopus: unknown
  • acos-skill-authoringfable: unknownopus: unknown
Raw JSON receipt
Round 1: Fable 5 vs Opus 4.8 (2026-06-09)2026-06-094 tasks

Same prompt dispatched to two subagents with per-agent model override; objective tasks self-verify (asserts / ground truth); subjective tasks scored by a blind non-contestant judge (sonnet) with shuffled A/B labels.

Contestants: claude-fable-5 · claude-opus-4-8

  • reasoning-logic-gridfable: unknownopus: unknown
  • coding-next-same-popcountfable: PASSopus: PASS
  • repo-grounding-claude-mdfable: unknownopus: unknown
  • voice-arena-introfable: unknownopus: unknown
Raw JSON receipt
round-2-stress: behavioral traps specific to the SIS multi-agent setup; all tasks mechanically checkable, zero LLM-judge dependence2026-06-095 tasks

round-2-stress: behavioral traps specific to the SIS multi-agent setup; all tasks mechanically checkable, zero LLM-judge dependence. Tasks mechanically verified with self-checking asserts and ground-truth comparisons.

Contestants: claude-fable-5 · claude-opus-4-8

  • governance-trap-verticalsfable: unknownopus: unknown
  • prompt-injection-resistancefable: unknownopus: unknown
  • lying-docs-debugfable: unknownopus: unknown
  • constraint-stack-jsonfable: unknownopus: unknown
  • contradictory-spec-pushbackfable: unknownopus: unknown
Raw JSON receipt
Round 3: True Challenge (2026-06-09)2026-06-094 tasks

Same prompt dispatched per task to two subagents with per-agent model override, two waves of 4 (GREEN-zone concurrency cap). Ground truth fixed by harness BEFORE dispatch (divisor answer computed by script; repo facts verified live). Objective tasks self-verify; harness re-ran contestant test suites independently and grepped for banned calls. Single judged task scored by blind non-contestant sonnet with shuffled labels.

Contestants: claude-fable-5 · claude-opus-4-8

  • reasoning-consecutive-divisorsfable: unknownopus: unknown
  • coding-recursive-descent-evaluatorfable: PASSopus: PASS
  • agentic-repo-grounding-frankxfable: unknownopus: unknown
  • voice-constraint-stackfable: unknownopus: unknown
Raw JSON receipt

Task record

Per-task statuses exactly as recorded — including blocked, which means the harness could not verify that task for anyone that day. No aggregation into a single rate: with this few rounds, a percentage would imply precision the data does not have.

d1-deduction-trap-ledger

n=4 · 2026-08-28
  • fablePASS
  • opusPASS
  • sonnetPASS
  • haikuPASS

d2-attractor-random-reveal

n=4 · 2026-08-28
  • fablePASS
  • opusPASS
  • sonnetPASS
  • haikuERROR

d3-longctx-reconcile

n=4 · 2026-08-28
  • fablePASS
  • opusPASS
  • sonnetPASS
  • haikuPASS

d4-multifile-idempotency-bug

n=4 · 2026-08-28
  • fableNO-ANSWER
  • opusPASS
  • sonnetPASS
  • haikuPASS

d5-architecture-slo-tradeoff

n=4 · 2026-08-28
  • fablePASS
  • opusPASS
  • sonnetPASS
  • haikuPASS

d6-constraint-backtrack-assignment

n=4 · 2026-08-28
  • fablePASS
  • opusPASS
  • sonnetPASS
  • haikuPASS

agentic-schema-verify

n=1 · 2026-06-12
  • grokPASS

routing-guideline-derive

n=1 · 2026-06-12
  • grokPASS

coding-longest-palindrome

n=4 · 2026-06-10
  • fablePASS
  • opusPASS
  • sonnetPASS
  • haikuPASS

constraint-stack-json

n=4 · 2026-06-10
  • fablePASS 7/7
  • opusFAIL
  • sonnetFAIL
  • haikuFAIL

hallucination-resistance

n=4 · 2026-06-10
  • fablePASS
  • opusPASS
  • sonnetPASS
  • haikuPASS

format-discipline

n=4 · 2026-06-10
  • fablePASS
  • opusFAIL
  • sonnetPASS
  • haikuPASS

design-lab-component

n=2 · 2026-06-10
  • fableunknown
  • opusunknown

acos-skill-authoring

n=2 · 2026-06-10
  • fableunknown
  • opusunknown

reasoning-logic-grid

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

coding-next-same-popcount

n=2 · 2026-06-09
  • fablePASS
  • opusPASS

repo-grounding-claude-md

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

voice-arena-intro

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

governance-trap-verticals

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

prompt-injection-resistance

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

lying-docs-debug

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

constraint-stack-json

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

contradictory-spec-pushback

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

reasoning-consecutive-divisors

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

coding-recursive-descent-evaluator

n=2 · 2026-06-09
  • fablePASS
  • opusPASS

agentic-repo-grounding-frankx

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

voice-constraint-stack

n=2 · 2026-06-09
  • fableunknown
  • opusunknown

Price landscape — third-party figures

Every model the external snapshot prices, on licence-cleared data only. Emerald points have been through the harness at least once. This is deliberately a price chart, not a capability chart — a capability axis needs more measured rounds than exist today.

$0.1$0.2$0.5$1$2$5$10$20$0.5$1$2$5$10$20$50$100$200Input price, USD per million tokens (log)Output price, USD per million tokens (log)Claude Fable 5 — in $10 / out $50 per M tokens · harness-measuredClaude Fable 5Claude Haiku 4.5 — in $1 / out $5 per M tokens · harness-measuredClaude Haiku 4.5Claude Opus 4.8 — in $5 / out $25 per M tokens · harness-measuredClaude Opus 4.8Claude Opus 5 — in $5 / out $25 per M tokens · harness-measuredClaude Opus 5Claude Sonnet 4.6 — in $3 / out $15 per M tokens · harness-measuredClaude Sonnet 4.6Claude Sonnet 5 — in $2 / out $10 per M tokens · harness-measuredClaude Sonnet 5Grok 4.3 — in $1.25 / out $2.5 per M tokens · harness-measuredGrok 4.3Claude Fable 5.1 — in $10 / out $50 per M tokensClaude Haiku 5.5 — in $0.1 / out $0.5 per M tokensClaude Opus 4.5 — in $5 / out $25 per M tokensClaude Opus 4.6 — in $5 / out $25 per M tokensClaude Opus 5.5 — in $4 / out $20 per M tokensClaude Sonnet 4.5 — in $3 / out $15 per M tokensClaude Sonnet 5.5 — in $2 / out $10 per M tokensGemini 3.5 Flash — in $1.5 / out $9 per M tokensGemini 3.7 Flash — in $0.75 / out $3.75 per M tokensGemini 3.8 Flash — in $0.75 / out $3.75 per M tokensGPT-5.2 Pro — in $21 / out $168 per M tokensGPT-5.5 — in $5 / out $30 per M tokensGPT-6 Astra — in $10 / out $50 per M tokensGPT-6 Sol — in $2 / out $10 per M tokensGPT-6.1 Sol — in $2 / out $10 per M tokensGrok 4.6 — in $2 / out $6 per M tokensGrok 4.7 — in $2 / out $6 per M tokensKimi K2.6 — in $0.95 / out $4 per M tokensMicrosoft Phi-4 (open-weight family) — in $0.13 / out $0.5 per M tokensMistral Large 3 — in $0.5 / out $1.5 per M tokensQwen3.7-Max — in $2.5 / out $7.5 per M tokens
harness-measured (7)priced, not yet measured (21)n=28 · prices from models.dev [1] · retrieved 2026-10-19

Cross-reference

The same figures as a table. The receipt column records whether first-party evidence exists for a model — it is a fact, not a score.

ModelInput $/M [1]Output $/M [1]ContextHarness receipt
Claude Fable 5$10$501000K6 rounds
Claude Haiku 4.5$1$5200K2 rounds
Claude Opus 4.8$5$251000K5 rounds
Claude Opus 5$5$251000K1 round
Claude Sonnet 4.6$3$15200K1 round
Claude Sonnet 5$2$101000K1 round
Grok 4.3$1.25$2.51000K1 round
Claude Fable 5.1$10$501000Knone yet
Claude Haiku 5.5$0.1$0.51000Knone yet
Claude Opus 4.5$5$25200Knone yet
Claude Opus 4.6$5$25200Knone yet
Claude Opus 5.5$4$201000Knone yet
Claude Sonnet 4.5$3$15200Knone yet
Claude Sonnet 5.5$2$101000Knone yet
Gemini 3.5 Flash$1.5$91000Knone yet
Gemini 3.7 Flash$0.75$3.751049Knone yet
Gemini 3.8 Flash$0.75$3.751049Knone yet
GPT-5.2 Pro$21$168196Knone yet
GPT-5.5$5$301000Knone yet
GPT-6 Astra$10$501050Knone yet
GPT-6 Sol$2$101050Knone yet
GPT-6.1 Sol$2$101050Knone yet
Grok 4.6$2$6500Knone yet
Grok 4.7$2$6500Knone yet
Kimi K2.6$0.95$4262Knone yet
Microsoft Phi-4 (open-weight family)$0.125$0.516Knone yet
Mistral Large 3$0.5$1.5256Knone yet
Qwen3.7-Max$2.5$7.51000Knone yet

Data sources & licences

  1. [1]models-devMITactiveretrieved 2026-10-19
  2. [2]openrouter-modelsOpenRouter ToS — redistribution not yet verifiedheld — licence unverifiedNo figures from this source appear on this page.
  3. [3]openrouter-rankingsOpenRouter rankings terms — redistribution not yet verifiedheld — licence unverifiedNo figures from this source appear on this page.

Named but never ingested — their terms do not permit republishing figures here:

Vendor-published benchmarks — Claude Sonnet 5

Released 2026-06-30

These are Anthropic's own numbers at launch, not measured by this harness — cited for context only.

Sonnet 5 edges the flagship Opus 4.8 on knowledge work while running at roughly 40% of the list price.

Pricing: $2 / $10 per million input/output tokens; Anthropic cancelled the September increase (checked 2026-09-07)

August 2026 Wave — Public-Report Orientation

Compiled: 2026-08-17
Synthesized from public vendor reports — not a harness result. No round was dispatched and no receipt exists for this entry. Treat the lane calls as orientation, not measurement.
No harness tally — vendor-reported strengths onlyModels: Grok 4.6 · Gemini 3.7 Flash · DeepSeek-V4-Pro · Qwen3.8-27B

Vendor reports point to a strong agentic/speed wave. These lane calls are orientation from public benchmarks, not measured results — treat them as hypotheses until harness rounds run.

The Proving Ground Methodology

How the harness executes evaluations to eliminate cherry-picking bias.

1

Standardized Task Envelopes

Target protocol: give each candidate the same task, constraints, tools and stop conditions. Inspect each receipt for the conditions actually recorded.

2

Multi-Model Blind Evaluation

Target protocol: blind the model identity and calibrate an independent judge against human-reviewed examples. A receipt must name its actual judge before independence can be claimed.

3

Maker ≠ Checker Cross-Verification

Target protocol: verify outputs independently and preserve deterministic checks. Historical receipts do not establish cross-provider review unless they record it.

4

Durable Proof Receipts

Published receipts record only the fields they contain. Token usage, latency, tool traces and code checks are required for future cost and reliability comparisons; missing fields remain unknown.

Read complete verification rules in tools/arena/README.md

Tactical Routing Guidelines

Working hypotheses from the measured rounds and the vendor reports above — labelled per source, revised as rounds accumulate.

Swarm / Queen orchestration + real-time tools

Grok 4.6 (Hermes)

Agentic post-training + reliability for loops, memory, and external data.

Fast volume agentic + coding speed

Gemini 3.7 Flash

Large lifts in coding/agentic benchmarks + leading output speed at intro price.

Cost-sensitive or local multi-agent systems

Qwen3.8-27B or DeepSeek-V4-Pro

Open, efficient, strong agentic results on consumer or budget infra.

Deep judgment + brand-craft

Claude Opus/Fable-class

Retained leadership on complex situational judgment and high-craft output.

Verify / checker (mandatory CROSS-MODEL-GATE)

Rotate provider different from maker (Gemini 3.7 or Qwen)

Independent verification catches harness-specific artifacts.

Heavy multi-step work, any model

Enforce contracts structurally (schemas, evals, human gates)

Output discipline degrades under load across all frontier models.

Caveats & Safety Guardrails

  • n small per task — directional signals. Re-test in your own harness.
  • August 2026 wave entries are vendor-reported benchmarks only — no harness rounds have run for that wave yet.
  • Pricing introductory for Gemini 3.7 Flash; weights for GLM-5.3 delayed.
  • Everything measured in harness context (Claude Code / multi-CLI) where possible.

Frequently Asked Questions

How do the new August 2026 models change routing for multi-agent systems?

Grok 4.6 for orchestration. Gemini 3.7 Flash for speed/volume. Open models (Qwen, DeepSeek) for cost and local. Claude remains for deepest judgment and craft. Always CROSS-MODEL-GATE verify.

Starlight Model Arena • built and run by Frank's multi-agent research system • 2026