Agentic Evals
Trajectory receipts, swarm proof, and quality that can be replayed
Final-answer green-wash hides unsafe intermediate steps. Production agentic quality needs trajectory evaluation, independent checkers, exportable receipts, offline suites, and online traces. Agentic evals fix failure mode #4 of agentic life infrastructure: "the swarm worked" is not evidence.
Research briefs like this, when the evidence is ready. Source links, limitations, and open questions.
Subscribe2
Planes: offline regression + online tracing
Production eval doctrine
≠
Maker ≠ checker
Reliability pattern
traj
Trajectory > final answer alone
LangSmith/Braintrust-class practice
The Problem: Unverifiable Quality
Chat demos score vibes. Multi-tool agents score trajectories: tool choice, order, intermediate policy, retrieval, and side effects. Without evals, multi-agent systems scale confabulation with confidence.
Final-answer only
FailCorrect output via forbidden intermediate steps still fails production policy.
No offline suite
FailEvery change is a prod experiment; regressions ship silently.
No receipts
FailClaims of swarm success cannot be audited or replayed.
Maker judges self
FailSame model/agent grades its own work — circular quality theater.
Trajectory Evaluation
A trajectory is the sequence of thoughts, tool calls, observations, and intermediate states. Score tool selection, ordering, stop conditions, policy violations, and retrieval relevance — not only the last string.
Heuristic scorers
MethodDeterministic checks: required tools, banned tools, schema validity, latency budgets.
LLM-as-judge
MethodRubrics for intermediate quality; calibrate with human labels; track judge drift.
Human review
MethodHigh-stakes lanes (money, health, publish, legal) need human sampling or full review.
Reference-free RAG evals
MethodRagas-class faithfulness/relevance when ground truth is partial.
Platform Landscape (2026)
Choose platforms by stack fit and sovereignty posture. LangChain-native teams often start with LangSmith; eval-first CI teams with Braintrust; open/self-host with Langfuse/Phoenix/DeepEval/MLflow. FrankX uses estate receipts + starlight-evals-class harnesses.
Braintrust
PlatformEval-first scoring, experiments, CI-oriented quality loops.
LangSmith
PlatformTrajectory evals and tracing tightly integrated with LangChain/LangGraph.
Arize Phoenix
PlatformOTel-native observability and agent evaluators; strong self-host path.
DeepEval / Ragas / MLflow
PlatformOpen-source / pytest-style and experiment tracking options for offline suites.
Starlight Evals / Model Arena
OursReceipt JSON, harness doctrine, multi-round capability cards for estate-specific truth.
Benchmarks That Matter
Public benches are necessary but not sufficient. Always add private harness suites that rehydrate your real tools, memory, and gates.
AgentBench / SWE-Bench / tau-bench
BenchEnvironment and software-agent capability signals.
LongMemEval / LOCOMO
BenchMulti-session memory and temporal recall.
Private estate suites
BenchRehydrate new session; assert critical facts, gates, and non-overlapping write scopes.
Swarm dry-run
BenchSimulate multi-agent plans without irreversible side effects; score handoffs.
Receipts And Evidence Standards
A receipt is machine-checkable proof: paths, hashes, scores, screenshots, commit SHAs, or structured JSON. Chat assertions are not receipts.
Offline receipt
EvidenceSuite ID, dataset version, model version, scorecard, pass/fail gates.
Online receipt
EvidenceTrace ID, tool spans, policy flags, human override events.
Design/visual receipt
EvidenceExport inspection + score gates for premium UI/media (estate design loop).
Money/health fail-closed
EvidenceNo green merge or live action without explicit human approval record.
Maker ≠ Checker And Swarm Quality
Independent verifier agents, separate model families where possible, and non-overlapping write scopes prevent cosplay multi-agent teams. Isolation costs tokens — use when domain noise is real.
Independent verifier
PatternChecker does not share the maker’s prompt contamination or write tools.
Handoff contracts
PatternStop conditions, evidence required, and owner for each agent step.
Regression on merge
PatternPR gates run offline suites; draft PRs skip expensive jobs until ready.
Cost discipline
PatternDo not pay 15× multi-agent isolation for single-domain tasks.
How Evals Plug Into Life Infrastructure
Evals are the proof plane of agentic life architecture. They pair with sovereignty (exportable traces), memory (did retrieve/write happen?), and multi-domain gates (was the right module used?).
With ACOS
IntegrationContent quality gates, integrity-guard patterns, publish blockers.
With memory
IntegrationScore retrieval before act and write after act — not only chat recall quizzes.
With sovereignty
IntegrationTraces and scorecards must be exportable and rehostable.
With ops
IntegrationQueen/swarm boards only claim success when receipts exist.
Key Findings
Trajectory evaluation is required for multi-tool agents; final-answer scoring green-washes intermediate policy failures
Production quality uses two planes: offline regression suites and online tracing/observability
Maker ≠ checker is non-negotiable for swarm reliability
Platform choice depends on stack fit and sovereignty (self-host vs SaaS), not brand alone
Receipts are machine-checkable artifacts — paths, hashes, score JSON — not chat claims
Public benchmarks + private estate harnesses are complementary; either alone is incomplete
Money, health, publish, and secrets stay fail-closed with human approval records
Evals + sovereignty together make life infrastructure auditable and portable
Research Transparency
Limitations
- •Vendor platform pricing and feature matrices change quickly
- •LLM-as-judge variance can dominate small sample scores
- •Public benchmark leaderboards may not match private tool ecosystems
What We Don't Know
- ?Stable industry standard for cross-vendor trajectory export formats
- ?Optimal human-review sampling rates for multi-domain founder fleets
- ?How well RL-trained memory/eval policies transfer outside lab tasks
Frequently Asked Questions
No. Use heuristics for hard constraints, LLM-as-judge for rubrics, and human review for high-stakes lanes. Calibrate judges and watch drift.
Sources & References
16 source references · Last updated 2026-07-16
Published Articles
From research to practice
Learn these tools hands-on
The research maps the landscape. These portals curate the videos, docs, and experts to actually build with the platforms it covers.
Claude & Anthropic Mastery
Master Anthropic's full Claude stack — Opus 4.8, Sonnet 4.6, Haiku 4.5, Claude Code, the Agent SDK, MCP, Computer Use, and Skills — from first prompt to production agents.
Codex & OpenAI Agent Mastery
Master OpenAI Codex for agentic software work: setup, local CLI workflows, AGENTS.md, code review, and production-ready iteration.
ChatGPT & OpenAI Mastery
Master ChatGPT for everyday work, prompting, data analysis, custom workflows, and practical OpenAI fluency.
Gemini & Google AI Mastery
Master Google's full AI stack — Gemini 3.5 Flash, Gemini 3.1 Pro, Antigravity 2.0, NotebookLM, Veo 3.1, and Nano Banana Pro — from your first prompt to production agents.
Antigravity Mastery
Master Google Antigravity — the standalone agent-first development platform (desktop app, CLI, SDK) that replaced Gemini CLI — from first install to production multi-agent workflows.