5. Evaluation-Driven Development

5.1 Evaluation is the behavioral contract

An AI feature does not enter production until the team can state what acceptable behavior means and demonstrate it on representative tasks. I treat evaluation as part of the product specification: it defines the evidence required to change behavior, not just a score reported after implementation.

The contract must cover more than answer quality. Depending on the use case, it includes retrieval, grounding, task completion, tool behavior, authorization, safety, latency, and cost. Some criteria are optimization targets; others are non-negotiable gates. An average quality score must never compensate for a cross-tenant disclosure or an unauthorized action.

Evaluation begins before prompt or model selection. A team that cannot describe successful and unacceptable outcomes has an underspecified product. Adding a test suite cannot repair that ambiguity.

5.2 Evaluate the system at multiple layers

Testing only the final response makes failures difficult to diagnose. A poor answer may originate in ingestion, retrieval, context construction, model behavior, a tool, orchestration, or policy. Each independently changing component needs a corresponding evaluation layer.

Layer	What it establishes	Example grader
Code and contracts	Deterministic components behave correctly	Unit tests, schema checks, invariants
Retrieval	The right permitted evidence is found	Recall@k, ranking, freshness, ACL assertions
Generation	The response uses evidence and follows instructions	Groundedness, citation accuracy, rubric score
Tools and trajectory	Calls, state transitions, and effects are valid	Argument checks, policy assertions, state inspection
End-to-end outcome	The user task is completed acceptably	Outcome verifier or domain-expert rubric
Safety and security	Known harmful or unauthorized behavior is blocked	Adversarial suite and red-team scenarios
Operations	The service meets its runtime envelope	Latency, availability, token and cost limits

Table 5-1. Evaluation layers localize failures while preserving an end-to-end product decision.

Google’s 2025 evaluation guidance makes the same diagnostic distinction for RAG: retrieval quality and generation accuracy must be measured separately. For agents, the trajectory as well as the response becomes an evaluation object.¹

flowchart LR
    CH["Candidate change"] --> RUN["Evaluation runner"]
    DS["Versioned datasets and slices"] --> RUN
    RUN --> SUT["System under test"]
    SUT --> EV["Outputs, traces, and resulting state"]
    EV --> DG["Deterministic graders"]
    EV --> MG["Model graders"]
    EV --> HG["Human review sample"]
    DG --> REP["Evaluation report"]
    MG --> REP
    HG --> REP
    REP --> GATE{"Release gate"}
    GATE -->|"pass"| DEP["Shadow, canary, or deployment"]
    GATE -->|"fail"| ERR["Error analysis"]
    DEP --> MON["Production monitoring and feedback"]
    MON --> MINE["Curate new cases"]
    MINE --> DS

Figure 5-1. Evaluation is a lifecycle. Production evidence becomes curated test data; it does not bypass review and flow directly into the suite.

5.3 Build datasets that represent decisions

An evaluation case contains an input, controlled environment, relevant identity and data state, expected properties, grader configuration, and metadata describing why the case matters. For agents, it also defines available tools, budgets, initial state, and acceptable end states.

A useful portfolio contains several datasets:

• Smoke suite: small, fast checks for broken contracts and infrastructure
• Regression suite: previously supported behavior that should remain near-perfect
• Capability suite: difficult tasks used to discover and measure improvement
• Challenge suite: rare, ambiguous, multilingual, long-context, and boundary cases
• Safety suite: prompt injection, data leakage, excessive agency, and prohibited content
• Production sample: privacy-reviewed, stratified cases used to test representativeness

Anthropic’s 2026 agent-evaluation guidance distinguishes capability evaluations, which provide a hill to climb, from regression evaluations, which prevent loss of established behavior.² Mixing the two into one aggregate number makes a system appear to improve simply because easy cases dominate.

Dataset slices reflect product risk. Start with intent, user or tenant boundary, language, and high-consequence tool paths; add dimensions such as geography, input length, model route, and known failure class where they change the decision. Include common traffic for overall performance, deliberately overweight costly failures and underrepresented groups, and preserve the natural production distribution separately so business-level estimates remain interpretable.

Every dataset and label should be versioned with provenance, owner, collection window, annotation instructions, and permitted uses. Keep a blind holdout where practical. Prevent evaluation examples and reference answers from entering prompts, retrieval corpora, fine-tuning data, or judge context.

5.4 Choose graders by what can be verified

Use the most objective grader available:

1. Deterministic graders for schemas, exact facts, permissions, calculations, state changes, tests, database results, and citations.
2. Model graders for qualities such as coherence, completeness, groundedness, or tone when deterministic verification is insufficient.
3. Human graders for domain judgment, ambiguous policy, calibration, user impact, and disputes.

Model graders require their own evaluation. Define a narrow rubric with observable criteria, hide irrelevant information such as model identity, randomize order in pairwise comparisons, and compare judge decisions with expert labels. Track false-pass and false-fail rates on important slices. A grader that agrees overall but misses security or minority-language failures is not calibrated for the release decision.

Prefer pairwise comparison when experts can more reliably identify the better of two responses than assign an absolute score. Require a written rationale for debugging, but compute the gate from structured grader fields. Never use the candidate system as its sole judge.

OpenAI’s 2026 account of its internal data agent provides a concrete pattern: manually authored questions are paired with golden SQL; candidate and reference queries are executed; both the SQL and resulting data are compared so semantically correct alternatives are not rejected by string matching.³

5.5 Measure variability and uncertainty

One successful run does not establish reliability. A task is a test case; each attempt is a trial. Run multiple trials when sampling, tool choice, concurrency, or external data can alter the outcome.

Report the distribution. The mean alone hides important release risk:

• Pass rate and confidence interval
• Worst and lower-percentile performance on critical slices
• All-trials-pass rate for behavior that must be consistent
• Severity-weighted failure count
• Latency, tokens, tool calls, and cost distributions

State what the result generalizes to. Performance on a fixed benchmark is not automatically performance on future production inputs. NIST’s 2026 work on statistical models for AI evaluation warns that common reporting can conflate fixed-benchmark accuracy with generalized performance and can misstate uncertainty.⁴

Small changes close to a threshold should not be promoted on noise. Use paired comparisons on the same cases, confidence intervals or another predeclared statistical method, and a minimum meaningful effect. Preserve raw trial results so conclusions can be recomputed.

Illustrative trial distributions with the same mean

Figure 5-2. The average can hide the release risk. These illustrative candidates have the same mean score, but one has severe low-tail failures that require slice and case-level investigation.

5.6 Evaluate RAG as retrieval plus generation

For the Operations Copilot, the retrieval evaluation corpus should contain questions, the documents or passages that support them, relevant distractors, effective dates, and the identity permitted to access each source.

Retrieval measures include:

• Whether at least one sufficient permitted passage appears in the top results
• Rank and coverage of all evidence required for the answer
• Exclusion of expired, superseded, cross-tenant, or unauthorized material
• Performance by document type, language, source, and query ambiguity
• Retrieval latency and index freshness

Generation measures include factual correctness, groundedness, citation precision and completeness, instruction following, appropriate abstention, and whether uncertainty is communicated. Test retrieval and generation both independently and end to end. A model should not receive credit for answering correctly from memorized knowledge when the product contract requires current company policy and citations.

5.7 Evaluate agents by outcomes and constrained trajectories

Agent evaluation adds an environment and resulting state. Grade:

• Whether the requested outcome occurred
• Whether external state is correct and free of duplicate effects
• Whether every action was authorized and within budget
• Whether required approval and validation boundaries were observed
• Whether the agent recovered safely from tool errors
• Whether it terminated rather than looping or claiming false completion

Do not require one exact tool sequence unless order is part of the safety or business contract. Several valid paths may reach the same correct state. Anthropic recommends grading outcomes where possible while inspecting trajectories for policy violations, inefficiency, and diagnosis.⁵

Agent test environments are isolated and resettable. Tools need deterministic fixtures for success, timeout, denial, malformed output, partial completion, and conflicting state. For consequential actions, verify the external system instead of accepting the agent’s statement that the action succeeded.

5.8 Turn evaluation into a release gate

Every candidate release should be compared with the current production baseline using the same datasets, environment, and grader versions. The report records code, model, prompt, retrieval index, tools, policy, dataset, grader, and runtime configuration.

An illustrative scorecard for the Operations Copilot might be:

Criterion	Gate	Type
Unauthorized passage reaches model context	0 cases	Hard gate
Unapproved or duplicate external action	0 cases	Hard gate
Current-policy retrieval recall@5	≥ 98%	Quality gate
Citation-supported material claims	≥ 97%	Quality gate
Appropriate clarification or abstention	≥ 95%	Quality gate
Previously passing critical regressions	100%	Regression gate
End-to-end p95 latency	≤ product objective	Operational gate
Mean cost per completed task	No material regression without approved trade-off	Optimization

Table 5-2. Illustrative thresholds only. Actual gates require product-specific baselines, risk analysis, and sufficient sample sizes.

Hard gates are evaluated before weighted scores. Results must also be inspected by slice: a passing aggregate cannot hide a failure in Kenya-specific policy, a language, a tenant, or a high-risk tool path.

Not every change needs the full suite. Run smoke and targeted tests during development, regression tests on each candidate, and broad capability and safety suites before significant releases. High-risk changes require independent review and recorded sign-off.

5.9 Continue evaluation in production

Offline datasets cannot reproduce every user, dependency, and failure condition. Use shadow evaluation where data handling permits, then canary releases or controlled experiments with explicit rollback criteria. Sample production traces for automated scoring and expert review, stratified by risk and low-confidence signals rather than only random traffic.

Monitor shifts in intents, retrieval sources, model routes, tool use, abstention, quality signals, latency, and cost. User feedback is useful evidence but not ground truth: a thumbs-up may reward style over correctness, while users may not detect a plausible error.

NIST’s 2026 report on deployed AI monitoring emphasizes that predeployment testing occurs in controlled conditions and must be complemented by postdeployment observation of nondeterminism, changing inputs, and unforeseen consequences.⁶ Production failures should enter an error taxonomy, receive expert review, and become regression cases after privacy and representativeness checks.

For every change, the release owner should be able to answer three questions: Which behavior changed? What evidence supports the new behavior? How will production tell us if that evidence was wrong?

---

References

---

1. Google Cloud Blog, “Master Generative AI Evaluation: From Single Prompts to Complex Agents”, December 15, 2025.

2. Anthropic Engineering, “Demystifying evals for AI agents”, January 9, 2026.

3. OpenAI Engineering, “Inside OpenAI’s in-house data agent”, January 29, 2026.

4. NIST, “Expanding the AI Evaluation Toolbox with Statistical Models”, February 17, 2026.

5. Anthropic Engineering, “Demystifying evals for AI agents”, January 9, 2026.

6. NIST, “Challenges to the Monitoring of Deployed AI Systems”, NIST AI 800-4, March 2026.