6. Production Operations

6.1 Operate the outcome, not the endpoint

An AI system is available only when it produces a valid outcome within its operating envelope. A model endpoint can return HTTP 200 while the product supplies an unsupported answer, retrieves stale policy, skips an approval, or claims that a tool action succeeded when it did not. Infrastructure health is necessary; it is not the user contract.

Production operations therefore join four views:

  • Service health: availability, latency, saturation, errors, and dependency health
  • Behavioral health: groundedness, valid structure, abstention, tool success, and task completion
  • Control health: authorization, policy decisions, approvals, auditability, and safety signals
  • Economic health: tokens, accelerator time, provider spend, and cost per successful outcome

NIST’s 2026 monitoring report separates functionality, operational, human-factors, security, compliance, and large-scale-impact monitoring. It also identifies fragmented logs and difficulty detecting degradation as practical barriers.1 Keep these views distinct, but correlate them around the same request, task, and release.

6.2 Build an end-to-end telemetry spine

Every synchronous request and asynchronous task should carry a trace identifier through the gateway, harness, retrieval system, model calls, policy checks, tools, and external effects. Metrics reveal that a population is unhealthy; traces explain which path failed; structured events preserve significant decisions; audit records provide durable evidence for consequential actions.

sequenceDiagram
    actor U as User
    participant G as Gateway
    participant H as Harness
    participant R as Retrieval
    participant M as Model
    participant P as Policy broker
    participant T as Tool

    U->>G: request · trace_id 7f3…
    Note right of G: span ai.request
    G->>H: verified identity + trace context
    H->>R: permitted query
    Note right of R: span retrieval.query
    R-->>H: passages + provenance
    H->>M: versioned prompt + context
    Note right of M: span gen_ai.client.operation
    M-->>H: answer or action proposal
    H->>P: actor + proposal + policy version
    Note right of P: span policy.evaluate + audit event
    P->>T: approved narrow capability
    Note right of T: span tool.execute + effect record
    T-->>H: verified result
    H-->>G: outcome + artifact versions
    G-->>U: response

Figure 6-1. One request shows the correlation argument: a shared trace identifier joins named spans, while policy decisions and external effects also produce durable audit evidence. Span names are illustrative platform conventions.

At minimum, a trace should make these questions answerable:

  • Which tenant, product, environment, release, and pseudonymous actor initiated the work?
  • Which model, provider, route, prompt, policy, tool, retrieval index, and configuration versions ran?
  • Where was time spent, including queueing, time to first token, generation, retrieval, tools, retries, and verification?
  • Which sources entered model context, what freshness and authorization decisions applied, and what citations were returned?
  • What tokens and estimated cost were consumed, and did a cache, retry, fallback, or rate limit affect the path?
  • What final state was observed in the system of record?

OpenTelemetry provides common conventions for traces, metrics, logs, and events and now maintains GenAI-specific definitions in a dedicated repository.23 Use that vocabulary where it fits, but pin the schema version and add a small platform namespace for product outcomes. Conventions are evolving; uncontrolled instrumentation upgrades can silently break dashboards and alerts.

Do not record raw prompts, retrieved passages, outputs, secrets, or hidden reasoning by default. Prefer document identifiers, classifications, hashes, structured outcomes, and redacted samples. Apply field-level access, retention limits, regional controls, and an emergency access trail. Observe externally meaningful actions and decisions—not private chain-of-thought.

6.3 Define service objectives in user terms

Conventional service-level indicators remain important, but AI-specific objectives must distinguish transport success from valid response success.

Objective Illustrative indicator Operational use
Valid availability Requests completing on time with valid schema, required evidence, and no system error Primary page and error budget
Responsiveness p50/p95/p99 time to first token and total completion time Detect queueing and tail regressions
Task reliability Tasks reaching a verified acceptable state without duplicate effects Agent/workflow objective
Retrieval freshness Eligible sources indexed within the promised interval Knowledge pipeline alert
Safety and authorization Confirmed policy violations or unauthorized context/action Zero-tolerance incident signal
Quality Sampled groundedness, citation accuracy, abstention, and expert-review results Trend, release gate, investigation
Efficiency Tokens and cost per valid answer or completed task Budget and optimization review

Table 6-1. The exact thresholds and windows are product decisions. “Model API success rate” measures a dependency; product availability uses the valid-outcome SLI.

Page on conditions that require immediate human action: user-visible error-budget burn, widespread invalid outcomes, authorization violations, stuck or duplicate actions, and exhausted capacity. Quality metrics are often sampled, delayed, and uncertain; use them for trend detection and investigation unless a high-confidence signal indicates severe harm. Every objective needs an owner, measurement query, window, threshold, and response playbook.

Anthropic’s 2025 postmortem shows why behavioral monitoring matters: several infrastructure bugs intermittently degraded output quality, some affected small or route-dependent traffic slices, and normal service metrics did not make the problem obvious.4 Monitor by provider, model, hardware or endpoint route when visible, region, tenant, language, context length, tool path, and release. Aggregates hide partial failures.

6.4 Budget latency end to end

Set one product deadline, reserve time to return a controlled response, and propagate decreasing deadlines to every dependency. If retrieval, model, and tool calls each receive the full product timeout, the composed system cannot honor it.

Segmented eight-second latency budget

Figure 6-2. An illustrative latency budget. Actual budgets depend on the interaction; voice, chat, batch, and long-running agents need different contracts.

Measure queue time, time to first token (TTFT), inter-token latency, and total duration separately. Streaming improves perceived responsiveness but does not reduce completion time, and it complicates post-generation validation. Parallelize independent retrieval or policy work, shorten irrelevant context, cap output, cache stable prefixes, and move genuinely long work to durable asynchronous execution. Google’s 2026 GKE account provides a concrete serving example: prefix-aware routing avoided repeated prompt-prefix computation and materially improved TTFT and inter-token latency in its benchmark.5

Keep optimization subordinate to correctness. A faster fallback that cannot satisfy the grounding or tool-use contract is a different product behavior and must be evaluated and disclosed as such.

6.5 Contain failure and overload

Retries are load multipliers. Retry only transient, safe operations; honor Retry-After; use exponential backoff with jitter; cap attempts; and stop when the remaining deadline cannot support another try. Never blindly retry a mutation.

For external effects, issue an idempotency key tied to the approved intent, persist the attempt before execution, and reconcile against the system of record after ambiguous timeouts. The user-visible state is proposed, approved, submitted, confirmed, failed, or unknown; an ambiguous result never becomes a convenient “done.”

Use circuit breakers and bulkheads per provider, model route, tool, tenant, and workload class. Add concurrency limits and admission control at the gateway and harness; protect interactive traffic from batch traffic; bound the number of agent steps, parallel tools, tokens, and elapsed time. Capacity plans model token throughput and long-lived concurrency because request count alone conceals workload size. Queue durable work when waiting is acceptable and shed low-priority work before the platform collapses.

A multi-provider gateway can centralize rate limits, routing, spend, retries, and failover; AWS’s 2025 reference architecture illustrates this operational pattern.6 Provider diversity, however, is not automatic resilience. A fallback may differ in context size, tool schema, safety behavior, regional processing, latency, and quality. Qualify it with the same evaluation and compliance contract, then continuously exercise the path.

6.6 Design explicit degraded modes

Graceful degradation preserves truth and control even when the complete product experience is unavailable.

The following cases describe user-visible behavior when a shared dependency or control plane is unhealthy. They complement the boundary invariants in Chapter 3 and the single-execution failure contract in Chapter 4.

Failure Operations Copilot behavior Never claim
Primary model unavailable Use an evaluated equivalent route; otherwise queue or decline That any model is an equivalent fallback
Retrieval unavailable or stale Withhold policy answer; offer known source links or escalation That remembered model knowledge is current policy
Citation/grounding verifier unavailable Return a clearly unverified draft only where risk permits; otherwise abstain That the answer was verified
Service desk unavailable Answer policy if evidence is healthy; retain an approved proposal or ask the user to retry That a case exists without system-of-record confirmation
Control plane unavailable Run a time-bounded last-known-good signed bundle That new configuration or revocation has propagated
Telemetry pipeline unavailable Buffer bounded low-risk telemetry; fail closed for actions requiring an audit record That missing evidence can be reconstructed later

Table 6-2. Degraded modes are part of the product contract and must be tested. The safe behavior depends on data classification and action consequence.

This distinction also prevents correlated failure. If both the primary and fallback depend on the same gateway, identity provider, region, or policy store, provider diversity alone does not remove the common point of failure. Map dependencies, test fault injection at boundaries, and include control-plane and observability outages—not only model outages.

6.7 Engineer human control as an operational dependency

A “human in the loop” is often drawn as one box, but people have roles, queues, latency, limited attention, and distinct failure modes. Oversight is credible only when the organization defines who decides, what evidence they receive, how much work they can absorb, and what happens when they are unavailable or uncertain.

Role Decision or task Evidence required Common failure
End user Clarify intent, confirm data, approve a visible effect Plain-language action, destination, important fields, reversibility, data leaving the boundary Consent without understanding; habituated approval
Action approver Accept, reject, or narrow a consequential proposal Identity, policy basis, exact effect, model uncertainty, source provenance, prior related actions Rubber-stamping, queue pressure, missing domain context
Domain reviewer Judge sampled quality and adjudicate uncertain cases Stable rubric, source evidence, candidate and baseline behavior, relevant slice metadata Inconsistent labels, hindsight bias, treating fluency as correctness
On-call operator Diagnose, degrade, contain, reconcile, and restore End-to-end trace, release markers, dependency state, policy decisions, external-effect ledger, runbook Restarting healthy infrastructure during a behavioral failure; trusting an unverified completion claim

Table 6-3. Human control is a set of engineered roles and interfaces. “A person reviewed it” is insufficient evidence unless the decision, information, timing, and authority are recorded.

Review capacity belongs in the operating model. Measure arrival and completion rates, oldest-item age, time to decision, escalation rate, reversal rate, and the number of actions stuck in unknown. Define what the product does when the queue exceeds capacity: reduce exposure, disable consequential actions, route to another trained group, or decline safely. It must never respond to reviewer overload by silently widening automation.

Reviewer quality also needs calibration. Give multiple reviewers an overlapping sample, measure agreement by decision and risk slice, adjudicate material disagreements, and revise the rubric or interface when experts interpret it differently. Feedback from a reviewer is evidence, but it becomes a training label or release criterion only through a defined curation process. This protects the evaluation suite from reproducing hurried or inconsistent judgments.

Operators need a mental model of the compound system. Training and game days should require them to distinguish at least model-route regression, stale or unauthorized retrieval, policy denial, tool ambiguity, provider outage, and telemetry loss. They must be able to identify the safe degraded mode, determine whether external state changed, and know when to involve a domain, security, data, or vendor owner. NIST AI 800-4 treats human-factors monitoring as a distinct category; the practical implication is that operator and reviewer performance must be observed alongside software behavior.7

For the Operations Copilot, the employee approves the exact benefits queue and submitted fields. A reviewer sees the supporting policy and why the action requires review. If case creation times out, the operator sees an unknown mutation tied to its idempotency key and reconciles it against the service desk before retrying. Each person receives evidence suited to their decision rather than a transcript and a generic “approve” button.

6.8 Make incidents diagnosable and recoverable

Dashboards should move from outcome to cause: product SLO and business outcome; then route, model, retrieval, tool, and policy slices; then dependency saturation and infrastructure. Preserve deployment markers for prompts, models, indexes, policies, routing, and code so an operator can correlate behavior with change.

Each alert links to a runbook containing impact, likely causes, safe diagnostic queries, degradation options, rollback or traffic-shift steps, evidence to preserve, escalation owners, and communication guidance. Keep a last-known-good artifact set and practice rollback. Provider status pages are one diagnostic input; request traces and dependency evidence establish product impact.

Incident response should explicitly classify behavioral, security, data, dependency, capacity, and cost failures. A quality incident may require disabling one route or prompt version rather than restarting a service. After recovery, quantify affected requests and tenants, verify external effects, add the failure to evaluations, repair missing telemetry, and record why existing detection or containment failed.

OpenAI’s 2026 low-latency voice architecture is a useful systems example: it separates a narrow, horizontally scalable relay from stateful session termination and uses protocol-native routing information to preserve ownership while reducing the public network surface.8 The broader pattern is to isolate hot-path responsibilities and keep state ownership explicit so failure recovery is predictable.

The production contract is complete only when the platform can answer: what failed, who and what was affected, what behavior is safe now, whether any external state changed, and how the system returns to a verified state.

References

  1. NIST, “Challenges to the Monitoring of Deployed AI Systems”, NIST AI 800-4, March 2026.↩︎

  2. OpenTelemetry, “OpenTelemetry semantic conventions 1.43.0”, accessed July 31, 2026.↩︎

  3. OpenTelemetry, “GenAI Semantic Conventions”, accessed August 3, 2026.↩︎

  4. Anthropic Engineering, “A postmortem of three recent issues”, September 17, 2025.↩︎

  5. Google Cloud Blog, “GKE Inference Gateway prefix caching accelerates AI inference”, June 9, 2026.↩︎

  6. AWS Machine Learning Blog, “Streamline AI operations with the Multi-Provider Generative AI Gateway reference architecture”, November 21, 2025.↩︎

  7. NIST, “Challenges to the Monitoring of Deployed AI Systems”, NIST AI 800-4, March 2026.↩︎

  8. OpenAI Engineering, “How OpenAI delivers low-latency voice AI at scale”, May 4, 2026.↩︎