flowchart TB
C1["1 · Principles<br/>Define a valid outcome"] --> C2["2 · Operating model<br/>Separate control and runtime planes"]
C2 --> C3["3 · Identity and knowledge<br/>Control who and what enters context"]
C2 --> C4["4 · Compound systems<br/>Bound model, tool, and agent execution"]
C3 --> C5["5 · Evaluation<br/>Turn acceptable behavior into evidence"]
C4 --> C5
C5 --> C6["6 · Operations<br/>Correlate behavior with service health"]
C5 --> C7["7 · Security and governance<br/>Limit consequence and authority"]
C5 --> C8["8 · Delivery<br/>Release the evaluated system"]
C6 --> C9["9 · Economics and resilience<br/>Fund valid outcomes and recovery"]
C7 --> C9
C8 --> C9
C9 --> C10["10 · Decisions and readiness<br/>Preserve why and prove readiness"]
1. Purpose, Scope, and Engineering Principles
1.1 Purpose
A production AI platform is the shared technical and operational foundation through which an organization builds, deploys, evaluates, governs, and operates AI-enabled applications.
Its purpose extends well beyond model access. A model is one probabilistic dependency inside a system. The platform makes the surrounding identity, data, runtime controls, evaluation, and operations reusable across applications.
The platform makes the safe path the easiest path. It centralizes controls that benefit from consistency while application teams retain responsibility for domain logic, user experience, and use-case quality. Success is measured by how reliably teams turn evaluated ideas into accountable production services, rather than by the number of models exposed.
This handbook presents an opinionated approach to engineering that foundation.
1.2 Scope and audience
The handbook focuses on compound AI systems: applications in which models interact with prompts, retrieval systems, tools, APIs, deterministic workflows, policies, and human reviewers. Representative workloads include retrieval-augmented generation (RAG), document processing, conversational assistants, search, tool-using agents, and multi-step AI workflows.
The guidance applies to managed APIs, self-hosted models, and multi-provider environments. It is written for the engineers, architects, reliability and security practitioners, and technical leaders moving AI systems into production.
Foundation-model training, accelerator design, and jurisdiction-specific legal analysis are outside the primary scope. The platform must support governance and compliance, but this handbook does not replace organizational policy or legal advice.
1.3 Platform versus harness
An AI platform is the shared environment in which many AI systems are built and operated. It supplies capabilities such as model access, identity, retrieval, evaluation, observability, deployment, governance, and cost controls.
An agent harness is the runtime structure around a particular model or agent. It determines how that agent receives context, calls tools, maintains state, retries, verifies work, and stops. Frameworks and SDKs can help implement a harness, which then runs within—or consumes services from—the wider platform.
In short, the platform defines where AI systems operate and how they are governed; the harness defines how one agent executes reliably. I use “platform” in this deliberately demanding sense: a shared model proxy alone does not qualify. A harness is one component within the wider platform.
1.4 A running example: the Operations Copilot
The handbook uses one system to make its design choices concrete: an internal Operations Copilot that answers company-policy questions and, with user approval, creates service-desk cases.
Suppose an employee asks:
What is our parental-leave policy for employees in Kenya?
A 200 OK response does not establish success. The answer must retrieve the current Kenyan policy, respect the employee’s permissions, distinguish policy from inference, cite evidence, and meet its latency and cost objectives. A fluent answer based on an expired policy is still a failure.
If the employee asks the copilot to open a benefits case, the system crosses an authority boundary. The model may propose the action, but software must validate its arguments, confirm identity and permission, obtain approval, invoke the API, and record the result.
Chapter 3 develops the retrieval authorization boundary in detail, including why relevance cannot substitute for access control.
1.5 What makes an AI platform different
Traditional monitoring distinguishes successful requests from timeouts and errors. AI systems add another category: requests that execute as designed but produce an unacceptable outcome.
| Example outcome | Infrastructure view | System view |
|---|---|---|
| Current policy with valid citations | 200 OK |
Success |
| Expired policy stated confidently | 200 OK |
Quality failure |
| Restricted document included in context | 200 OK |
Security incident |
| Correct answer returned after 40 seconds | 200 OK |
Performance failure |
| Case created without confirmation | 200 OK |
Authorization failure |
| Provider unavailable; request safely declined | 503 |
Controlled degradation |
Table 1-1. Transport success and system success are different judgments. A useful platform preserves both.
A production AI platform manages product validity, operational reliability, and bounded risk together. These dimensions conflict: more context can improve grounding while increasing latency and data exposure; greater autonomy can complete more work while expanding the blast radius.
The platform exists to make these trade-offs explicit, measurable, and governable.
1.6 Engineering principles
Engineer the system, not the model
Production quality emerges from models, prompts, retrieval, tools, policies, application logic, and users. Model selection matters, but it is not the architecture.
Yelp’s 2025 LLM query-understanding rollout combined representative datasets, human correction, smaller fine-tuned models, caching, experiments, and staged rollout. For some tasks, the team reported up to a 100-fold cost reduction compared with directly using a complex frontier-model prompt.1
Treat behavior-changing components as versioned artifacts
Models, prompts, tools, policies, evaluation datasets, embeddings, and retrieval indexes change independently. A release must identify the combination evaluated and deployed.
If that state cannot be reconstructed, a regression cannot be reliably explained or rolled back. Provider model aliases should not be assumed to be immutable.
Make evaluation part of delivery
Tests establish whether code executes as specified; evaluations establish whether the AI system behaves acceptably. Material changes should be tested against representative tasks, known failures, adversarial inputs, and release thresholds.
Agent evaluation must inspect tool use and resulting state as well as the final response. Anthropic’s 2026 guidance describes grading complete multi-turn trajectories with deterministic checks, model graders, and human calibration.2
Bound autonomy and blast radius
Prefer deterministic workflows when the sequence of operations is known. Introduce model-directed planning only where variability creates measurable value.
Every agent needs limits on tools, data, steps, runtime, cost, concurrency, and approval-requiring actions. Sandboxing enables autonomy inside a boundary; Anthropic reported in 2025 that filesystem and network isolation reduced permission prompts in internal usage by 84% while limiting prompt-injection impact.4
Design for failure and degraded operation
Providers will experience rate limits, outages, regressions, and deprecations. Retrieval will become stale or incomplete. Tools will fail independently.
Define degraded modes: use a smaller model, disable optional tools, return conventional search results, queue work, escalate, or decline safely. A fallback is acceptable only when it preserves the safety and quality contract.
Optimize and observe outcomes, latency, cost, and risk together
There is no universally best model. Define acceptable quality, latency, cost, and risk envelopes, then select an architecture within them.
Telemetry must connect models, retrieval, tools, policy decisions, retries, and fallbacks to user-visible outcomes. It exposes errors, latency, tokens, cost, evaluation signals, and policy violations with suitable redaction and retention. AWS’s multi-provider gateway reference architecture illustrates the narrower gateway role: central routing, quotas, observability, and provider adaptation behind one interface.5
1.7 Definition of success
An AI platform is successful when teams can reach production without rebuilding foundational controls and when the organization can answer, for every deployed system:
- Who owns it? (Chapters 2 and 10)
- Which identities, data, models, and tools can it access? (Chapters 3, 4, and 7)
- What exact version is running? (Chapter 8)
- How was its behavior evaluated? (Chapter 5)
- What quality, reliability, latency, and cost objectives apply? (Chapters 6 and 9)
- What happens when a dependency fails? (Chapters 6 and 9)
- Which actions require human approval? (Chapters 3 and 7)
- How can the system be investigated, contained, and rolled back? (Chapters 6–8)
The following chapters develop an architecture that makes these answers concrete and verifiable.
Figure 1-1. The book follows an engineering argument rather than a catalog of components: define the outcome, establish control boundaries, gather evidence, operate the result, and preserve the decisions.
References
Yelp Engineering, “Search Query Understanding with LLMs: From Ideation to Production”, February 4, 2025.↩︎
Anthropic Engineering, “Demystifying evals for AI agents”, January 9, 2026.↩︎
Spotify Engineering, “Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3)”, December 9, 2025.↩︎
Anthropic Engineering, “Beyond permission prompts: making Claude Code more secure and autonomous”, October 20, 2025.↩︎
AWS Machine Learning Blog, “Streamline AI operations with the Multi-Provider Generative AI Gateway reference architecture”, November 21, 2025.↩︎