4. Model Access and Compound AI Systems

4.1 Choose the simplest sufficient topology

A compound AI system combines a model with deterministic software, context, retrieval, tools, state, and policy. The model is used where interpretation or generation is valuable; ordinary software retains responsibility for known rules, authorization, validation, and transaction control.

“Agentic” is not a maturity level. More autonomy introduces more possible execution paths, which increases evaluation scope, latency, cost, and operational risk. The appropriate topology is the least complex one that meets the product requirement.

Topology Use when Avoid when
Single model call The task is bounded: classify, extract, transform, rank, or generate External state or multiple dependent operations are required
Deterministic workflow Steps and decision points are known in advance The path cannot be enumerated without excessive branching
Tool-using agent The next step depends on intermediate results or open-ended exploration A fixed workflow can accomplish the task
Multi-agent system Valuable work can be decomposed into independent, parallel investigations Agents require the same context or have tightly coupled dependencies

The Operations Copilot therefore uses different topologies for different jobs. Answering a policy question is a fixed retrieve-and-generate workflow. Opening a benefits case adds a structured proposal, validation, approval, and commit step. Neither requires an open-ended agent. A request to compare employment rules across twenty countries may justify parallel research agents, provided their work remains read-only and their evidence is independently verified.

4.2 Govern model access through capability contracts

Applications access models through the platform gateway instead of embedding provider credentials and endpoints. They request a logical capability such as grounded-answer/standard or document-extraction/low-latency, and the gateway resolves that capability to an evaluated model and configuration.

A capability contract specifies:

  • Supported task and modality
  • Required output schema and maximum context
  • Data classification, provider eligibility, and residency
  • Quality baseline and evaluation suite
  • Latency and availability objective
  • Token, request, and cost limits
  • Streaming, tool-use, and structured-output requirements
  • Approved fallback behavior

Routing policy first applies deterministic constraints: a model that cannot receive the data class or operate in the required region is ineligible regardless of quality. Among eligible routes, selection may consider task evaluation, latency, availability, and cost.

Do not treat fallback as “call the next model.” A fallback is safe only if it has passed the same use-case evaluations and supports the same schema and policy contract. For a high-impact task, declining or queueing the request may be preferable to returning a lower-quality result. AWS’s multi-provider gateway reference architecture illustrates the narrower operational value of a shared interface: centralized authentication, provider routing, quotas, observability, and cost controls without embedding provider-specific integration in every application.1

Pin model versions where the provider permits it, record the resolved provider and version on every trace, and test provider aliases for behavioral change. Model promotion should use offline evaluation followed by shadow traffic, a canary, or a controlled experiment. Rollback changes the capability mapping rather than requiring an application deployment.

Model adaptation: fine-tuning and distillation

Use the most reversible mechanism that fixes the measured failure. Prompt and context changes are usually the fastest intervention; retrieval supplies current or access-controlled knowledge; tools add deterministic capability. Model adaptation is justified when the remaining behavior is stable, repeated, measurable, and valuable enough to own as another model artifact.

Need First mechanism to test Why
Current facts, private documents, or changing policy Retrieval with source authorization Knowledge can change or be revoked without training a new model
A small set of output rules or examples Prompt, schema, and deterministic validation Easier to inspect, evaluate, and roll back
Stable domain language, style, classification, or structured behavior at volume Supervised fine-tuning Repeated examples can move task behavior into a reusable checkpoint
Lower latency or cost for a narrow, well-evaluated task Distillation into a smaller student model A stronger teacher can generate or label task-specific training examples
A missing external capability or business transaction Tool or deterministic workflow Training cannot create authority, current state, or a reliable side effect

Fine-tuning updates a base model using curated examples or preference data. Distillation trains a student to reproduce selected behavior from a stronger teacher, often to reduce inference cost or latency. Neither mechanism is a general upgrade. Moving behavior into weights makes it harder to inspect and patch, and improvement on the target task can coincide with regression elsewhere. A July 2026 AWS study, for example, reported substantial loss of retained reasoning ability under some supervised fine-tuning configurations and evaluated self-distillation as one mitigation.2 Treat such results as a warning about evaluation scope, not as a universal recipe.

The platform contract for an adapted model includes:

  • Permitted, versioned training and validation datasets with provenance, consent or license, retention, and contamination checks
  • Base-model snapshot, adaptation method, hyperparameters, code, random seed where supported, and resulting artifact digest
  • Separate target-task, safety, general-capability, and holdout evaluations against the current production route
  • Serving requirements, data eligibility, cost, latency, capacity, and compatibility with schemas and tools
  • Named owner, approval record, rollback target, review triggers, and deletion or retirement procedure

For the Operations Copilot, company policy facts should remain in permission-aware retrieval because they change and may be revoked. A fine-tuned or distilled model could be justified for a stable, high-volume subtask such as classifying question intent or normalizing case fields, but only if it beats the existing route on representative slices without weakening authorization, multilingual behavior, or abstention. Yelp’s 2025 query-understanding system demonstrates this narrower pattern: smaller fine-tuned models were combined with representative data, human correction, caching, experiments, and staged rollout rather than treated as a standalone solution.3

4.3 The application harness

The harness is the application-owned execution layer around the model. It assembles context, runs the model loop, dispatches tools, checkpoints state, enforces budgets, and decides when work is complete. The platform supplies common primitives; the application owns the behavior.

flowchart LR
    IN["Verified request"] --> PB["Policy and budget"]
    PB --> CB["Context builder"]
    CB --> ML["Model loop"]
    ML -->|"tool proposal"| TB["Tool broker"]
    TB -->|"validated result"| ML
    ML -->|"candidate outcome"| VR["Verifier"]
    VR -->|"correctable failure"| ML
    VR --> OUT["Response or approved action"]

    ST["Task state and checkpoints"] <--> ML
    REG["Versioned prompts, tools, and model route"] --> CB
    LIM["Steps, time, tokens, cost, and permissions"] -.-> ML
    LIM -.-> TB

Figure 4-1. A harness turns model inference into bounded execution. The model proposes; deterministic components constrain, persist, validate, and commit.

A production harness makes its control state explicit. At any point, operators can identify the task, current phase, artifact versions, tools invoked, external effects, remaining budget, and termination reason. Framework-specific message objects are transient implementation details; the durable record uses stable platform contracts.

Anthropic’s 2026 Managed Agents architecture similarly separates the session log, harness, and sandbox behind stable interfaces so that context strategies and model behavior can evolve without redefining execution storage.4

4.4 Context is constructed, not accumulated

Context includes more than a prompt. It contains system instructions, tool definitions, user input, retrieved evidence, message history, tool results, memory, and the current task state. Context engineering determines which of these items enter each model call and how their authority is represented.

Treat the context window as a scarce working set. Durable state belongs in a database or object store. Anthropic’s 2025 context-engineering guidance describes the problem as continually curating the most useful tokens from a growing universe of possible state.5

The harness:

  • Keep trusted platform policy and application instructions separate from untrusted user, document, web, and tool content
  • Attach source and trust labels to retrieved evidence and tool results
  • Include only tools available in the current phase and authority scope
  • Prefer handles, targeted queries, and summaries over dumping large datasets into context
  • Compact history around durable facts, decisions, unresolved work, and provenance
  • Reinsert non-negotiable instructions after compaction rather than relying on a summary to preserve them
  • Measure context size, cacheability, retrieval contribution, and information discarded

Compaction is a lossy transformation and should be versioned and evaluated. It must not silently turn untrusted text into trusted instructions or erase the provenance needed to verify a claim.

Illustrative context budget before and after compaction

Figure 4-2. A context window is an allocation decision. After compaction, trusted instructions are reinserted from their source; selected history becomes a provenance-bearing summary; working reserve remains for the next step. Percentages are illustrative.

4.5 Design tools as narrow capabilities

Tools are the model’s interface to the outside world. MCP can standardize discovery and invocation, but the protocol itself does not make a tool safe, comprehensible, or reliable.

By 2026, an internal MCP registry is a platform capability rather than a list in each developer’s configuration. The registry records server version, owner, supported transports, data classifications, approved tools, OAuth resource and audience, and revocation state. It exposes only the servers and tools permitted for the current tenant and task, detects remote contract changes, supports emergency removal, and emits discovery and invocation telemetry. The authorization rules in Chapter 3 still apply at each server; registry approval never turns returned content into trusted instructions.

Effective tools:

  1. Perform one clear operation with a precise name and description.
  2. Accept and return bounded, typed schemas with explicit units and enums.
  3. Separate read, propose, validate, and commit operations.
  4. Validate all model-generated arguments on the server.
  5. Use idempotency keys for retried mutations and expose operation status.
  6. Return concise, structured errors that distinguish retryable, denied, invalid, and terminal outcomes.
  7. Enforce identity, permissions, timeouts, quotas, and audit independently of the harness.

Keep the tool catalog small for each task. Anthropic’s 2025 tool-design work emphasizes selective tool design, clear namespaces, and evaluation against representative tasks. It also describes using agents to analyze transcripts and improve tool implementations and descriptions, with held-out tests to detect overfitting.6 Spotify reached a similar conclusion for its background coding agent: limiting tools and hooks reduced context noise and sources of unpredictable failure.7

Tool results remain untrusted input. A document, issue description, webpage, or database field can contain instructions intended to redirect the model. The harness must identify results as data, constrain their size and format, and never allow returned content to expand tool permissions.

4.6 State, memory, and long-running work

Production systems distinguish three forms of state:

  • Session history: an append-only record of messages, calls, results, approvals, and errors
  • Task state: explicit progress, checkpoints, pending effects, budgets, and completion status
  • Durable memory: selected facts intended to influence future sessions

Session and task state are operational necessities. Durable memory is a product feature and should be opt-in, attributable, editable, expirable, and subject to access and deletion policy. Do not promote model-generated assumptions into durable memory without validation.

Long-running work requires workflow-engine semantics: durable checkpoints, leases, idempotent steps, retry policies, cancellation, deadlines, and recovery after process failure. Persist externally observable decisions and effects; do not make hidden model reasoning a recovery dependency. OpenAI’s 2026 description of a stateful agent runtime likewise identifies state storage, tool invocation, error handling, resumability, approvals, and permission boundaries as the operational work behind multi-step agents.8

4.7 Bound and verify execution

Every model-directed loop needs limits on elapsed time, turns, model tokens, tool calls, parallel workers, spend, and external effects. Termination should be explicit: success, insufficient evidence, policy denial, budget exhaustion, cancellation, repeated lack of progress, or unrecoverable failure.

Verification should be independent of the model where possible. Use schema validation, invariants, permission checks, database constraints, tests, reconciliation queries, and domain rules before using another model as judge. Spotify’s 2025 background-agent design runs deterministic verifiers before a pull request can be opened, then adds a model judge for less mechanical scope errors.9

flowchart TD
    Q["Requested outcome"] --> E{"Can the path and decisions<br/>be enumerated?"}
    E -->|"yes"| X{"Does it change<br/>external state?"}
    X -->|"no"| W["Deterministic workflow<br/>with bounded model calls"]
    X -->|"yes"| WA["Deterministic workflow<br/>with validation and approval"]
    E -->|"no"| O{"Is the work independently<br/>parallelizable?"}
    O -->|"no"| A["Bounded tool-using agent"]
    O -->|"yes"| M["Supervisor + bounded workers"]
    W --> V["Verify outcome"]
    WA --> V
    A --> V
    M --> V

Figure 4-3. Choose topology from the uncertainty in the path and the consequence of external effects. Agent loops are an exception justified by the task.

4.8 Use multiple agents deliberately

Multiple agents can create independent context windows, parallelize searches, and isolate toolsets. They also multiply tokens, failure modes, coordination paths, and evaluation cases.

Anthropic reported that an Opus 4 lead with Sonnet 4 subagents outperformed a single Opus 4 agent by 90.2% on one internal, breadth-oriented research evaluation. Its single research agent used roughly four times the tokens of ordinary chat; the multi-agent configuration used roughly fifteen times. The post also warns that tightly coupled tasks are a poor fit.10 These results support a bounded decision rule; they are not a universal performance claim. I use multiple agents only when the work is valuable, independently parallelizable, and too broad for one context.

A supervisor should assign bounded tasks with expected outputs, budgets, and tool scopes. Workers return evidence and status rather than conversational prose alone. Partial failure should be visible: the final result records which branches completed, failed, or timed out. Subagents must never inherit broader permissions merely because a parent agent delegated work.

4.9 Execution failure contract

Failure Required behavior
Primary model is unavailable Use only an evaluated fallback; otherwise queue or decline
Structured output is invalid Reject or repair within a bounded retry budget
Mutation times out Reconcile using the idempotency key before retrying
Agent repeats a step without progress Terminate and preserve diagnostic state
Context exceeds its budget Compact using a tested policy or reduce task scope
Tool returns hostile instructions Treat as untrusted data; do not alter policy or authority
One subagent fails Report incomplete coverage or retry only that bounded branch

This table covers failures contained within one execution: invalid output, lost progress, ambiguous effects, exhausted context, and bounded worker failure. Identity and knowledge invariants appear in Chapter 3; service-level dependency degradation appears in Chapter 6.

Before approving a more autonomous topology, the design review must answer four questions: Which uncertainty requires model-directed planning? What bounds each loop and tool? How are effects verified? What evaluation evidence shows that the added complexity improves the product?

References

  1. AWS Machine Learning Blog, “Streamline AI operations with the Multi-Provider Generative AI Gateway reference architecture”, November 21, 2025.↩︎

  2. AWS Machine Learning Blog, “Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova”, July 21, 2026.↩︎

  3. Yelp Engineering, “Search Query Understanding with LLMs: From Ideation to Production”, February 4, 2025.↩︎

  4. Anthropic Engineering, “Scaling Managed Agents: Decoupling the brain from the hands”, April 8, 2026.↩︎

  5. Anthropic Engineering, “Effective context engineering for AI agents”, September 29, 2025.↩︎

  6. Anthropic Engineering, “Writing effective tools for agents—with agents”, September 11, 2025.↩︎

  7. Spotify Engineering, “Background Coding Agents: Context Engineering (Honk, Part 2)”, November 24, 2025.↩︎

  8. OpenAI, “Introducing the Stateful Runtime Environment for Agents in Amazon Bedrock”, February 27, 2026.↩︎

  9. Spotify Engineering, “Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3)”, December 9, 2025.↩︎

  10. Anthropic Engineering, “How we built our multi-agent research system”, June 13, 2025.↩︎