9. Economics and Resilience

9.1 Optimize cost per valid outcome

The cheapest model call is not necessarily the cheapest system. A route may have a low token price but require more retries, longer prompts, extra verification, or human correction. An agent may reduce cost per step while taking twice as many steps. The useful economic unit is therefore:

Cost per valid outcome = fully loaded platform cost ÷ outcomes that satisfy the product contract

For the Operations Copilot, an outcome is not “one model response.” It is a permission-correct, grounded answer or a verified service-desk case completed within the latency and safety envelope. Failed, abandoned, duplicated, and manually reworked tasks remain in the numerator.

Fully loaded cost includes model input, cached input, reasoning and output tokens; embeddings and reranking; tool APIs; storage and networking; self-hosted accelerator idle time; orchestration; evaluation; observability; security controls; support; and human review. Separate fixed platform cost from marginal request cost, but show both when deciding whether a use case is viable.

flowchart LR
    subgraph A["Route A · evaluated baseline"]
        A1["1,000 tasks"] --> A2["$100 fully loaded spend"]
        A1 --> A3["900 valid outcomes"]
        A2 --> A4["$0.111 per valid outcome"]
        A3 --> A4
    end
    subgraph B["Route B · cheaper calls, weaker outcomes"]
        B1["1,000 tasks"] --> B2["$70 fully loaded spend"]
        B1 --> B3["500 valid outcomes"]
        B2 --> B4["$0.140 per valid outcome"]
        B3 --> B4
    end

Figure 9-1. Illustrative economics: Route B cuts total spend by 30% but raises cost per valid outcome by 26%. Component savings are not product savings when validity falls.

9.2 Attribute consumption before optimizing it

The gateway and harness record usage by tenant, team, product, feature, release, model route, environment, and task type. Attribute shared infrastructure through a declared method and report unallocated cost rather than hiding it. Produce:

  • Per-request tokens, cache status, model and provider cost, tool cost, retries, and latency
  • Cost per valid answer, completed case, active user, and business process
  • Distribution and tail, not only the mean; agent loops create expensive outliers
  • Forecast versus actual by team and use case
  • Cost changes by release, traffic mix, quality slice, and provider route

Real-time enforcement and financial billing have different consistency needs. Reserve or decrement a conservative budget before expensive work, then reconcile it against authoritative provider and infrastructure records. Preserve idempotency and an adjustment ledger. OpenAI’s 2026 account of scaling Codex and Sora describes a related pattern: rate limits protect fairness and capacity, while real-time credit accounting is reconciled for accurate, auditable consumption.1

Budgets govern behavior before the month-end report arrives. Apply per-task token, step, time, and tool limits; per-tenant concurrency and daily spend limits; and platform-wide anomaly stops. Warn before rejecting ordinary traffic, but stop recursive or abusive work quickly. A budget override is scoped, expiring, attributable, and visible to the owner.

9.3 Optimize in the right order

First remove work that creates no value. Then choose the least expensive architecture that still meets the evaluation, latency, security, and reliability contract.

Lever Engineering action Failure to test
Eliminate waste Remove duplicate calls, runaway steps, unnecessary judges, retry storms, and unused context Hidden task failures or missing safety checks
Reduce context Improve retrieval, summarize safely, trim history, lazy-load tool definitions Lost evidence, permissions, or conversational state
Bound output Use concise schemas and task-specific limits Truncated facts or invalid structured output
Route by task Use smaller models for evaluated slices; escalate uncertainty or complexity Quality regressions concentrated in rare slices
Cache stable work Cache deterministic retrieval, embeddings, or stable prompt prefixes Stale policy, cross-tenant leakage, invalidation errors
Batch and defer Move non-interactive evaluation, enrichment, and indexing to cheaper capacity Violated freshness or response objectives
Improve serving Increase utilization, cache locality, batching, quantization, and autoscaling efficiency Tail-latency or quality loss under production traffic
Change the product Ask for clarification, limit features, or use deterministic software Moving cost to users or human operations

Table 9-1. Each cost lever is a behavioral change and goes through the release and evaluation process from Chapters 5 and 8.

Model routing is explicit, observable, and evaluated by task slice. AWS’s 2025 prompt-routing report is a concrete example of measuring routing against response quality, cost, and latency instead of assuming that the smaller model is adequate.2 A route must also account for the router’s own latency and cost. I would reject an optimization that reports token savings without the candidate-versus-baseline outcome distribution.

Caching is most effective when prompts place stable instructions and shared context in reusable prefixes. It is also a data store: partition by tenant and policy, encrypt it, set retention, and define invalidation on prompt, model, permission, or source change. In self-hosted inference, prefix-aware routing can improve cache locality and utilization; AWS reported lower cost and higher throughput in its 2025 tiered KV-cache benchmark, but the result depends on workload and hardware.3

Do not cache a generated policy answer merely because two questions are semantically similar. Cache evidence or deterministic intermediate results where possible; if final answers are cached, bind them to identity, source versions, policy, model release, and expiry.

9.4 Plan capacity in tokens and concurrent work

Requests are a poor capacity unit. One short classification and one hour-long agent task are both one request. Forecast input and output tokens, context-length distribution, time to first token, concurrent generations, tool concurrency, queue duration, and accelerator memory by workload class. Include evaluation jobs, reindexing, incident bursts, retries, and failover headroom.

Protect interactive, consequential, and batch work with separate queues or quotas. Admission control rejects or defers work before dependencies saturate. Maintain a demand waterfall: reserved entitlement, burst allowance, paid overage or lower priority, then controlled rejection. This preserves fairness while making capacity economics visible.

flowchart LR
    D["Incoming demand"] --> R{"Within reserved<br/>entitlement?"}
    R -->|"yes"| Q1["Reserved queue"]
    R -->|"no"| B{"Burst capacity<br/>available?"}
    B -->|"yes"| Q2["Burst queue"]
    B -->|"no"| O{"Paid overage or<br/>lower priority allowed?"}
    O -->|"yes"| Q3["Deferred / overage queue"]
    O -->|"no"| X["Controlled rejection<br/>with retry guidance"]

Figure 9-2. The demand waterfall turns capacity policy into deterministic admission behavior before dependencies saturate.

For provider APIs, validate quotas by region and model, obtain required reservations, and exercise increases before launch. For self-hosting, compare total cost at realistic utilization—not theoretical tokens per second. Include engineering, upgrades, security patching, model storage, spare capacity, energy, and idle accelerators. A hybrid design can keep predictable baseline traffic on committed capacity and absorb bursts through an evaluated provider route, but complexity is itself a cost.

9.5 Buy resilience deliberately

Resilience has an explicit price: spare capacity, duplicated state, multiple providers, recovery engineering, and regular exercises. Spend according to business impact rather than making every component active-active.

High availability handles expected component failures with minimal interruption. Disaster recovery restores service after a wider loss of region, control plane, data, credentials, or operator access. Define for each use case:

  • Recovery time objective (RTO): maximum acceptable time to restore an agreed service level
  • Recovery point objective (RPO): maximum acceptable loss of committed state
  • Degraded service objective: what safe capability remains during recovery
  • Maximum tolerable outage: the business deadline beyond which alternative processes take over

An RTO without a tested procedure and provisioned dependencies is an aspiration. A backup without a successful restore is an unverified copy.

9.6 Classify state by how it is recovered

Not every AI artifact needs replication. Derived data can often be rebuilt; authoritative and transactional state cannot.

State Recovery source Typical design concern
Source documents and ACLs Authoritative content systems plus protected backups Preserve version, deletion, effective date, and permission history
Vector and search indexes Rebuild recipe, source snapshot, change log; optional snapshot Rebuild time and embedding/model availability may dominate RTO
Release manifests and policies Replicated, signed version-control and artifact stores Near-zero loss; last-known-good must be independently accessible
Sessions, tasks, approvals, idempotency Durable transactional store and replicated log Prevent duplicate effects and reinterpretation after failover
External business effects System of record and reconciliation log Verify what happened; never infer from agent conversation
Audit and security evidence Immutable protected store with independent retention Recovery must preserve integrity and investigation access
Evaluation and telemetry data Replication or rebuild according to business value Do not let observability recovery block critical service unnecessarily
Model capability Pinned provider route or deployable model artifact A model endpoint is a dependency, not a backup of application state

Table 9-2. RTO and RPO values are product-specific. The platform should supply recovery classes; application owners select and fund the appropriate class.

For the Operations Copilot, the policy repository remains authoritative. The vector index is disposable only if its source snapshot, index recipe, embedding model, credentials, and enough compute to meet the RTO are available. Test a full rebuild; a theoretical rebuild that takes three days violates a four-hour RTO.

9.7 Remove correlated failure domains

flowchart TB
    U["Users"] --> DNS["Health-aware traffic control"]
    DNS --> A["Primary region<br/>gateway, harness, tools"]
    DNS -. "failover" .-> B["Recovery region<br/>warm minimum capacity"]
    A --> PA["Primary model route"]
    B --> PB["Evaluated regional/provider route"]
    A --> SA["Transactional state"]
    SA -->|"replication + recovery checks"| SB["Recovery state"]
    V["Versioned artifacts, policy, source snapshots"] --> A
    V --> B
    A --> X["External systems of record"]
    B --> X

Figure 9-3. An illustrative warm-standby design. Exact topology depends on residency, consistency, external dependencies, and recovery objectives.

Multi-provider inference reduces one dependency but does not provide full disaster recovery. Providers may share a cloud, network, identity path, gateway, or upstream model family. The fallback may lack a required model, context length, tool schema, region, quota, or safety behavior. Build a dependency graph and test correlated failures.

AWS’s June 2026 resilience guidance describes incremental patterns spanning cross-region inference, quota isolation, and multi-model gateway routing.4 These patterns trade availability against response time, cost, throughput, and behavioral consistency; the correct choice comes from the application’s SLO and risk tier.

Cross-region or global inference also changes where data is processed. AWS’s 2026 security guidance distinguishes the source region, where stored configuration and logs remain, from destination regions that may process an inference request.5 Apply the same scrutiny to every provider: data residency, encryption, logging, retention, and contractual controls remain in force during failover. An emergency does not silently expand permitted processing geography.

9.8 Recover the Operations Copilot

Assume the primary region becomes unavailable while employees are asking policy questions and several approved cases are being submitted.

  1. Health checks stop new traffic; in-flight mutation states become unknown, not failed or complete.
  2. Traffic shifts to a warm recovery region running the same signed release bundle and an already evaluated model route.
  3. The harness starts read-only until replicated policy, identity, task, and audit state pass freshness and integrity checks.
  4. Policy answers resume if the recovery index meets its freshness and ACL contract. Otherwise the product offers source links or escalation rather than model-memory answers.
  5. Case creation resumes only after the service-desk dependency and idempotency ledger are reachable. Ambiguous submissions are reconciled against the system of record before retry.
  6. Operators communicate the degraded mode, preserve evidence, and later fail back through another controlled change.

Practice this sequence. Game days should include provider throttling, region loss, corrupted index, unavailable identity, expired credentials, control-plane loss, backup restoration, and a compromised release. Measure actual RTO, RPO, data integrity, duplicate effects, degraded-mode behavior, and staff response. Microsoft’s 2025 disaster-recovery guidance similarly treats recovery as a business-priority-driven, measurable plan rather than an infrastructure feature.6

Economics and resilience meet in one decision. Fund the least expensive system that continues to deliver valid outcomes under expected load and declared failures, then prove both claims with production evidence and recovery exercises.

References

  1. OpenAI Engineering, “Beyond rate limits: scaling access to Codex and Sora”, February 13, 2026.↩︎

  2. AWS Machine Learning Blog, “Use Amazon Bedrock Intelligent Prompt Routing for cost and latency benefits”, April 22, 2025.↩︎

  3. AWS Machine Learning Blog, “Managed Tiered KV Cache and Intelligent Routing for Amazon SageMaker HyperPod”, November 26, 2025.↩︎

  4. AWS Machine Learning Blog, “Implementing resilience patterns with Amazon Bedrock and LLM gateway”, June 30, 2026.↩︎

  5. AWS Machine Learning Blog, “Securing Amazon Bedrock cross-Region inference: Geographic and global”, January 13, 2026.↩︎

  6. Microsoft Azure Well-Architected Framework, “Develop a disaster recovery plan for multi-region deployments”, 2025.↩︎