8. Delivery and Lifecycle Management

8.1 Release a system, not a model name

An AI application’s behavior is produced by a compound configuration: code, model route, prompts, retrieval corpus and index, tool schemas, policies, memory rules, evaluators, and runtime limits. Changing any one of them can alter quality, safety, latency, or cost. A ticket that says “upgrade the model” does not identify a reproducible release.

The deployable unit should be an immutable AI release bundle. Its manifest records exact artifact versions, compatibility constraints, evaluation evidence, approvals, owner, creation time, and rollback target. The runtime resolves a release identifier to this signed manifest rather than independently choosing “latest” components.

apiVersion: ai.platform/v1
kind: ReleaseBundle
metadata:
  name: ops-copilot-27
  owner: people-platform
spec:
  code: { image: "registry/copilot@sha256:9bd…" }
  modelRoute: { policy: grounded-answer-v12, fallback: evaluated-route-v8 }
  prompt: { digest: "sha256:412…", outputSchema: case-proposal-v3 }
  retrieval: { index: kenya-policy-2026-07-28, embedding: embed-v5 }
  tools: { catalog: benefits-readwrite-v4, mcpRegistry: registry-2026-07 }
  policy: { bundle: employee-assist-v11 }
  evaluation: { suite: ops-regression-42, report: eval-2026-08-01-17 }
  limits: { maxSteps: 8, timeoutSeconds: 20, budgetUsd: 0.30 }
  rollback: ops-copilot-26

Figure 8-1. An illustrative release manifest. Large artifacts live elsewhere, but the bundle pins immutable identifiers, evaluated fallback behavior, limits, evidence, and rollback target.

Artifact Version by Rollback or compatibility concern
Code and infrastructure Commit and image digest Database and state migrations must remain compatible
Model Provider, model snapshot, route policy, parameters Provider aliases may move; old snapshot may be retired
Prompt and schemas Template digest and schema version Tool/output fields must match harness parsers
Retrieval Source snapshot, chunker, embedding model, index build Old and new indexes may need parallel service
Tools and policies Contract and policy bundle version External API and authorization semantics may change
Evaluations Dataset, grader, rubric, and runner version Scores from different graders are not automatically comparable

Table 8-1. Lineage must cover all behavior-producing artifacts. AWS’s 2025 account of generative-AI asset tracking similarly connects datasets, evaluators, models, and deployments so a production result can be traced and reproduced.1

8.2 Keep the desired state in version control

Prompts, routing, tool definitions, policy rules, index recipes, infrastructure, dashboards, and evaluation thresholds belong in reviewable, versioned repositories. Administrative consoles are useful views, but production changes should flow through a controlled change path with identity, diff, review, validation, and deployment evidence. Prevent silent prompt editing or provider rerouting in production.

The repository should also contain the decisions an engineer or agent needs to change the system safely: architecture rules, interfaces, ownership, runbooks, migration procedures, and Architecture Decision Records. OpenAI’s 2026 harness-engineering report describes repository-local, versioned artifacts as the system of record and emphasizes executable architectural invariants rather than documentation alone.2

An emergency override may be necessary, but it should be narrow, expiring, attributable, and reconciled back into desired state. Otherwise configuration drift turns the next deployment into an incident.

8.3 Separate environments and promote evidence

Use isolated development, test, staging, and production environments with separate identities, secrets, budgets, indexes, queues, and tool destinations. Production credentials must never be available in development. Test tools should act on resettable fixtures; staging mutations should target a sandbox service desk or a synthetic tenant.

Do not copy unrestricted production conversations into lower environments. Create privacy-reviewed, representative datasets with stable identifiers and controlled access. When a production sample is necessary, minimize and redact it, record its permitted use, and delete it on schedule.

Build once and promote the same immutable bundle. Environment-specific values—endpoints, secret references, quotas—are supplied by declared configuration, not by rebuilding the artifact. Promotion carries forward evaluation reports, security evidence, schema checks, and approvals; it does not depend on someone recreating a successful notebook experiment.

AWS’s December 2025 GenAIOps guidance explicitly includes versioning prompts, RAG data, and evaluation data across a development-to-production lifecycle.3 The broader lesson is that reproducibility requires capturing every input to behavior. Packaging application code is only one part of the release.

8.4 Automate the release path

flowchart LR
    CH["Reviewed change"] --> BUILD["Build and sign bundle"]
    BUILD --> TEST["Contracts, evals, security, performance"]
    TEST --> STAGE["Staging and migration rehearsal"]
    STAGE --> SHADOW["Shadow or replay"]
    SHADOW --> CANARY["Sticky canary"]
    CANARY --> GATE{"SLO and quality gates"}
    GATE -->|"pass"| RAMP["Progressive ramp"]
    GATE -->|"fail"| ROLL["Rollback or disable"]
    RAMP --> OBS["Post-release observation"]
    OBS --> REG["Evidence and release registry"]

Figure 8-2. Deployment is a controlled experiment. Each gate has predeclared success, stop, and rollback criteria.

The pipeline should run deterministic contracts first, then the smallest evaluation set capable of rejecting the change quickly, followed by broader regression, safety, and performance suites. Test the assembled bundle; testing a prompt against one model and deploying it through another route is not release evidence.

Change Required focused evidence Typical rollout
Prompt or context template Format, grounding, injection, and task regressions Shadow, then canary
Model or routing policy Full quality/safety suite, tool compatibility, latency and cost Replay, sticky canary, gradual ramp
Retrieval/index pipeline ACL isolation, recall, freshness, citation and backfill checks Dual index and query shadowing
Tool or agent trajectory Contract, authorization, approval, idempotency, timeout and effect tests Synthetic environment; canary writes with tight limits
Policy/guardrail False-pass and false-block rates by risk slice Shadow decisions before enforcement where safe
Infrastructure only Contracts, load, resilience, and behavioral smoke suite Ordinary canary plus AI outcome monitoring

Table 8-2. “No prompt change” does not justify skipping behavioral tests: infrastructure, routing, and compiler changes can alter model outputs.

8.5 Use shadowing, canaries, and controlled exposure

Offline evaluations are the first gate. Production simulation adds traffic shape, live dependencies, and realistic context that an offline suite cannot reproduce. Replay privacy-approved historical traffic or shadow live requests to a candidate without returning its result. Shadow tools must be read-only or simulated; never duplicate a real-world action. Capture differences in answer, citations, route, tokens, latency, tool plan, and policy decision.

OpenAI’s 2026 deployment-simulation work provides a recent example of replaying privacy-preserved production contexts against candidate models, including agentic trajectories with simulated tools, to estimate behavior before release.4 This complements curated evaluations because realistic context exposes different failure modes.

Canaries then test real traffic and dependencies. Assign consistently by tenant, user, or conversation so one session does not alternate behavior. Start with internal users or low-risk slices, but deliberately include representative languages, long contexts, providers, and tool paths before broad release. Compare the candidate with the production baseline using predeclared SLO, quality, safety, and cost thresholds. A rollout controller should pause or revert automatically on high-confidence operational failures; ambiguous behavioral changes require human review.

Deployment and release are separate decisions. A bundle can be present in production but disabled behind a feature flag. Flags need owners, expiry dates, audit history, and cleanup; a permanent maze of flags makes combinations untestable.

8.6 Migrate knowledge and state deliberately

Retrieval releases resemble database migrations more than static file deployments. A new chunker or embedding model requires a new index generation rather than an in-place mutation of the serving index. Build it from a recorded source snapshot, validate counts and ACL coverage, measure retrieval quality, and shadow queries against both generations. Switch a logical alias only after validation; keep the prior index available for a bounded rollback window.

flowchart LR
    S["Record source snapshot"] --> B["Build index generation N+1"]
    B --> V{"Counts, ACLs, deletion,<br/>and retrieval gates pass?"}
    V -->|"no"| X["Quarantine generation"]
    V -->|"yes"| Q["Shadow queries against N and N+1"]
    Q --> G{"Quality and latency<br/>gates pass?"}
    G -->|"no"| X
    G -->|"yes"| A["Atomically switch alias to N+1"]
    A --> W["Bounded rollback window<br/>N remains read-only"]
    W -->|"regression"| R["Revert alias to N"]
    W -->|"stable"| D["Retire N after retention checks"]

Figure 8-3. An index migration is a reversible release with explicit validation, shadowing, alias switch, and retirement stages.

During a long backfill, define whether updates are dual-written, replayed from a change log, or frozen. Prove that deletions, permission changes, legal holds, and document supersession propagate to both indexes. A semantically better index that serves stale permissions is an invalid release.

Agent state also needs schema evolution. Long-running tasks may have been created under an older prompt, tool contract, or policy. Record the release with each task and choose explicitly whether it completes on the original compatible runtime, migrates through a tested state transformer, or stops for human recovery. Never allow a newly deployed harness to reinterpret old approval state implicitly.

8.7 Make rollback a designed capability

Keep a tested last-known-good bundle and a one-step way to shift traffic back. Rollback includes prompts, routes, policies, tool contracts, and index aliases as well as the application image. Use backward-compatible state changes: expand schemas before use, deploy readers that understand old and new forms, migrate, then remove the old form in a later release.

A provider may retire a model, making literal rollback impossible. Maintain an evaluated replacement path and begin migration before the deadline. AWS’s 2026 model-agility guidance describes migrations as a repeatable process of ground-truth collection, evaluation, and validation rather than a one-off model swap.5 Pin model snapshots where the provider permits, monitor deprecation notices, and record which releases depend on each model, region, or API version.

For an unsafe release, favor containment over perfect diagnosis: stop the ramp, disable consequential tools, revert the affected bundle or route, preserve evidence, and reconcile external effects. Recovery is incomplete until in-flight tasks and writes are accounted for.

8.8 Walk through an Operations Copilot release

Assume release ops-copilot-27 changes the model route, revises the case-creation prompt, and rebuilds the policy index with a new embedding model.

  1. The manifest pins code, two model routes, prompt and tool-schema digests, index generation, policy bundle, evaluation suite, and rollback v26.
  2. CI rejects incompatible tool fields, then runs retrieval ACL tests, grounded-answer regressions, prompt-injection cases, case-effect tests, and latency/cost benchmarks.
  3. Staging rebuilds the index from a recorded source snapshot and creates cases only in a sandbox queue.
  4. A production replay compares v27 with v26; tool calls are simulated.
  5. A sticky employee canary receives answers, while case creation begins with a stricter approval limit and idempotency checks.
  6. The release ramps only if valid availability, policy retrieval, citation support, unauthorized-context rate, case correctness, latency, and cost satisfy their gates.
  7. On regression, traffic returns to v26, the index alias reverts, new case creation is disabled if necessary, and created cases are reconciled.

OpenAI’s 2026 description of Presence summarizes a similar operating principle: proposed agent changes are tested against the production version and then approved for controlled rollout.6 The important artifact is not the vendor workflow; it is the closed loop from observed need to versioned change, comparative evidence, controlled exposure, and accountable promotion.

8.9 Own the entire lifecycle

Every production use case needs an owner for behavior, operations, data, and risk. The release registry records what is active, where, for whom, why it was approved, and when it must be reviewed. Automatically identify unsupported models, expired exceptions, stale indexes, unused flags, orphaned tools, and releases without current evaluation evidence.

Retirement is a normal lifecycle stage. Stop new tasks, drain or transfer in-flight work, revoke credentials and connectors, archive required evidence, delete data according to policy, remove routes and flags, notify users, and verify that cost and telemetry cease. A platform that can launch an AI application but cannot reproduce, roll back, migrate, and retire it is a prototype factory—not a production delivery system.

References

  1. AWS Machine Learning Blog, “Tracking and managing assets used in AI development with Amazon SageMaker AI”, December 17, 2025.↩︎

  2. OpenAI Engineering, “Harness engineering: leveraging Codex in an agent-first world”, February 11, 2026.↩︎

  3. AWS Machine Learning Blog, “Operationalize generative AI workloads and scale to hundreds of use cases with Amazon Bedrock – Part 1: GenAIOps”, December 15, 2025.↩︎

  4. OpenAI Research, “Predicting model behavior before release by simulating deployment”, June 16, 2026.↩︎

  5. AWS Machine Learning Blog, “AWS Generative AI Model Agility Solution: A comprehensive guide to migrating LLMs for generative AI production”, April 30, 2026.↩︎

  6. OpenAI, “Introducing OpenAI Presence”, July 22, 2026.↩︎