2. Platform Operating Model and Reference Architecture

2.1 Design objective

An AI platform reduces the distance between an evaluated idea and a production service without concealing the controls that make the service trustworthy. It is a product for internal users: application engineers, AI engineers, data owners, security teams, and operators. Its roadmap and service levels matter as much as its infrastructure.

The platform provides golden paths: supported ways to create a project, access models, connect approved data, run evaluations, deploy, observe behavior, and respond to failure. These paths encode organizational defaults while allowing reviewed exceptions. Google described a similar platform-engineering approach in 2025 as “shifting down”: placing recurring security, quality, and operational decisions into the underlying platform rather than transferring their implementation to every development team.1

The desired operating model is centralized foundations with decentralized product ownership. Centralizing everything makes the platform slow and detached from domain needs. Decentralizing everything produces duplicated gateways, inconsistent security, fragmented telemetry, and no credible inventory of deployed AI.

2.2 Logical reference architecture

The reference architecture separates two kinds of responsibility:

  • The control plane records desired state: approved models, policies, project configuration, budgets, artifact versions, evaluation results, and release decisions.
  • The runtime plane—often called the data plane—handles live requests: model inference, retrieval, agent execution, tool calls, enforcement, and telemetry emission.

flowchart TB
    subgraph C["Consumers"]
        DEV["Application and AI teams"]
        OPS["Platform, security, and operations"]
    end

    subgraph CP["Control plane — desired state"]
        PORTAL["Portal, APIs, and catalog"]
        REG["Model, prompt, agent, and tool registry"]
        POL["Identity, policy, tenancy, and budgets"]
        EVAL["Evaluation and release management"]
    end

    subgraph RP["Runtime plane — live execution"]
        GW["AI gateway"]
        HAR["Application harnesses and workflows"]
        RET["Retrieval services"]
        TGW["Tool gateway and approval service"]
        MOD["Managed or self-hosted models"]
    end

    subgraph F["Enterprise foundations"]
        IDP["Identity and secrets"]
        DATA["Authoritative data sources"]
        TEL["Telemetry, audit, and cost systems"]
        CICD["Source control and delivery platform"]
    end

    DEV --> PORTAL
    OPS --> PORTAL
    CP -. "versioned configuration" .-> RP
    DEV --> GW
    GW --> HAR
    HAR --> RET
    HAR --> TGW
    HAR --> MOD
    RET --> DATA
    TGW --> DATA
    IDP --> GW
    CICD --> EVAL
    RP --> TEL

Figure 2-1. A logical AI platform. The control plane governs desired state; the runtime plane executes live work using enterprise identity, data, delivery, and telemetry foundations.

This is a logical decomposition; it does not require a separate product or service per box. A small organization may implement several capabilities in one service. The boundaries matter because they determine ownership, failure isolation, and where policy can be enforced.

Control-plane responsibilities

The control plane provides:

  • A catalog of projects, owners, models, prompts, agents, tools, datasets, and knowledge indexes
  • Policy configuration for model eligibility, data classes, tool permissions, regions, quotas, and approvals
  • Versioned artifacts and environment promotion
  • Evaluation execution, comparison, evidence, and release gates
  • Tenant provisioning, budgets, usage attribution, and exception records
  • Self-service APIs and templates suitable for both people and automation

Control-plane unavailability does not normally stop live inference. Approved configuration is compiled into an immutable, versioned runtime bundle and distributed to the runtime plane. Runtime services retain a last-known-good bundle, reject invalid or expired configuration according to policy, and report configuration age. This avoids turning the management layer into a synchronous dependency for every user request.

Runtime-plane responsibilities

The runtime plane provides:

  • A single authenticated entry point for approved model access
  • Routing, quotas, timeouts, retries, streaming, and provider adaptation
  • Application-owned harnesses for prompts, workflows, memory, and agent loops
  • Permission-aware retrieval and controlled tool execution
  • Input, output, and action policy enforcement
  • Traces, metrics, audit events, and cost records linked to the initiating identity and project

AWS’s 2025 multi-provider reference architecture is one concrete implementation of part of this plane. It centralizes authentication, routing, quotas, provider adaptation, observability, and cost controls while preserving a provider-neutral application interface.2

The gateway is important, but it is not the whole platform. It does not define application quality, curate knowledge, own domain evaluations, or decide whether a proposed tool action is appropriate for a particular business process.

2.3 A request through the platform

The Operations Copilot demonstrates how the planes interact. Its deployed runtime bundle identifies the approved model route, prompt version, retrieval policy, tool allowlist, budget, and evaluation baseline.

sequenceDiagram
    actor U as Employee
    participant A as Copilot
    participant G as AI gateway
    participant R as Retrieval
    participant M as Model
    participant P as Policy and approval
    participant T as Service-desk tool

    U->>A: Ask policy question
    A->>G: Request + user, tenant, trace context
    G->>G: Authenticate, authorize, enforce quota
    G->>R: Retrieve with delegated identity
    R-->>G: Permitted passages + provenance
    G->>M: Versioned prompt + permitted context
    M-->>G: Answer or proposed action
    G-->>A: Answer + citations
    A-->>U: Present grounded response
    U->>A: Approve creation of benefits case
    A->>P: Validate identity, arguments, and approval
    P->>T: Execute narrowly scoped action
    T-->>A: Case identifier + audit result

Figure 2-2. The model can generate an answer or propose an action, but identity, retrieval authorization, approval, and execution remain explicit system responsibilities.

Every step emits the same project, tenant, user, session, trace, artifact-version, and cost context. This shared envelope makes an end-to-end investigation possible. Secrets and unnecessary personal data should not be copied into prompts or telemetry merely because the envelope supports them.

2.4 Ownership model

The platform removes repeated engineering work; it does not remove accountability from application teams.

Owner Accountable for Not delegated to the platform
Platform team Shared APIs, gateway, registries, tenancy, golden paths, platform SLOs and support Whether a domain answer or action is acceptable
Application team Product behavior, prompt and harness, domain evaluations, user experience, product SLOs and runbook Enterprise identity, central audit, or shared-service operation
Data owner Source quality, classification, permissions, freshness, retention and deletion Answer quality after data is combined with a model
Security and risk Baseline controls, threat models, risk tiers, approval policy and exceptions Day-to-day product ownership
Reliability and FinOps Operational standards, capacity guidance, incident practice and cost policy Use-case value and quality thresholds

Each production project needs one accountable application owner and one operational escalation path. Shared ownership without a named decision maker is usually unowned work.

Platform teams operate as product teams: interview users, publish service levels and roadmaps, measure adoption, and remove friction. Useful measures include time to first evaluated deployment, percentage of traffic using approved paths, exception lead time, evaluation-gate adoption, platform reliability, and cost attribution coverage. The 2025 DORA report likewise identified a relationship between high-quality internal platforms and an organization’s ability to realize value from AI.3

2.5 Tenancy and isolation

A tenant is an accountable boundary for policy, cost, and data. An API key alone is insufficient. Depending on risk, the boundary may represent an application, team, business unit, customer, or regulated environment.

At minimum, the platform should isolate:

  • Identities, service accounts, secrets, and delegated credentials
  • Prompts, agents, evaluation datasets, traces, and stored conversations
  • Knowledge indexes and document permissions
  • Quotas, rate limits, budgets, and cost attribution
  • Caches whose entries depend on user or data authorization
  • Deployment environments and release authority

Logical isolation is often sufficient for low-risk internal applications. Separate accounts, projects, networks, encryption keys, or clusters may be justified for regulated data, external customers, or high-impact tools. Choose the boundary from the threat model and recovery requirements rather than the organization chart.

AWS’s 2025 enterprise GenAI portal architecture illustrates this trade-off with a shared gateway and dedicated use-case accounts, allowing centralized access and monitoring while separating quotas, data perimeters, logs, and costs.4

2.6 Golden paths without a golden cage

A new project should be able to request a supported template that provisions:

  1. A registered owner, risk tier, tenant, and budget
  2. Workload identity and gateway access to approved models
  3. A versioned prompt or harness repository
  4. An evaluation suite and initial release gate
  5. A trace namespace, dashboard, alerts, and cost attribution
  6. A deployment pipeline, rollback path, and runbook skeleton

This is the minimum viable platform. Retrieval, tool execution, advanced routing, self-hosted inference, and multi-region operation should be added in response to validated use cases rather than built speculatively.

Golden paths must expose extension points. An application may need a specialized retriever, model, or evaluator; it should be able to integrate one without bypassing identity, audit, release evidence, and cost controls. AWS’s 2025 GenAI gateway guidance describes a useful evolution: co-develop a capability with an initial use case, turn it into a reusable artifact after proof of value, and promote it into a managed platform service only when repeated demand justifies the abstraction.5

2.7 Build, buy, and retain control

Here, “own” means controlling the contract, policy, and operational outcome. It does not require writing every implementation.

Capability Default stance
Foundation-model inference Consume managed services first; self-host for demonstrated regulatory, latency, capability, or economic reasons
AI gateway Own the interface and policies; buy or adopt the proxy implementation where practical
Agent harness Own application behavior; use replaceable SDKs and frameworks
Retrieval Buy storage and search primitives; own corpus semantics, permissions, provenance, freshness, and evaluation
Evaluation Own datasets, criteria, and release thresholds; buy execution and review tooling as useful
Observability Extend the organization’s telemetry stack rather than creating an isolated AI-only system
GPU scheduling and serving Buy by default unless sustained scale and specialist capability justify operation

Provider-specific features are reasonable when they create measurable value. The architecture should preserve portability at deliberate control points: identity, application-facing APIs, artifact records, evaluation evidence, telemetry export, and source data. A multi-provider gateway can support routing and failover—as AWS demonstrated in a 2025 reference architecture—but portability is incomplete if prompts, tools, evaluations, and operational procedures still assume one provider’s behavior.6

The platform therefore optimizes for controlled change. Theoretical vendor neutrality is less useful than keeping application ownership, evidence, and policy boundaries legible as models, providers, and frameworks evolve.

References

  1. Google Cloud Blog, “How Google does it: Your guide to platform engineering”, August 13, 2025.↩︎

  2. AWS Machine Learning Blog, “Streamline AI operations with the Multi-Provider Generative AI Gateway reference architecture”, November 21, 2025.↩︎

  3. Google Cloud Blog, “Announcing the 2025 DORA Report”, September 24, 2025.↩︎

  4. AWS for Industries, “Democratizing GenAI through a Global Enterprise Portal”, May 9, 2025.↩︎

  5. AWS for Industries, “How to Build an Enterprise-Scale GenAI Gateway”, December 15, 2025.↩︎

  6. AWS Machine Learning Blog, “Streamline AI operations with the Multi-Provider Generative AI Gateway reference architecture”, November 21, 2025.↩︎