flowchart LR
S["Untrusted sources<br/>users, documents, web, tool results"] --> C["Context builder"]
C --> M["Model<br/>proposes text or actions"]
M --> V["Schema and safety validation"]
V --> B["Policy and action broker"]
I["Identity, consent, risk policy"] --> B
B -->|"permitted read"| R["Scoped retrieval"]
B -->|"approved capability"| T["Sandboxed tool or external system"]
B -->|"deny / clarify"| X["Controlled response"]
R --> C
T --> E["Verified external state"]
B -.-> A["Audit evidence"]
T -.-> A
7. Security, Safety, and Governance
7.1 Design for a model that can be wrong or manipulated
An AI platform joins probabilistic models to deterministic systems. Its security model must assume that a model can misunderstand intent, hallucinate, follow hostile instructions, or choose an unsafe tool call. Model alignment and content filters reduce likelihood; they do not define the maximum damage.
Three disciplines meet at this boundary:
- Security protects confidentiality, integrity, availability, identity, and system control against mistakes and adversaries.
- Safety limits harmful behavior even when no attacker is present.
- Governance establishes which uses are permitted, who accepts risk, what evidence is required, and when a system must be changed or stopped.
The engineering objective is to let the model propose useful work inside deterministic limits. The model itself remains an untrusted security principal. Anthropic’s 2026 containment account makes the distinction concrete: model-level defenses shape behavior probabilistically, while sandboxes, virtual machines, filesystem boundaries, and egress controls cap what an agent can reach.1
7.2 Threat-model the compound system
The attack surface includes more than the model endpoint. Threat-model identities, prompts, retrieval corpora, embeddings, indexes, memory, model and tool supply chains, orchestration state, code execution, network egress, logs, evaluation data, and human approval flows. Include malicious users, compromised content or tools, careless operators, model misbehavior, insiders, and dependency compromise.
Figure 7-1. Content can influence a proposal, but only the policy and action broker grants capability. Authorization state must not be inferred from model text.
Use a source-to-sink analysis. A source is anything that can influence the model: a user message, retrieved document, email, webpage, image, memory entry, or tool response. A sink is a capability that can create harm: disclose data, write a record, send a message, execute code, spend money, or alter access. Security controls should prevent an untrusted source from silently driving a consequential sink.
| Threat | Operations Copilot example | Primary containment |
|---|---|---|
| Prompt injection | A policy document says to ignore rules and export employee data | Treat content as data; restrict sinks and inspect transfers |
| Broken authorization | Retrieval returns another subsidiary’s HR policy | Filter with authoritative identity and ACLs before model context |
| Sensitive disclosure | A response or URL contains private case details | Data minimization, output/egress checks, redaction, destination policy |
| Excessive agency | The agent closes or deletes cases although it only needs creation | Narrow tools, permissions, autonomy, and budgets |
| Insecure output handling | Model-generated markup, query, or arguments reach an interpreter | Typed schemas, allowlists, escaping, parameterization, sandboxing |
| Supply-chain compromise | A remote connector changes behavior after approval | Inventory, pinning where possible, isolation, continuous review |
| Resource abuse | An attacker triggers long contexts and recursive tool loops | Quotas, step/token limits, admission control, anomaly detection |
| Memory poisoning | Malicious instructions persist into later sessions | Scoped provenance, write controls, expiry, review, reset capability |
Table 7-1. The OWASP 2025 list gives useful names to prompt injection, sensitive disclosure, supply-chain risks, improper output handling, and excessive agency; a platform threat model must connect those categories to actual assets and effects.2
7.4 Give agents capabilities, not ambient credentials
The harness never places long-lived secrets in prompts, model-visible memory, or a general execution environment. A tool broker stores credentials server-side and issues narrow capabilities bound to tenant, actor, purpose, resource, action, environment, and expiry. Prefer user-delegated identity where the action carries the user’s authority; use workload identity where the platform itself is accountable. Record both.
Separate read and write tools. Avoid a generic HTTP client, database console, or shell when a constrained business operation will suffice. For example, create_case(category, summary, approved_by) is safer than call_service_desk_api(method, path, body). Apply rate and value limits at the destination, not only in the prompt.
Approval is not a vague chat utterance. It is a structured, expiring authorization over a visible action: exact target, important arguments, expected effect, data leaving the boundary, and whether the operation is reversible. Revalidate identity, policy, and state at execution time. Never allow the model to approve its own action. Repeated low-value prompts create approval fatigue; reserve interruption for decisions a user can understand and meaningfully control.
OpenAI’s 2026 account of internal Codex deployment illustrates the combined pattern: technical sandbox boundaries, managed network policy, explicit approval policy, credential controls, and agent-aware telemetry.5
7.5 Contain computation and egress
Any model-generated code, command, document conversion, or browser automation runs in an isolated environment with a minimal filesystem, resource quotas, no host credentials, and denied network access by default. Isolation must be enforced by the operating system or virtualization layer and propagate to child processes. OpenAI’s Windows sandbox design, for example, composes restricted process identities, filesystem permissions, credential separation, and firewall enforcement rather than relying on the agent to obey a policy statement.6
Network allowlists grant capabilities; they do not certify a company or every path under its domain. An approved domain may support redirects, uploads, arbitrary accounts, or attacker-controlled paths. Control method, destination, account, data classification, redirect behavior, and request shape; proxy traffic where provenance must be enforced. OpenAI’s 2026 link-safety design demonstrates one narrow control: automatic retrieval is limited to exact URLs independently observed as public, while unverified URLs require intervention.7
A poisoned-policy example
Suppose an authorized Operations Copilot user asks, “What is our parental-leave policy, and open a case for clarification?” The current HR document contains hidden text instructing the model to append the employee’s profile to a tracking URL.
The document is allowed into context because the user may read it; authorization alone cannot detect poisoned content. The model may even follow the instruction. The blast radius is nevertheless contained:
- The model has no general network or email tool.
- The action broker accepts only a typed service-desk case proposal.
- Data-loss policy rejects unnecessary personal fields and external destinations.
- The user approves the exact case summary and queue, not an opaque command.
- A short-lived delegated credential creates one case; idempotency and reconciliation verify it.
- The trace retains source provenance, the proposed and executed action, policy decisions, and final case identifier.
The answer may still be behaviorally wrong, so injection evaluations and review remain necessary. The architecture prevents that wrong answer from silently becoming unrestricted authority.
sequenceDiagram
actor U as Employee
participant H as Harness
participant D as Retrieved policy
participant M as Model
participant B as Action broker / DLP
participant S as Service desk
U->>H: Ask policy question and request a case
H->>D: Retrieve authorized policy
D-->>H: Policy + hidden exfiltration instruction
H->>M: Labeled untrusted evidence
M-->>H: Case proposal containing external URL + profile data
H->>B: Typed proposed action
B--xH: Reject destination and unnecessary fields
H->>M: Regenerate within allowed schema
M-->>H: Benefits-case proposal
H->>U: Show exact queue and summary for approval
U->>H: Approve
H->>B: Approved intent + delegated identity
B->>S: Create one idempotent case
S-->>H: Verified case identifier
Figure 7-2. Authorization permits the document to be read; it does not make the document trustworthy. The poisoned instruction dies at the typed action and egress boundary.
7.6 Protect data across its lifecycle
Classify data before ingestion and enforce classification through retrieval, prompt assembly, caching, provider routing, output, telemetry, and deletion. Minimize what is sent to a model; redact or tokenize unnecessary identifiers; encrypt in transit and at rest; isolate tenants; and test deletion across indexes, caches, memory, logs, evaluation stores, and backups.
Provider approval must cover processing region, retention, training use, abuse monitoring, subcontractors, incident notification, model changes, and deletion. Do not log raw content merely because it is useful for debugging. Separate restricted audit evidence from general observability and control who can rehydrate content.
Treat prompts, models, datasets, tools, policies, connectors, and evaluation graders as supply-chain artifacts. Maintain owners, versions, provenance, integrity checks, vulnerability status, permitted data classes, and a revocation path. A remote tool can change after onboarding, while a trusted connector can still return an attacker-controlled document; trust the code, operator, and content as separate decisions.
7.7 Govern by consequence
Governance should make safe delivery faster by predefining evidence and decision rights for each risk tier.
| Tier | Typical capability | Minimum governance |
|---|---|---|
| 1 — Assistive | Public-data drafting; no external effect | Owner, inventory, baseline safety and quality evaluation |
| 2 — Internal advisory | Permission-aware answers from company data | Data review, threat model, access tests, monitoring, user disclosure |
| 3 — Transactional | Reversible writes such as creating a service case | Independent security review, explicit approval, reconciliation, rollback and incident plan |
| 4 — Consequential | Decisions affecting employment, finance, health, safety, or legal rights | Human decision authority, impact assessment, specialist review, stringent evidence—or prohibition |
Table 7-2. Risk depends on affected people, data, autonomy, reversibility, scale, and detectability. Model size alone is a poor proxy. Jurisdiction- and sector-specific obligations must be added separately.
These handbook tiers are an internal engineering scheme, not the legal categories in the EU AI Act. Organizations in scope must map each use case separately to the Act and applicable sector law; as of July 2026, the EU’s amended timeline applies the general Act from August 2, 2026 and most high-risk-system rules from December 2, 2027.8 This is a planning signal, not legal advice.
Each registered use case should name the accountable business owner, technical owner, intended users, prohibited uses, data and model inventory, risk tier, evaluation evidence, residual risks, approvers, monitoring plan, incident owner, review date, and retirement procedure. Material changes to model, prompt, retrieval sources, tools, permissions, autonomy, user population, or data class trigger reassessment. NIST’s Generative AI Profile likewise recommends adapting organizational risk tiers and applying additional review, tracking, and oversight where warranted.9
7.8 Continuously prove the controls
Security assurance combines conventional application testing with AI-specific adversarial work:
- Test tenant isolation, authorization, secret handling, output encoding, dependencies, and infrastructure as usual.
- Red-team direct and indirect injection across documents, images, web content, connectors, tool results, and memory.
- Verify that every high-impact tool fails safely with malformed arguments, stale approval, confused identity, replay, timeout, and partial completion.
- Measure control performance by attack class and slice, including adaptive repeated attempts—not only a single benchmark pass.
- Exercise credential revocation, connector removal, model rollback, memory purge, audit-pipeline failure, and the platform kill switch.
Security telemetry correlates user intent, content provenance, model route, tool proposal, policy decision, approval, egress, and verified effect without exposing unnecessary sensitive content. Alert on boundary violations and suspicious sequences rather than every unusual sentence. Preserve evidence for investigation and feed confirmed attacks into threat models and regression suites.
The security contract is simple to state: untrusted content may influence reasoning, but it cannot grant authority; the model may propose actions, but deterministic systems bound, authorize, execute, and record their effects.
References
Anthropic Engineering, “How we contain Claude across products”, May 25, 2026.↩︎
OWASP GenAI Security Project, “OWASP Top 10 for LLM Applications 2025”, 2025.↩︎
Anthropic Research, “Mitigating the risk of prompt injections in browser use”, November 24, 2025.↩︎
OpenAI Security, “Designing AI agents to resist prompt injection”, March 11, 2026.↩︎
OpenAI, “Running Codex safely at OpenAI”, May 8, 2026.↩︎
OpenAI Engineering, “Building a safe, effective sandbox to enable Codex on Windows”, May 13, 2026.↩︎
OpenAI, “Keeping your data safe when an AI agent clicks a link”, January 28, 2026.↩︎
European Commission, “AI Omnibus enters into force”, July 27, 2026.↩︎
NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”, July 2024.↩︎