3. Identity, Data, and Knowledge Foundations

3.1 Trust begins before the prompt

An AI system cannot be more trustworthy than the identities and data supplied to it. Give the model only the context needed for the task, and keep authorization decisions outside it.

The platform must preserve a chain of accountability across four distinct actors:

  • The human or calling service that initiated the request
  • The application workload hosting the AI feature
  • The agent or harness instance selecting models and tools
  • The resource service that owns the data or performs an action

These actors may share infrastructure, but they are not the same security principal. An audit record that says only “the AI called the API” is insufficient. It should identify who initiated the work, which application and agent acted, which authority was delegated, which policy was evaluated, and which resource accepted the operation.

NIST’s National Cybersecurity Center of Excellence framed this as an active standards problem in 2026, emphasizing identification, authorization, auditability, and non-repudiation for software and AI agents.1 The practical starting point, however, is established identity engineering: workload identities, OAuth or equivalent delegation, least privilege, short-lived credentials, and authorization at the resource boundary.

3.2 Identity and delegated authority

An agent does not acquire authority because a model proposed an action. It acts through a registered workload or agent identity and, when appropriate, through authority delegated by a user.

sequenceDiagram
    actor U as User
    participant A as Application / agent
    participant B as Credential broker
    participant R as Resource service

    U->>A: Authenticated request + delegated intent
    A->>B: Token exchange for resource R
    Note over A,B: subject, actor, tenant, action, purpose
    B->>B: Evaluate policy and requested audience
    B-->>A: Short-lived token scoped to R
    A->>R: Operation + audience-bound token
    R->>R: Validate issuer, audience, scope, subject, actor
    R-->>A: Result + decision identifier
    Note over B,R: grants, denials, and effects join the audit trail

Figure 3-1. Authority flows through explicit delegation and is enforced by deterministic services. The model is not part of the credential chain.

Three execution modes cover most applications:

Mode Appropriate use Authority
Workload operation Shared, application-owned resources Narrow service or workload identity
On-behalf-of operation User-specific documents, records, or tools User authority delegated to a named application or agent
Step-up operation Sensitive, high-cost, or consequential action Fresh authorization and, where justified, explicit approval

A dedicated agent identity can improve inventory, revocation, policy targeting, and attribution. Microsoft’s 2026 Entra Agent ID design, for example, represents an agent as a specialized service principal with an accountable sponsor; tokens can distinguish the user subject from the agent actor.2 The pattern is useful regardless of identity product.

Identity alone does not grant permission. Authorization evaluates the subject, application or agent actor, tenant, requested action, resource, purpose, and execution context. Both the broker and receiving service validate the token audience and scope.

3.3 Authentication and credential boundaries

The platform follows these rules:

  1. Authenticate users and workloads at the platform boundary, then propagate verified identity context end to end.
  2. Authorize again at each data source and tool. Gateway authorization does not replace resource authorization.
  3. Use short-lived, narrowly scoped, audience-restricted credentials. Prefer workload federation or token exchange over static keys.
  4. Keep credentials outside prompts, model-visible memory, tool output, and traces. The harness passes them through a protected side channel.
  5. Separate development, test, and production identities. A test agent must not inherit production access.
  6. Record grants, denials, delegation, approval, token issuer, and actor in the audit trail without recording the secret itself.

These recommendations align with the IETF’s 2025 OAuth security best current practice, which calls for minimum necessary privileges, audience restriction, and sender-constrained tokens where feasible.3 The 2025 MCP authorization specification similarly requires resource indicators so a token requested for one MCP server is not silently reused against another.4

Human approval adds control but cannot replace authorization. Repeated low-information prompts create approval fatigue. Anthropic reported in 2026 that users approved roughly 93% of Claude Code permission prompts; Claude Code auto mode was its direct response, automating safer approvals inside stronger containment.5 Ask for approval at meaningful commitment boundaries, and make the proposed action, target, arguments, and consequence understandable.

3.4 The knowledge index is a derived replica

A production knowledge system has two planes:

  • The ingestion plane converts authoritative content into searchable representations.
  • The query plane retrieves permitted evidence for a particular request.

flowchart LR
    subgraph I["Ingestion plane"]
        S["Authoritative sources"] --> D["Discover and classify"]
        D --> X["Extract and normalize"]
        X --> C["Chunk with provenance and ACL metadata"]
        C --> E["Embed and index"]
        E --> V["Validate and publish index version"]
    end

    subgraph Q["Query plane"]
        ID["Verified identity and tenant"] --> AZ["Resolve permitted scope"]
        AZ --> RET["Filtered retrieval and ranking"]
        RET --> RE["Authoritative access re-check"]
        RE --> CTX["Permitted context with citations"]
        CTX --> M["Model generation"]
    end

    V --> RET
    S -. "permissions, versions, deletions" .-> RE

Figure 3-2. The index accelerates discovery, but authoritative sources remain responsible for access and lifecycle state.

The platform should treat extracted text, chunks, embeddings, summaries, and caches as derived data. They inherit the source’s classification, residency, retention, and deletion requirements. Copying content into a vector database does not lower its sensitivity.

Every indexed unit needs a metadata contract. At minimum it should carry:

Field Purpose
tenant_id and security domain Prevent cross-boundary retrieval
Stable source and document identifiers Reconcile with the authoritative object
Source version and content hash Detect changes and reproduce results
Effective and expiry times Exclude drafts, superseded, or future policies
Classification and access-control reference Filter and re-authorize candidates
Parser, chunker, and embedding versions Explain retrieval changes
Index version and ingestion time Measure freshness and support rollback

Missing mandatory security or provenance metadata fails ingestion. Silently indexing an unclassified document is not a reasonable fallback.

3.5 Permission-aware retrieval

Authorization must occur before content enters the model context. Once the model sees a restricted passage, output filtering cannot reliably undo the disclosure.

A robust retrieval path uses two complementary layers:

  1. Retrieval filtering narrows search to the tenant and scopes the caller is likely permitted to access. This protects relevance and avoids ranking inaccessible documents above accessible ones.
  2. Authoritative revalidation checks candidate document or chunk identifiers against the current source or policy service before returning their content to the harness.

The first layer is fast but may lag permission changes. The second closes that consistency gap. AWS’s 2025 RAG authorization guidance makes the same distinction: synchronized vector metadata is useful for filtering, but the source of truth should validate access before retrieved content is added to the prompt.6

Fail closed when the authorization decision cannot be made. Do not reinterpret an authorization outage as “no filters.” For hard customer or regulatory boundaries, use separate indexes, accounts, projects, or encryption domains. Within one tenant, policy-driven metadata filtering may be sufficient when combined with independent API and document-level checks. A 2026 AWS reference architecture demonstrates this two-layer, deny-by-default pattern and explicitly distinguishes it from hard cross-tenant isolation.7

3.6 Freshness, provenance, and deletion

Retrieval quality depends on lifecycle correctness as much as semantic similarity.

The ingestion system consumes creates, updates, permission changes, and deletions; operates idempotently; quarantines extraction failures; and exposes lag by source and security domain. Publishing is atomic: queries see either the prior complete index version or the new one. A partially rebuilt mixture is never served.

Deletion must propagate through raw staging data, extracted text, chunks, embeddings, caches, stored conversations, evaluation samples, and backups according to policy. Maintain tombstones long enough to prevent a delayed job from resurrecting deleted content. Permission revocation usually requires faster propagation than ordinary content refresh, so it should have its own service objective and alert.

Every answer retains enough provenance to identify the source object, source version, retrieved passages, retrieval configuration, and index version. Citations present part of that evidence to users; the full provenance record supports investigation and reproduction.

3.7 Operations Copilot: applying the design

For the parental-leave question, the copilot receives the employee identity, tenant, country, and group claims through a trusted request envelope. The retrieval service searches only the employee’s permitted scope, excludes policies outside their effective dates, and revalidates the selected document before sending passages to the model.

Opening a benefits case uses a separate path. The model produces proposed structured arguments but never receives a service-desk credential. After the employee reviews the destination and submitted fields, the policy broker issues a short-lived token limited to creating one case in the benefits queue. The tool validates the token and records both the employee subject and copilot actor.

Failure Required behavior
Document has no access metadata Quarantine; do not index
Group membership changed but index metadata is stale Authoritative re-check denies access
Old and current policies both rank highly Effective-date and version rules exclude the old policy
Authorization service is unavailable Fail closed or use a narrowly defined safe degraded mode
Model requests a broader tool scope Deny; the model cannot expand its authority
User revokes approval before execution Discard the capability and require a new authorization
Document is deleted Tombstone it and purge all derived representations

These are identity and knowledge boundary failures. Their required behaviors are invariants of authorization and data lifecycle. Chapter 4 defines execution-level failures inside the harness; Chapter 6 defines user-visible degraded modes when shared services are unhealthy.

My default at identity and retrieval boundaries is to fail closed. A team proposing a degraded mode must be able to name the evidence it will withhold, the authority it will preserve, and the precise condition that ends the exception.

References

  1. NIST CSRC, “Accelerating the Adoption of Software and Artificial Intelligence Agent Identity and Authorization”, initial public draft, February 5, 2026.↩︎

  2. Microsoft Learn, “Overview of agent identities in Microsoft Entra”, accessed August 3, 2026.↩︎

  3. IETF, RFC 9700: Best Current Practice for OAuth 2.0 Security, January 2025.↩︎

  4. Model Context Protocol, “Authorization”, November 25, 2025 specification.↩︎

  5. Anthropic Engineering, “How we contain Claude across products”, May 25, 2026.↩︎

  6. AWS Security Blog, “Authorizing access to data with RAG implementations”, September 18, 2025.↩︎

  7. AWS Architecture Blog, “Secure multi-tenant RAG with Amazon Bedrock and Verified Permissions”, June 22, 2026.↩︎