AI Security

Watch a normal AI assistant get targeted by a prompt injection attempt, an unauthorized data access attempt, and an automated abuse attack — then see how authentication, authorization, monitoring, and filtering defend against each one.

What are the main security risks in LLM applications? Prompt injection, data leakage across users or tenants, excessive tool privileges, model-endpoint abuse and supply-chain compromise. An AI application is attacked through the same doors as any web service (stolen credentials, over-broad permissions, leaky logs, vulnerable dependencies) plus one door that is new: the model reads instructions and data in the same stream of tokens, so any text that reaches the prompt can try to give it orders. That text may come from the user (direct prompt injection) or from a retrieved document, web page or tool result (indirect prompt injection). Layered controls protect the system by assuming the model will sometimes follow a hostile instruction and making sure that doing so is not enough to cause harm: identity and authorization checked in ordinary code, retrieval filtered by the caller's permissions, tools executed under the user's scoped credentials after a deterministic policy check, per-tenant budgets on the endpoint, verified supply chains, redacted logs, and tests that try to break all of it before release.

The rule that organises the rest of this chapter is short: security is enforced in deterministic code outside the model, never by the prompt. A system prompt that says "never reveal other customers' data" is a request, not a control. It lowers the probability of a bad outcome; it cannot bound it. Everything below is about where to put the controls that can.

This lesson is about threats, trust boundaries, controls and testing. Observing a running system is covered in AI Observability; the economics of token spend in Cost Optimization; and the agent-side view of approvals in Human-in-the-Loop and Guardrails.

Take the enterprise support assistant that runs through this track. A customer of tenant acme asks: "Can I return an opened item after 30 days?" The retriever searches the policy corpus, and one of the top five chunks comes from a page that a vendor-managed wiki synchronises into the index. Most of the page is a normal returns policy. Buried in a footer, in white-on-white text that no human reviewer ever sees, is a sentence addressed to an AI system: it tells the assistant to look up the customer's most recent orders and include the account email and the last four card digits in its answer.

Nothing in this chain was exotic. The ingestion job did its job; the index returned the chunk because the footer shares vocabulary with the question; the prompt builder concatenated the chunk into the context exactly as designed (see Context Injection). The model then received one long string in which the system's rules, the customer's question and the attacker's sentence are all just tokens. There is no type system at that level. Whether the model complies depends on training, phrasing and chance, and it can change when the provider updates the model.

The attack surface therefore includes every place text can enter the context: the chat box, uploaded files, retrieved chunks, web pages fetched by a browsing tool, the JSON returned by an order-lookup tool, email bodies the assistant summarises, and even tool descriptions supplied by a third-party server. The defensive stance is to treat all of it as data that may contain hostile instructions, and to make sure that instructions found in data have no authority, because authority comes only from the authenticated principal and from server configuration.

Read the figure from the bottom up in terms of reliability. C4 and C5 are ordinary code with ordinary tests, and they bound the damage even if everything upstream fails. C1 to C3 reduce how often hostile text reaches the model and how persuasive it is, but they are probabilistic or heuristic. A design that relies only on the upstream controls is a design that works until the first clever phrasing.

A threat model is a written answer to four questions: what are we protecting, from whom, through which entry points, and what happens if they succeed. For the support assistant the assets are not "the model". They are customer personal data (names, emails, order history, partial payment details), tenant-private policy documents, credentials and API keys the runtime holds, the business actions the tools can perform (look up an order, and later issue a refund), the model endpoint itself as a metered and capacity-limited resource, and the audit trail that lets you prove what happened.

The attackers are mostly ordinary. A curious or malicious end user of tenant A who wants tenant B's documents. An external party who can write to something you ingest (a public web page, a shared wiki, an inbound email, a product review). A compromised or careless internal account. An automated client that scrapes responses or burns your token budget. A malicious or compromised third-party package, model artifact or tool server. And non-adversarial failure: an engineer who writes a cache key without a tenant, which causes exactly the same breach as an attacker and is far more common.

Entry points follow the data flow: the HTTP edge, the chat input, uploaded files, the ingestion pipeline, retrieved chunks, tool arguments produced by the model, tool results, the model provider connection, logs and dashboards, the CI/CD and artifact path that ships prompts, models and indices, and the administrative consoles. Traditional threats (broken authentication, injection into SQL or shell, SSRF from a fetch tool, exposed inference ports on a self-hosted model server, secrets in environment dumps) stay in scope and are usually the cheapest route for an attacker. The AI-specific additions are untrusted text steering model behaviour, a model whose outputs cannot be trusted as authorisation decisions, a corpus that can be poisoned, and metered inference that can be turned into a bill.

Two published references are worth using as checklists once you have your own model: the OWASP Top 10 for LLM applications, and the NIST AI Risk Management Framework for the governance view. This lesson does not reproduce their item lists or numbering; consult the current versions directly, since they are revised.

Hosted versus self-hosted changes who owns which risk

With an API-hosted model, you do not operate the serving stack, but you do send prompts, which contain retrieved documents and possibly personal data, to another organisation. The questions become contractual and configuration-level: which data is retained and for how long, whether it may be used for training, which region it is processed in, how the provider handles abuse monitoring, and what the API key can do if leaked. With a self-hosted model (see Model Serving) the data stays in your network, but you now own the endpoint's authentication, network exposure, artifact provenance, GPU-node patching and multi-tenant isolation between requests sharing a server. Neither choice removes prompt injection; the model reads untrusted text in both.

The label on the model box is deliberate. A model can help you detect suspicious content, but it cannot be a boundary: it has no authenticated identity for the user, no access to your authorisation data unless you place it in the prompt, and its behaviour is statistical. Every other box in the figure can be given a deterministic rule and a test.

asset                    threat                         entry point              enforcing control
-----------------------  -----------------------------  -----------------------  ----------------------------
customer PII             cross-tenant disclosure        retrieval, cache, logs   ACL filter, scoped cache key
order-lookup tool        call for someone else's order  model tool proposal      gateway ownership check
refund tool (later)      unauthorised / repeated refund model proposal, retries   approval + idempotency key
provider API key         leak, abuse                    env dump, repo, logs     secret manager, per-env keys
model endpoint           cost exhaustion, DoS           public chat API          per-tenant token budget
policy corpus            poisoned or stale document     ingestion pipeline       provenance, review, versions
model artifact (self)    malicious or tampered weights  registry, download       digest pin, signed provenance

Prompt injection is untrusted text that tries to change what the model does, overriding the task the operator configured. It is called direct when the author of the text is the person typing in the chat box, and indirect when the text arrives inside content the system fetched on the user's behalf: a retrieved chunk, a web page, an email, a PDF, a tool response. Indirect injection is the more dangerous case for an enterprise assistant, because the victim is a legitimate user who never typed anything hostile, and the author is someone you may never have heard of.

It helps to separate it from two things that look similar. A legitimate request can sound like an instruction ("answer in French", "summarise only section 3") and the user is entitled to give it, because it changes style or scope within their own permissions. What makes an instruction hostile is not its imperative mood but that it tries to obtain something the sender's authority does not cover: other users' data, a tool the session is not entitled to, a policy override. And an ordinary model mistake (a hallucinated policy, a wrong citation; see Hallucinations) has no adversary: it is a quality failure measured by groundedness checks, not blocked by security controls. The distinction matters operationally because the metrics, owners and responses differ.

Why the prompt cannot be the boundary

A system prompt is the operator's text in the same context window as everything else (see System Prompts vs User Prompts). Modern models are trained to give system-level instructions more weight, and that does reduce successful attacks. But "reduces" is the operative word: the model is making a probabilistic judgement about which text to follow, and an attacker can iterate on phrasing at the cost of a few API calls. Defences that live in the prompt (delimiters around retrieved text, a reminder that documents are untrusted, a request to refuse certain instructions) are worth using as a first filter and should be measured, but they must never be the only thing between a malicious sentence and a sensitive action.

type Source = 'system' | 'user' | 'retrieved' | 'tool_result';

interface Segment {
  source: Source;
  origin: string;        // e.g. "kb:RET-POLICY-7#c0003" or "tool:order_lookup#call_91"
  text: string;
}

function renderContext(segments: Segment[]): string {
  return segments.map((s) => {
    if (s.source === 'system' || s.source === 'user') return s.text;
    // Untrusted content is wrapped and labelled; it is evidence, not instruction.
    return [
      '<untrusted source="' + s.origin + '">',
      s.text.replace(/<\/?untrusted[^>]*>/g, ''),   // keep the content from closing its own wrapper
      '</untrusted>',
    ].join('\n');
  }).join('\n\n');
}

Provenance is the more valuable half. When every segment carries its origin, the runtime can later answer questions the model cannot: did this tool proposal occur in a turn whose context included a chunk from an external source? Which chunk? That enables policies such as "a turn that included untrusted content may call read-only tools but may not trigger side effects without approval", which is a deterministic rule built on facts the runtime knows for certain.

This trace uses the page from the opening. Identifiers are invented; the point is to see which component makes each decision. The customer is user u_1042 of tenant acme, who owns order ORD-504821. The footer text attempts to make the assistant look up a different order, ORD-777003, belonging to another customer of the same tenant, and quote its contents.

1. 1

   Retrieval runs under the caller, not the service

   The retriever receives tenant=acme and the user's groups from the authenticated session, not from anything the model or the page said. Chunks the user may not read are never candidates. The tainted chunk is legitimately readable (it is part of the public returns policy), so it is returned. Retrieval ACLs cannot remove content that is authorised but hostile; that is why the later steps exist.
2. 2

   Context is assembled with provenance

   The runtime records that segment kb:RET-POLICY-7#c0003 has source retrieved, wraps it as untrusted evidence, and flags the turn as tainted=true because an external-origin chunk is present.
3. 3

   The model may comply, and that is survivable

   Suppose it does: it emits a structured tool proposal for ORD-777003. At this point nothing has happened. A proposal is text produced by an untrusted component.
4. 4

   The gateway decides from facts it controls

   It receives the principal (u_1042, acme) from the runtime's session, not from the model, validates arguments against the schema, and asks the order service whether that principal may read ORD-777003. The answer is no, so it denies with a machine-readable reason.
5. 5

   Denial is a first-class, logged event

   The audit record carries the call id, principal, tool, argument hash, decision, reason and the id of the tainted chunk. A counter for denied.not_owner increments; repeated hits from the same source chunk will page a human, and the chunk is quarantined for review.
6. 6

   The model gets a safe error and answers from policy

   The runtime returns a generic error, not details about the other order. The final answer cites the returns policy and says nothing about orders, since the user never asked about one.

{
  "ts": "2026-03-14T09:12:41.208Z",
  "request_id": "req_8f21c",
  "call_id": "call_91",
  "tenant": "acme",
  "principal": "u_1042",
  "tool": "order_lookup",
  "args_sha256": "5b0c…e4",
  "turn_tainted": true,
  "tainted_sources": ["kb:RET-POLICY-7#c0003"],
  "decision": "deny",
  "reason": "not_owner",
  "policy_version": "2026-03-01.2"
}

A model with no tools can only produce text, and text, though harmful, is bounded by what was placed in the context. Tools change the economics: now a successful injection can perform actions with real credentials. Remember also that an LLM never has live transaction data of its own; every fact about an order arrives through a tool the application chose to expose, under credentials the application chose to give it. That makes the tool layer the place where agent safety is decided (see Tools and Tool Calling for the mechanics).

Six recurring failures show up, and each has a deterministic answer.

Unauthorised actions. The model proposes a tool the session should not have. The gateway holds the allowlist per session mode (a docs-only session has no order tool) and refuses anything outside it, whatever the model says.

Parameter manipulation. The tool is legitimate but the argument is not: another customer's order id, a wildcard, an amount larger than the order. Schema validation catches malformed values; authorisation catches well-formed but forbidden ones. The two are different checks, and the second needs the principal.

Overprivileged credentials. If the order tool runs with a service account that can read every order, then every authorisation bug becomes a data breach. Prefer calling downstream services with the user's delegated, short-lived token or with a scoped credential that encodes the tenant, so the downstream system enforces row-level access as a second, independent check.

Side-effect retries. Models, networks and orchestrators retry. A refund call that times out and is repeated can pay twice. Side-effecting tools take an idempotency key derived by the runtime from the approved action, not chosen by the model, and the downstream system deduplicates on it.

Malicious tool output. A tool result is untrusted input like any retrieved chunk: an order's free-text "customer note" field may contain instructions. Results re-enter the context as labelled data and mark the turn as tainted. The same applies to descriptions and schemas served by third-party tool servers; see the supply-chain section.

Approval bypass. If the model can write the text a human approver reads, it can describe a refund of 900 as a "routine correction". The approval request must be rendered by the gateway from the validated arguments, and the approval must be bound to a hash of exactly those arguments so that the executed call cannot differ from what was approved.

export interface Principal { userId: string; tenantId: string }
export interface ToolCall { callId: string; tool: string; args: unknown }
export type Decision =
  | { allow: true }
  | { allow: false; reason: 'tool_not_allowed' | 'bad_args' | 'not_owner' | 'budget' | 'needs_approval' };

export interface GatewayDeps {
  allowedTools: ReadonlySet<string>;                    // from session mode, not from the model
  ownsOrder(p: Principal, orderId: string): Promise<boolean>;
  callsSoFar(sessionId: string): number;
}

const ORDER_ID = /^ORD-\d{6}$/;
const SIDE_EFFECT_TOOLS = new Set(['issue_refund']);
const MAX_CALLS_PER_SESSION = 5;

export async function decide(
  deps: GatewayDeps, sessionId: string, p: Principal, call: ToolCall, turnTainted: boolean,
): Promise<Decision> {
  if (!deps.allowedTools.has(call.tool)) return { allow: false, reason: 'tool_not_allowed' };
  if (deps.callsSoFar(sessionId) >= MAX_CALLS_PER_SESSION) return { allow: false, reason: 'budget' };

  const a = call.args as { orderId?: unknown } | null;
  if (typeof a?.orderId !== 'string' || !ORDER_ID.test(a.orderId)) return { allow: false, reason: 'bad_args' };

  if (!(await deps.ownsOrder(p, a.orderId))) return { allow: false, reason: 'not_owner' };

  // A turn that read external text may not trigger side effects without a human.
  if (SIDE_EFFECT_TOOLS.has(call.tool) && turnTainted) return { allow: false, reason: 'needs_approval' };
  return { allow: true };
}

Human approval deserves precision, because it is often added as a vague safety net. An approval gate is effective when the approver sees accurate, gateway-rendered facts (who, what, how much), when the number of approvals per hour is small enough to be read rather than clicked through, and when the approved action is bound to its arguments:

const canonical = canonicalJson({ tool: 'issue_refund', orderId, amountMinor, currency, tenant, user });
const actionHash = sha256(canonical);

approval = await requestApproval({ summary: renderFromValidatedArgs(canonical), actionHash });

// at execution time
if (sha256(canonicalJson(currentArgs)) !== approval.actionHash) throw new Error('args changed after approval');
execute(currentArgs, { idempotencyKey: actionHash });   // same action -> same key -> at most one refund

Leakage in AI systems is rarely a model "memorising" another tenant's data at inference time. It is almost always plumbing: a place where data was stored or fetched without the caller's scope attached. The model then faithfully writes it into an answer. There are six places to look, and the case below shows one of them in detail.

Retrieval ACLs. If chunks carry no tenant or group metadata, or the filter is applied by application code after the search, a user can receive chunks they should not read. Post-filtering is also a quality bug: with illustrative numbers, if a tenant owns 2% of a shared index and the retriever fetches the top 10 before filtering, the expected number of that tenant's chunks among them is 10 × 0.02 = 0.2, so most queries return nothing even though relevant documents exist. Apply tenant and group filters inside the index query (pre-filtering), or use a separate index or namespace per tenant when the isolation requirement justifies the cost.

Cache keys. Any cache that stores a model response, retrieved chunks or an embedding keyed only by the question will serve one tenant's answer to another. See the validity discussion in Caching in AI Systems.

Tool credentials. A shared service credential means the order system cannot tell tenants apart. Logs. Prompts and responses contain whatever the context contained; a debug log of full prompts is a second, less-protected copy of your retrieval store. Provider handling. Prompts leave your boundary when you call a hosted API, so data classification decides what may be sent at all. Response channels. Even a correct answer can carry data out if the interface renders what the model writes: an image or link with a URL whose query string encodes sensitive text will transmit it to an external host as soon as a browser fetches it. Render model output with an allowlist for link and image hosts and do not auto-fetch arbitrary URLs.

A concrete tenant-isolation failure

To cut latency and cost, a team adds an exact-match response cache. The key is a hash of the normalised question. Tenant acme asks "What is our enterprise refund window?" and the answer, derived from Acme's private contract addendum, is cached. Tenant globex later asks the same words and receives Acme's number with a citation to Acme's document title. No attacker was involved, no prompt injection, and every retrieval ACL worked, because the cache short-circuited retrieval entirely.

import { createHash } from 'node:crypto';

export interface Scope { tenantId: string; groups: string[]; aclVersion: number; corpusVersion: string }

// Everything that can change the answer or who may see it belongs in the key.
export function responseCacheKey(question: string, s: Scope, promptVersion: string, modelId: string): string {
  const parts = [
    s.tenantId,
    [...s.groups].sort().join(','),
    String(s.aclVersion),
    s.corpusVersion,
    promptVersion,
    modelId,
    question.trim().toLowerCase().replace(/\s+/g, ' '),
  ];
  return createHash('sha256').update(parts.join('\u0000')).digest('hex');
}

// The filter travels inside the vector query; it is never applied after the fact.
export function retrievalFilter(s: Scope) {
  return { tenant_id: s.tenantId, allowed_groups: { any_of: [...s.groups, 'tenant-public'] } };
}

The control was one line, but the lesson is structural. For every component that stores or retrieves derived data (cache, index, memory store, evaluation set, log), ask what the key and the access check are, and write a test that stores under tenant A and reads as tenant B. A cache on a shared prompt-prefix is a more subtle instance of the same question: if it is shared across tenants, ensure that only content that is identical and non-sensitive (for example the system prompt) forms the shared prefix.

An inference endpoint is expensive per request and has limited capacity, which makes it a better target than a typical CRUD API. Abuse falls into three families. Denial of service exhausts GPU or provider capacity so legitimate users queue or are shed; a handful of requests with maximal input length and maximal output length can do more harm than thousands of short ones. Cost exhaustion ("denial of wallet") makes you pay: the attacker does not need to break anything, only to send valid requests. Scraping and extraction collects many responses to copy your proprietary content or to approximate model behaviour; the realistic concern for most enterprise assistants is content exfiltration at volume, which per-user quotas and anomaly detection address, rather than full model extraction.

An illustrative cost-exhaustion calculation

Use made-up prices: 3 currency units per million input tokens and 15 per million output tokens (illustrative, not any provider's price). A request with 6,000 input tokens and a 800-token output costs 6,000 × 3 / 10⁶ + 800 × 15 / 10⁶ = 0.018 + 0.012 = 0.030. A script that sends 5 requests per second for one hour sends 18,000 requests, costing 18,000 × 0.030 = 540. A request-rate limit alone does not stop this if 5 per second is below the limit; the limit has to count tokens. With a per-tenant budget of 200,000 tokens per minute and 6,800 tokens per request (6,000 + 800), the tenant can make 200,000 / 6,800 ≈ 29.4, so 29 full-size requests per minute. One hour is 29 × 60 = 1,740 requests, costing 1,740 × 0.030 = 52.20, about a tenth of the unbounded figure. Longer-term caps (daily spend per tenant, an alert at 50% of the cap) limit the remaining exposure.

Drag the budget below to see how the per-tenant token limit changes what the same script can spend in an hour.

export class TokenBudget {
  private level: number;
  private last: number;
  constructor(private capacity: number, private refillPerSec: number, now = Date.now()) {
    this.level = capacity;
    this.last = now;
  }
  /** Reserve estimated tokens (input + max_output) before calling the model. */
  tryConsume(tokens: number, now = Date.now()): boolean {
    this.level = Math.min(this.capacity, this.level + ((now - this.last) / 1000) * this.refillPerSec);
    this.last = now;
    if (tokens > this.level) return false;
    this.level -= tokens;
    return true;
  }
  /** Return unused reservation once the real output length is known. */
  refund(tokens: number) { this.level = Math.min(this.capacity, this.level + tokens); }
}

// 200,000 tokens per minute sustained = 3,333.3 tokens/s; burst capacity of 1 minute.
const perTenant = new TokenBudget(200_000, 200_000 / 60);

Two further points. First, budgets should be hierarchical (global, tenant, user, API key) so one noisy user cannot consume a tenant's entire allowance and one tenant cannot consume the fleet; capacity protections such as load shedding and queue limits are covered in AI Infrastructure Scaling. Second, a self-hosted inference server must not be reachable except through the gateway: model servers often ship with no authentication, so network policy and private endpoints are the primary control, not an afterthought.

AI systems import more third-party material than most services: model weights, tokenizers, Python and JavaScript packages, container images, datasets, embedding models, prompt templates, evaluation sets, and tool servers or plugins. Each is code or data that influences behaviour, so each needs provenance (where did it come from), integrity (is it what we reviewed) and a controlled update path.

Model artifacts. Some serialisation formats (notably Python pickle, still used by some checkpoint formats) can execute arbitrary code when loaded, so loading an untrusted checkpoint is equivalent to running an untrusted program. Prefer formats designed to hold only tensors, load in an isolated environment with no credentials, record the artifact's cryptographic digest in your registry, and deploy only by digest, never by a mutable tag like latest.

Dependencies and images. Pin versions with lockfiles and hashes, scan images, build from minimal base images, and keep GPU nodes patched. Serving frameworks are internet-facing software and receive security advisories like any other.

Plugins and MCP servers. A tool server is third-party code whose output re-enters the model's context, and whose tool descriptions are themselves text the model reads. Treat a new server like a new dependency: review who maintains it, run it with least privilege and network egress limits, pin its version, and treat its descriptions and results as untrusted input, since a changed description is a way to alter model behaviour without changing any code you reviewed. See Model Context Protocol for the protocol itself.

Documents. The corpus is also supply chain. A "poisoned document" is content written (or edited) by someone outside your trust to influence answers or smuggle instructions; this is the indirect injection of the opening scenario viewed from the ingestion side. Controls at the door: ingest only from registered sources, record the source and the last editor, require review for sources open to external edits, strip hidden or zero-width text and markup that renders invisible, quarantine content that trips detectors, and version the corpus so a bad ingest can be rolled back by version rather than hunted chunk by chunk.

model:
  name: support-assistant-llm
  artifact: registry.internal/models/support-llm@sha256:9c1f…7a   # by digest, never by tag
  format: tensors-only
  provenance: { source: vendor-release, verified_by: security-review-2026-02, scan: passed }
tool_servers:
  - name: order-lookup
    image: registry.internal/tools/order-lookup@sha256:41d0…b2
    egress: [orders.internal:443]
    credentials: per-request-delegated
prompts:   { version: "2026-03-01.2", reviewed: true }
corpus:    { version: "kb-2026-03-14", sources: [policy-wiki, contracts-store] }

It is worth keeping two timelines apart. Inference-time attacks (prompt injection, retrieval poisoning, tool abuse) happen while the system answers requests; their effect ends when the text leaves the context, and you can fix them by removing the document or changing a rule. Training-time attacks corrupt what the model or its fine-tuning data contains; the effect is baked into weights, persists across requests, and cannot be removed by deleting a prompt. Many enterprise assistants never train a model and only fine-tune or embed, which shrinks this section of the threat model, but not to zero: fine-tuning sets, embedding models and evaluation sets are all data pipelines.

Provenance and poisoning. If a fine-tuning set is assembled from support transcripts and scraped pages, someone who can influence those sources can influence behaviour: planting a trigger phrase that elicits a particular answer, or biasing responses. Mitigations are mostly data engineering: track each example's source, restrict who can write to training sources, sample and review, deduplicate, compare a candidate model against a held-out clean evaluation set and a set of trigger-style probes before promotion, and keep the dataset versioned and immutable so a model can be tied to exactly the data that produced it.

Privacy. Training on data containing personal information risks the model reproducing it. The reliable control is upstream: minimise and redact before data enters a training set, honour deletion and consent obligations at the dataset level, and separate production customer data from experimentation environments. Access to datasets, checkpoints and experiment trackers should be least-privilege and audited, since a checkpoint can embed training data. Retention applies here too: how long are raw transcripts kept after the training set has been built (see Training Infrastructure for the pipeline)?

Artifact integrity. Checkpoints written during training are valuable and trusted by downstream steps. Store them in access-controlled storage, record digests when written, and verify digests before they are promoted to the registry. A tampered checkpoint is a supply-chain compromise that arrives from the inside.

Defence in depth for an AI service is a chain of independent controls, arranged so that a failure of one is caught by the next, and ordered so that the cheapest checks fire first. The sequence of the support assistant looks like this:

Identity and authentication establish who is calling, at the edge, using the organisation's identity provider, with the tenant derived from the verified credential rather than a request field. Authorization decides what that principal may do and see, evaluated by services that hold the policy data. Least privilege shapes what each component can reach: the retriever has read access to the index only, the order tool calls the order service with a scoped token, the model container has no credentials at all. Retrieval filters put tenant and group scope inside the query. Secret isolation keeps keys in a secret manager and out of prompts, logs and model-visible tool output; the model should never be able to ask for a secret because it never possesses one. Sandboxed execution runs any code-running or browsing tool in a container with no ambient credentials, restricted egress and CPU, memory and time limits. Output validation parses structured outputs against a schema, checks citations point to retrieved chunks the user could read, scans for sensitive patterns before the response leaves, and sanitises rendered output. Audit and monitoring record every decision with identifiers. Approval gates reserve human judgement for irreversible or high-value actions.

A summary of which layer defends against which threat from earlier sections helps when reviewing a design for gaps. An empty cell is a question to answer, not a mark to fill:

threat                      authN  authZ  retr.filter  gateway  sandbox  output  audit  approval
--------------------------  -----  -----  -----------  -------  -------  ------  -----  --------
indirect prompt injection     .      ○         .           ●       ○        ○      ○       ○
cross-tenant retrieval        .      ●         ●           .       .        ○      ○       .
cache / log leakage           .      ○         .           .       .        ●      ●       .
unauthorised tool call        ○      ●         .           ●       .        .      ○       ○
side-effect abuse / retry     .      ○         .           ●       .        .      ○       ●
malicious tool output         .      .         .           ○       ●        ●      ○       .
endpoint abuse / cost         ●      ●         .           ●       .        .      ●       .

Personal and confidential data moves through the AI stack in more places than the primary database. Follow a customer's email address from the moment the user types it: it is in the HTTP request, then possibly in the conversation store, in the assembled prompt sent to the model, in the model's output, in the trace and log pipelines, in the response cache, in evaluation datasets built from production logs, in the provider's systems if the model is hosted, and in backups of all of the above. A deletion request is only honoured if you know every one of those locations.

Redaction belongs before data reaches places that are read by humans or pipelines: logs, traces, analytics and evaluation sets. A deterministic pass (patterns for emails, card numbers, national id formats, plus an allowlist of fields that may be logged raw) is predictable and testable; add an ML detector for free text only as a second layer, and measure its misses. For the model prompt itself, redact what the task does not need: a policy question does not require the customer's email, so do not place it in the context. Encryption in transit and at rest is table stakes; the interesting decisions are key scope (per-tenant keys where the isolation requirement demands it) and who can decrypt. Provider transmission is governed by classification: define which data classes may go to a hosted model, in which region, under which contractual retention and training terms, and enforce it in the gateway rather than relying on developers to remember. Read the provider's current data-handling documentation for specifics; they differ and change.

logging:
  prompts:       { store: false, sample_for_debug: 0.0 }         # default: do not store full prompts
  metadata:      { store: true, fields: [request_id, tenant, model_id, tokens_in, tokens_out, latency_ms] }
  tool_calls:    { store: true, args: hash_only }
retention_days:
  conversation_store: 30
  response_cache:     7        # TTL, plus purge on ACL or corpus change
  audit_log:          365      # security evidence is kept longer than content
  eval_datasets:      redacted_only
provider:
  region: eu
  data_classes_allowed: [public, internal]   # 'restricted' is blocked at the gateway

Incident response for a leak follows the inventory. Contain (disable the affected route or cache namespace, rotate any exposed key), scope (use audit records and request ids to list affected tenants, users and time window), purge (remove cached answers, index entries or log lines, then verify with a read-as-the-other-tenant probe), notify according to contractual and regulatory obligations, and fix the root cause with a regression test. The quality of step two depends entirely on whether the audit log was written before the incident.

Security controls that are not tested decay: a refactor drops a filter, a new tool skips the gateway, a model update changes how often an injection succeeds. The testing plan has two halves. Deterministic tests verify the code that enforces policy and run on every commit with pass/fail results. Adversarial evaluations measure the statistical behaviour of the model and run on every model, prompt or corpus change, with rates and thresholds.

Deterministic tests

Authorization and tenant isolation tests create two tenants with seeded, uniquely marked data (a canary string in each document) and assert that principal A can never obtain B's canary through retrieval, cache, tool, log or error message. Tool-call schema tests feed the gateway malformed, oversized, wildcard and cross-owner arguments and assert denial with the expected reason code. Audit replay takes recorded audit records for a window and re-evaluates them under the current policy version to see which would now be denied, which catches policy changes with unintended effects before they ship.

import { describe, it, expect } from 'vitest';
import { decide } from './gateway';
import { responseCacheKey } from './cache';

const acme   = { userId: 'u_1042', tenantId: 'acme' };
const scopeA = { tenantId: 'acme',   groups: ['support'], aclVersion: 3, corpusVersion: 'kb-1' };
const scopeB = { tenantId: 'globex', groups: ['support'], aclVersion: 3, corpusVersion: 'kb-1' };

describe('cache keys', () => {
  it('differ across tenants for identical questions', () => {
    expect(responseCacheKey('refund window?', scopeA, 'p1', 'm1'))
      .not.toBe(responseCacheKey('refund window?', scopeB, 'p1', 'm1'));
  });
  it('change when permissions change', () => {
    expect(responseCacheKey('q', scopeA, 'p1', 'm1'))
      .not.toBe(responseCacheKey('q', { ...scopeA, aclVersion: 4 }, 'p1', 'm1'));
  });
});

describe('gateway', () => {
  const deps = {
    allowedTools: new Set(['order_lookup']),
    ownsOrder: async (_p: unknown, id: string) => id === 'ORD-504821',
    callsSoFar: () => 0,
  };
  const call = (args: unknown, tool = 'order_lookup') => ({ callId: 'c1', tool, args });

  it('denies another customer\'s order even if the model asked politely', async () => {
    expect(await decide(deps, 's1', acme, call({ orderId: 'ORD-777003' }), true))
      .toEqual({ allow: false, reason: 'not_owner' });
  });
  it('rejects wildcard and oversized ids', async () => {
    for (const bad of ['ORD-*', 'ORD-1234567', '', 42]) {
      expect((await decide(deps, 's1', acme, call({ orderId: bad }), false)).allow).toBe(false);
    }
  });
  it('denies tools outside the session mode', async () => {
    expect(await decide(deps, 's1', acme, call({ orderId: 'ORD-504821' }, 'issue_refund'), false))
      .toEqual({ allow: false, reason: 'tool_not_allowed' });
  });
  it('allows the owner', async () => {
    expect(await decide(deps, 's1', acme, call({ orderId: 'ORD-504821' }), false)).toEqual({ allow: true });
  });
});

Adversarial evaluations

An adversarial corpus is a versioned set of hostile inputs: user messages that try to override instructions, documents with embedded instructions of varying subtlety (visible, hidden, multilingual, split across chunks), tool results with injected text, and requests that probe permission boundaries. Keep it defensive in purpose: each case records the attacker goal (for example "obtain another customer's order"), the expected safe outcome, and the control that should stop it. Run the corpus through the full pipeline against a test tenant with canary data and record two measures separately: how often the model attempts the forbidden action (a model-quality signal that you try to drive down) and how often the forbidden action succeeds (a control signal that must be zero). If attempts rise after a model change, you are leaning harder on the deterministic layers, which is fine; if successes are ever non-zero, a control is broken.

Regression after model or prompt changes. Treat a model version, prompt version or retrieval configuration change as a release that re-runs the whole adversarial suite and compares attempt rates with the previous baseline. Include a held-out set of cases that were never used to tune prompts, since a prompt tuned to the visible cases can overfit them. Human red-teamers add creativity the corpus lacks; feed what they find back as new corpus cases. Both kinds of testing see the same boundary: they should verify that the system is safe when the model misbehaves, which means the tests must be able to force the model to misbehave, for example by stubbing the model to emit a forbidden tool proposal, and assert that the gateway still denies.

Security telemetry is a specialised slice of the observability described in AI Observability. A metric is only useful if someone knows what to do when it moves, so each one below comes with a response.

Blocked unauthorised attempts, counted by reason code (not_owner, tool_not_allowed, bad_args). A steady low rate is normal: models make mistakes and users probe. A step change, or a cluster tied to one source chunk, one user or one tenant, is an investigation trigger. Data-leakage test results from the canary tests running continuously against production-like data: a single canary appearing in the wrong tenant's output is a page, not a ticket. Suspicious tool activity: tool calls per session above the usual distribution, calls in tainted turns, repeated denials followed by differently-worded retries. Token abuse: tokens per user per hour, share of requests at maximum length, spend as a fraction of the daily tenant cap. False positives: legitimate requests denied or answers blocked by output checks, measured from user complaints, retries and sampled review. Without it you will tune the controls to be quiet by making them loose, or loud by making them useless. Audit coverage: the fraction of tool calls and denials that produced an audit record with all required fields; this should be 100%, and a gap means incident response will be guesswork.

Escalation should be defined before it is needed. A possible shape (illustrative): low-severity signals (isolated denials, a quarantined chunk) are queued for next-business-day review; medium (a rising denial cluster, token spend at 80% of cap) page the on-call engineer; high (a canary leak, a side-effect action in a tainted turn, a suspected key exposure) page security and trigger the containment steps from the lifecycle section. Each level names an owner and a first action, such as disabling the affected tool or cache namespace by feature flag. This only works if disabling is cheap: ship a kill switch per tool and per cache layer.

A short incident walk-through

At 09:12 the denied.not_owner counter rises from a baseline of about 2 per hour to 40 in ten minutes. Every record carries tainted_sources: ["kb:RET-POLICY-7#c0003"], so the cluster maps to one chunk. The on-call engineer quarantines the chunk with a single index operation, confirms from the audit log that no call in a tainted turn was allowed (all 40 were denied), and finds from the ingestion record that the source page was edited by an account outside the company two days earlier. The page is reverted by corpus version, the source's edit rights are reviewed, and the page's text is added to the adversarial corpus. Impact: no data disclosed, 40 users saw a normal answer after one denied tool call. The incident worked because the audit log carried the source id and because denial happened in deterministic code.

Two systems share a retriever and a model but should not share a security posture. The first is an internal read-only documentation assistant for employees: it answers questions about engineering runbooks and has no tools. The second is the support assistant extended with an issue_refund tool that moves money.

For the docs assistant, the worst realistic outcome of a successful prompt injection is a wrong or misleading answer, or disclosure of a document the user was not allowed to read if retrieval ACLs are wrong. Both are bounded by the retrieval filter and answer verification. Controls that earn their cost are SSO authentication, a retrieval ACL filter, untrusted-content labelling, output citations to authorised documents, sane rate limits and redacted logs. Human approval would add friction and no safety, because there is nothing to approve. A poisoned runbook still matters (an engineer might follow a harmful command in an answer), which is addressed by corpus review and citing sources so a human can check, not by more filters.

For the refund agent the worst outcome is financial loss and wrong money movement, which cannot be undone by regenerating an answer. Illustratively, if an attack could trigger 40 refunds of 30 each before detection the loss is 40 × 30 = 1,200, so the controls are sized to that exposure. Every control from the docs assistant remains, and the following become mandatory: per-user ownership checks on the order; a refund amount capped by the order's remaining refundable balance as read from the order system, never from the model; an idempotency key derived from the approved action; approval for refunds above a threshold and for any refund proposed in a tainted turn; a per-session and per-tenant refund count and value limit; a separate narrowly-scoped credential for the payment system; alerts on refund rate; and a kill switch that disables the tool without redeploying.

The difference is not "more AI safety" in the second. It is that the blast radius of a compromised model output changed from text to transactions, so controls must move from "reduce the chance of a bad answer" to "make a bad proposal harmless". A useful review question for any new tool is therefore: if the model is fully adversarial for one turn, what is the worst thing this tool can do, and what stops it? If the answer depends on the model behaving, redesign the tool, narrow its permissions, or put a human in front of it.

The relationship between these systems and the rest of the operating architecture (how the gateway, the policy configuration and the evaluation gate are deployed and rolled back together) is the subject of the capstone, since a control that is correct in design but ships misconfigured, or is skipped on a degraded path, protects nothing. Two rules carry over: security configuration (policies, allowlists, schemas, budgets) is versioned and released like code with tests, and every degraded mode (fallback model, cache outage, retrieval timeout) must be reviewed for what it does to authorisation, because fallbacks are where fail-open behaviour tends to hide.

1. 1

   Enforce security in deterministic code (authentication, authorisation, tool gateway, output validation), never by instructing the model; a model can help detect suspicious content but is not a control.

2. 2

   Treat every place text enters the context as untrusted: chat input, uploads, retrieved chunks, tool results and plugin output. Keep provenance on each segment so the runtime can deny actions that originate from untrusted text.

3. 3

   Gateway checks for tools: allow-list per session, validate arguments against the caller's ownership, run with scoped credentials rather than a broad service account, make side-effecting calls idempotent, and gate high-impact actions behind approval shown from system data, not model-written text.

4. 4

   Apply tenant and group filters in retrieval, include tenant and permission scope in every cache key, and pass per-user credentials to tools, so cross-tenant leakage cannot rely on the model behaving.

5. 5

   Protect the endpoint with hierarchical token budgets (global, tenant, user, API key), and pin and verify model artifacts, dependencies, plugins and ingested documents as supply chain.

6. 6

   Redact before logs, traces and analytics; keep prompts and user IDs out of metric labels.

7. 7

   Test the controls in CI: two-tenant isolation tests, a versioned adversarial corpus, and regression runs on every model, prompt or retrieval-configuration change; count denials by reason code and define escalation in advance.