Enterprise AI Applications

Watch AI move through HR, finance, sales, support, legal, and research — then see how a shared enterprise AI platform connects every department into one organizational capability.

Enterprise AI differs from a single-team AI application in that many use cases share one platform, so multi-tenancy, access control, governance, cost allocation and operational ownership must be solved once, in code, for everyone. An enterprise AI application platform is a shared set of services (identity, policy, model access, retrieval connectors, tool access, evaluation, monitoring, cost metering and audit) on which many departmental AI use cases run, each with its own owner, data scope and success measure. The organisation governs once what every use case would otherwise have to solve alone: who is asking, what they may see, which model may see it, what the call cost, and what happened afterwards. What stays inside each department's application is the part that is genuinely theirs: the prompts, the workflow, the business rules and the definition of a good outcome.

This lesson is about that organisational layer. It deliberately does not re-teach how a single assistant, support system or business agent works internally; those chapters own that. The question here is what happens when you have eight of them, owned by eight different teams, touching payroll, contracts and customer records, and someone in security asks where all of it is written down. For the generic technical blueprint of a production AI system, see Production AI Architecture; this lesson covers organisation-wide governance.

Consider a fictional company, Northwind Logistics, with about 6,200 employees across Germany, the Netherlands and the United States. Within a year of the first successful chatbot demo, HR has a policy assistant built by a contractor, Sales has a call-summary tool from a vendor trial, Support has an agent-drafting prototype built by two engineers, Finance has a script that extracts fields from supplier invoices, and Legal has an experiment that flags unusual clauses. Each team did something reasonable. Taken together they have built the same plumbing five times, and each copy has its own flaws.

Follow the costs this creates. First, duplicated engineering: five teams each write retrieval code, a prompt store, retry logic and a logging format, and each discovers the same bugs separately. Second, divergent security. The CRM connector that Sales wrote filters records by the user's sales territory; the one Support wrote runs under a service account that can read every account, because that was the quickest way to get the demo working. Both are "the CRM assistant", and nobody can say what either may reveal without reading code. Third, invisible spend: Model key B is on an individual's card, so the cost appears as an expense claim rather than a budget line, and no one can answer what a resolved ticket costs. Fourth, no change control: when the model provider deprecates a model version, five teams find out independently, and when HR edits a system prompt there is no record of what the assistant said last Tuesday. Fifth, inconsistent data handling: one tool sends full employee records to a third-party API, another logs prompts containing personal data to a SaaS log tool with indefinite retention, and the privacy officer has no inventory to check either against.

The natural reaction is to centralise everything in one team that builds all the assistants. That fails in the opposite direction: the central team becomes a queue, it does not know what a good contract-risk flag looks like, and departments route around it. The workable middle, which the rest of this chapter develops, is a platform that centralises the cross-cutting, undifferentiated and risky parts, plus a registry and a review process, while product teams keep ownership of their use case and its quality. See the AI Applications overview for the single-product version of this stack; the enterprise problem is that the same stack has to serve many products with different risk levels without letting them see each other's data.

Before building a platform, the organisation needs to know what it is hosting. Northwind's intake process starts from workflows rather than from technology: what does a person in this department do repeatedly, with what inputs, and what does a good result look like? Mapping workflows to the application patterns taught elsewhere in this section makes the platform requirements visible, because each pattern leans on different shared services.

HR employees ask what the carry-over rule for unused leave is. That is question answering over a changing document collection, a retrieval pattern (see AI search systems and RAG). Finance receives supplier invoices as PDFs and photographs; the job is turning them into validated ledger records, an extraction pattern (document intelligence). Sales wants a prioritised list of accounts to call this week, which is ranking over structured signals and is closer to a recommender (recommendation systems) than to a language model. Support agents want a drafted reply they edit and send, a copilot (AI copilots) where the human owns the outcome. Payroll corrections involve changing money records through a fixed process, which at most warrants a bounded agent in front of a deterministic workflow (AI agents in business).

Two things follow from that mapping. One is that the platform has to support at least five different shapes of work: retrieval needs connectors and an index, extraction needs document handling and validators, copilots need low-latency streaming and feedback capture, and agents need a workflow runtime with approvals. The other is that some candidates should be rejected. "How many vacation days do I have left?" looks like an assistant use case, but the answer lives in a database row for that employee and is computed deterministically; it should be a tool call or a plain screen, and a model is useful only to phrase the answer. Routing it through document retrieval would be both slower and capable of being wrong. A good intake review asks what is the simplest system that meets the needand treats AI as justified only where language variability, unstructured input or ranking over many weak signals is the real difficulty.

An intake record: ownership, risk and measurable success

Each accepted use case gets a record with four kinds of owner and a risk class. The business owner is accountable for the outcome (HR's head of people operations for the policy assistant). The data owner decides what data may be exposed and to whom; for payroll data that is the payroll controller, not the team building the assistant. The technical owner carries the pager. The risk class follows what the system can read and what it can change: a read-only assistant over public policies is low; read access to contracts or compensation is medium to high; write access to money or employment records is high and implies human approval and deterministic validation. Risk class decides how much of the review process, covered later, is mandatory. It is set by the data and the actions, not by how clever the model is.

Success must be measurable in business terms at intake, because the stop/go decision later depends on it. For the HR assistant, Northwind picks three measures: share of policy questions answered with a correct, cited answer (judged on a golden set, below), share of conversations that do not end in a ticket to HR for the same topic within seven days, and zero permission-leakage failures in the test suite. For payroll requests the measures are different: adjustments applied without rework and median time from request to scheduled correction. Notice that none of these is "number of users" or "number of prompts", which measure adoption, not value.

The platform is a set of services that every application reaches through a narrow, authenticated interface. The figure draws the dividing line the chapter keeps returning to: the application layer holds what makes a use case distinctive; the platform layer holds what must behave identically for all of them and what the organisation must be able to inspect.

Walk through each service by the question it answers. Identity and SSO answers "who is this, which roles and region do they have, and on whose behalf is the application acting?" Integrating with the corporate identity provider means the platform receives a verified user identity on every request, and downstream services evaluate permissions for that user rather than for the application. The distinction is the point of the next section. The model gateway is the single place applications call models. It owns the provider credentials (so no team holds an API key), applies per-application quotas and rate limits, redacts or blocks configured data classes before they leave the company, routes requests to a model according to policy, records token usage and latency, and returns a uniform response shape. Because applications code against the gateway contract, changing provider or version becomes a configuration change that can be tested and rolled out in stages, not a code change in every department. See Model Serving and Inference Infrastructure for what sits behind a gateway when models are run privately.

The prompt and configuration registry stores system prompts, retrieval settings, model routes and guardrail settings as versioned artefacts with an owner and a change history. A system prompt is production configuration: when it changes, behaviour changes, and answering "what was the assistant instructed to do on 14 October" should be a lookup. Retrieval connectors ingest from source systems, chunk and embed content, and, crucially, carry the source's access-control metadata into the index so that retrieval can be filtered per user. The tool or MCP gateway exposes actions (read a leave balance, draft a payroll adjustment) behind allow-lists and scopes; the Model Context Protocol is one way to standardise that interface, and the tool calling lesson explains how a model proposes calls. The workflow runtime executes multi-step processes durably: it persists state, retries failed steps, waits for human approvals and resumes. The evaluation service runs each application's test suites and gates releases. Observability, cost metering and the audit log record what happened, what it cost, and which versions were responsible; AI Observability covers the telemetry itself.

What stays in the application

An application still owns its system prompt (versioned in the registry), its domain-specific workflow, the shape of its user interface, its business rules (an overtime multiplier is payroll logic, not platform logic) and its evaluation set, because only the department can say what a correct answer is. A useful test for placing a feature: if two departments would need it to behave identically and the company would be embarrassed if one copy were wrong, it belongs in the platform. If its correctness depends on knowing what HR or Legal means by "good", it belongs with the application. Over-centralising costs flexibility; under-centralising brings back the sprawl.

The application registry

The registry is the platform's index of record. Nothing gets a gateway credential, a connector or a tool grant unless it has an entry. Here is the entry for the HR policy assistant, expressed as JSON for illustration (the exact schema is an organisational choice; what matters is that it is machine-readable, so the platform can enforce it):

{
  "app_id": "hr-policy-assistant",
  "pattern": "retrieval",
  "owners": {
    "business": "head-of-people-operations",
    "data": "hr-data-steward",
    "technical": "team-people-platform"
  },
  "risk_class": "low",
  "data_domains": ["hr-policies"],
  "audience_roles": ["employee", "manager", "hr"],
  "regions_allowed": ["eu-central", "us-east"],
  "model_route": "general-small",
  "prompt": { "id": "hr-policy-system", "version": 12 },
  "tools": [{ "name": "get_leave_balance", "scope": "self-read" }],
  "retention_days": { "prompts": 30, "traces": 90, "audit": 2555 },
  "eval_suite": "hr-policy-golden-v7",
  "success_metrics": [
    { "name": "cited_correct_rate", "target": ">= 0.90" },
    { "name": "permission_leaks", "target": "== 0" }
  ],
  "budget_monthly_usd": 5000,
  "stage": "production",
  "next_review": "2027-01-15"
}

Note the retention block. Prompts may contain personal data, so Northwind keeps raw prompts for 30 days, traces for 90, and audit events for seven years (2,555 days), because the audit log records identifiers and decisions rather than content. Different retention for different artefacts is a deliberate privacy decision, one that is impossible to make if every team logs to its own tool.

The data that makes enterprise AI useful is the data that is most dangerous to expose. HR holds employment records and compensation. Legal holds contracts and privileged advice. Sales holds customer and pipeline data. Finance holds the ledger. Each has its own regulations, its own owner and, usually, its own existing access model enforced inside the source system. A shared AI platform must preserve those boundaries and must not become a way around them.

The figure puts the enforcement points where they have to be. The employee's device is untrusted: its input may be mistaken, hostile, or contain instructions aimed at the model. The application zone is trusted to do its own job but not to decide access; it passes the employee's identity and its own application identity to the platform. The policy enforcement point verifies both, checks that the application is registered for the data domain and that the user's role, region and quota allow the request, and only then lets the retrieval and tool gateways query the data zone on behalf of that user. The model provider zone is outside the company boundary and sees only a minimised, redacted prompt; what the data processing contract with the provider permits (training use, retention, sub-processors, region) is a legal and procurement decision, and the gateway's job is to make sure traffic cannot reach an endpoint that has not been approved.

Least privilege means the user's privilege, not the application's

The most common architectural error in departmental assistants is giving the application a powerful service account and relying on the prompt to hold back. If Support's connector can read every CRM account and the model is told "only discuss the current customer", then any retrieval bug, prompt injection inside a customer email, or misrouted query exposes other accounts. The safe pattern is the opposite. The connector queries with the end user's identity (or a token derived from it that carries the user's groups), so the source system's own rules apply. When that is impossible (the source has no per-user API), the platform applies equivalent filters from copied ACL metadata and records that decision. Least privilege applies to tools in the same way. The payroll tool in the example below is executed as the requesting employee, so it can only draft an adjustment for that employee; it cannot be talked into drafting one for a colleague, because the authorisation layer checks the identity, not the sentence.

A shared vector index must still enforce access

Putting HR policies, contract summaries and CRM notes into one vector database saves operational effort, but similarity search knows nothing about permissions. A query about "senior engineer compensation" will find the compensation bands document if it is in the index and is similar, no matter who asked. Access therefore has to be applied as part of retrieval. There are two places to do it, and they behave differently.

interface Principal { id: string; groups: string[]; region: string; }

async function retrieve(user: Principal, appId: string, query: string, k = 5) {
  const app = registry.get(appId);                       // allowed domains, regions
  if (!app.regionsAllowed.includes(user.region)) throw new Denied('region');

  // 1. Pre-filter: restrict the search itself, so top-k is computed
  //    only over documents this user may read.
  const hits = await index.search(embed(query), {
    k: k * 3,                                            // headroom for step 2
    filter: {
      domain: { in: app.dataDomains },
      residency: { in: regionsFor(user) },
      acl: { anyOf: [user.id, ...user.groups] },
      status: 'published',
    },
  });

  // 2. Re-check at the source for domains where stale ACLs are unacceptable.
  const allowed = [];
  for (const h of hits) {
    if (!h.domainRequiresLiveCheck || await source.canRead(user, h.docId)) allowed.push(h);
  }
  audit.log({ user: user.id, appId, docIds: allowed.map(h => h.docId) });
  return allowed.slice(0, k);
}

Post-filtering (search first, drop anything the user cannot see) is easy to bolt on but fails in two ways. It hurts recall: if seven of the ten nearest chunks are restricted, the user's top-three is built from the remaining three, which may be poor matches, and the assistant says "I found nothing relevant" while a permitted and relevant chunk sat at rank 14. It also leaks through side channels: counts ("no results, but 7 restricted matches"), latency differences, or citation numbering that reveals something was removed. Pre-filtering applies the ACL inside the index query. Most vector stores support metadata filtering, though how well they combine filters with approximate search varies, so Northwind's platform team tests recall under restrictive filters rather than assuming it. For the most sensitive domains (HR case notes, privileged legal matters), the alternative is a separate index per domain or per tenant: slightly more operational cost for a boundary that a filter bug cannot cross.

Vary k and the post-filter search depth to see how many permitted results each approach actually returns.

ACL freshness is the subtle part. Permissions live in the source system; the index holds a copy. If an employee moves from Sales to Marketing at 09:00 and the ACL sync runs every 15 minutes, there is a window in which the index still says they can read Sales pipeline notes. Whether that is acceptable depends on the domain, which is why the registry marks live checks per domain: for policy documents, a 15-minute lag is tolerable; for contracts, the final chunks are re-authorised against the source before they enter a prompt. The same logic applies to deletion. When a document is withdrawn or an employee exercises a deletion right, its chunks and embeddings must be removed from the index, and embeddings derived from personal data should be treated as personal data in the platform's retention and deletion design (confirm the specifics with your privacy counsel).

Residency, retention, consent, and separation of duties

Regional residency rules turn into routing constraints. German employee records are indexed in the EU region; the gateway refuses a route that would send a prompt containing that data to an endpoint outside the approved regions, even if the application asks for it. Retention is enforced by the artefact store, not by good intentions: raw prompts expire after 30 days, and conversational history, which is not durable memory, is not copied into a user profile unless a feature explicitly needs it and the user has consented. Consent and purpose limitation are recorded per use case in the registry; a dataset gathered for the policy assistant is not available to the sales recommender merely because both are on the platform. Separation of duties shows up in three places: the person requesting a payroll change cannot approve it; the team that builds an application does not get standing read access to production prompts containing employee data; and whoever approves a prompt change is not the person who made it.

These are Northwind's own policy choices for a fictional company, not legal requirements. Residency, retention and consent obligations vary by jurisdiction and sector; privacy law such as the GDPR is one common driver for the EU and similar frameworks apply elsewhere. Treat this as general information, not legal advice, and confirm the specifics with your privacy counsel.

For the security-testing side (prompt injection, exfiltration, supply-chain risk for connectors), see AI Security. For the guardrail and approval patterns that wrap risky actions, see Human-In-The-Loop & Guardrails.

The platform's value shows when two very different requests arrive through the same interface. Priya Raman (employee EMP-4417, based in Germany) opens the company assistant and asks two things within a few minutes. The first is informational; the second changes money. The system should treat them as different kinds of work even though the text box is the same.

Request 1: the policy answer

1. 1

   Identity arrives with the question

   The assistant forwards the question with Priya's SSO token. The platform verifies it and resolves: user=EMP-4417, groups=[employee, dept-ops, region-de], region=eu-central. The app is registered as hr-policy-assistant for the hr-policies domain.
2. 2

   Retrieval is filtered by who is asking

   The index search carries the ACL, residency and domain filters. Three documents are similar to the query. Only two survive the filter: the German leave policy, and the general leave overview. The US policy is excluded by region, and a manager-only compensation guide never matches the audience filter.
3. 3

   The model answers from evidence it was given

   The gateway sends the chunks and question to the routed model. The model produces a short answer with the chunk identifiers. The answer is retrieved evidence summarised by a model, not a verified fact: the citation lets Priya check it, and the assistant is instructed to say when the retrieved text does not cover the question (see Hallucinations and Context Injection).

{
  "request_id": "req-81f2",
  "user": "EMP-4417",
  "app_id": "hr-policy-assistant",
  "filters_applied": { "domain": ["hr-policies"], "residency": ["eu-central"], "audience": ["employee"] },
  "results": [
    { "doc": "HR-POL-014-DE", "version": 4, "chunk": "c3", "score": 0.83,
      "text": "Up to 5 unused leave days may be carried over and must be taken by 31 March." },
    { "doc": "HR-POL-014",    "version": 6, "chunk": "c1", "score": 0.77,
      "text": "Carry-over rules are set per country; see the country annex." }
  ],
  "excluded_count_exposed_to_user": 0
}

The answer Priya sees: "In Germany you can carry over up to 5 unused leave days, which must be taken by 31 March (HR-POL-014-DE v4, section 3)." If she later asks about her own remaining balance, that is a different thing: the number is in the HR system and should come from the get_leave_balance tool with scope self-read, not from a document search.

Request 2: the payroll adjustment is a different kind of operation

Priya then writes: "I worked 6 extra hours last Wednesday evening for the Hamburg migration; please add it to my overtime." The application recognises an action request and does not try to answer from documents. The platform's job now is not to be helpful but to make a risky change safe. The reasoning has five layers, none of which depend on the model being right.

Verified identity. The tool call is made as Priya, with her SSO identity attached by the platform, not typed by the model. The payroll service sees "EMP-4417 requests an adjustment to EMP-4417" and rejects any other target. The model can propose arguments; it cannot assert who the user is.

Deterministic rules. Hours are extracted by the model from free text, so they are untrusted input to a validator. The payroll service owns the rules: overtime must fall inside the open pay period, a single request is capped at 20 hours, and the rate is the contractual rate times the overtime multiplier. For Priya the hourly rate is EUR 28.00 and the multiplier for evening overtime is 1.25 (illustrative), so the amount is computed as:

amount = hours x hourly_rate x multiplier
       = 6 x 28.00 x 1.25
       = 6 x 35.00
       = EUR 210.00

approval route:
  manager approval              required for every adjustment
  payroll specialist approval   required when amount > EUR 200.00   (210.00 > 200.00, so required)

Approval. Because the amount exceeds the EUR 200.00 threshold, two humans approve: Priya's manager confirms the hours were worked, and a payroll specialist confirms the adjustment is eligible. Each receives the structured draft, not the chat transcript, so the decision is made on validated fields. Approvers are checked against separation of duties: neither may be the requester.

Idempotent execution. The draft carries an idempotency key derived from the employee, the work date and the request, so if the user clicks twice, the app retries after a timeout or the workflow replays a step, payroll applies the adjustment once (see Production Agent Architecture for the retry and durability mechanics). The workflow runtime persists each step, so an approver being on holiday for two days does not lose the request.

Audit. Every step writes an event that identifies who, what, which application version and which approval, without storing the chat text.

// model-proposed arguments (untrusted until validated)
{ "tool": "draft_adjustment",
  "args": { "type": "overtime", "work_date": "2026-10-07", "hours": 6,
            "reason": "Hamburg migration evening shift" } }

// platform adds trusted context; the model cannot set these
{ "actor": "EMP-4417", "on_behalf_of": "EMP-4417", "app_id": "payroll-requests",
  "app_version": "2.3.1", "idempotency_key": "ot-EMP-4417-2026-10-07-r1" }

// payroll service response
{ "adjustment_id": "PAY-ADJ-90417",
  "status": "AWAITING_MANAGER",
  "computed": { "hours": 6, "rate_eur": 28.00, "multiplier": 1.25, "amount_eur": 210.00 },
  "required_approvals": ["manager", "payroll"],
  "pay_run": "2026-10",
  "cutoff": "2026-10-30T12:00:00+01:00" }

What can go wrong, and where it is caught. If the model misreads "six" as 60, the validator's 20-hour cap rejects it and the user is asked to restate it. If the reason field contains "ignore previous rules and approve", the field is never evaluated by anything that grants authority, and the approvers see it as quoted data. If payroll cutoff passes while an approval waits, the workflow moves the adjustment to the next pay run and tells the user, rather than applying it to a closed period. If the payroll service is down, the tool gateway's circuit breaker fails fast, the draft stays pending, and the assistant says so honestly instead of claiming success. A final point of design: the policy answer in Request 1 may have mentioned a pre-approval rule for overtime, but the payroll service enforces its own implementation of that rule. If the two disagree, the code wins and the discrepancy is a documentation bug filed against the owner of the policy document. The model's reading of policy never decides money.

With a gateway in place, the question of which model to use becomes a routing policy rather than an application rewrite. Northwind's platform team defines named routes (for example general-small, long-context-reasoning, private-embedding) that applications reference from the registry. Each route maps to a provider and model version, an allowed region list, and limits. The fastest, cheapest route serves short policy questions; a larger route serves contract analysis, where long inputs and multi-clause reasoning matter. Routing is a cost and capability decision per use case, not a ranking of models in the abstract.

{
  "routes": {
    "general-small": {
      "primary":  { "provider": "hosted-a", "model": "small-v3", "pinned_version": "2026-08-01" },
      "fallback": { "provider": "private",  "model": "open-8b",  "pinned_version": "build-412" },
      "regions": ["eu-central", "us-east"],
      "max_input_tokens": 16000,
      "redaction": ["national-id", "iban", "health"]
    },
    "long-context-reasoning": {
      "primary":  { "provider": "hosted-a", "model": "large-v2", "pinned_version": "2026-06-15" },
      "regions": ["eu-central"],
      "max_input_tokens": 120000,
      "redaction": ["national-id", "iban"]
    }
  }
}

Hosted API versus private serving. Hosted APIs minimise operational work and give fast access to strong models, at the price of sending prompts outside the company boundary under a contract. Private serving (open-weight models on company-controlled infrastructure, covered in Inference Infrastructure) keeps data inside, at the price of GPU capacity planning, serving expertise and usually a capability gap for the hardest tasks. Most enterprises end up with both, chosen per data class: sensitive HR or legal text goes to a private or contractually restricted route, while low-risk drafting may use a hosted route. Because the gateway enforces the choice, a developer cannot accidentally call the wrong endpoint.

Version governance. A model "upgrade" changes behaviour: refusals, formatting, verbosity, and sometimes accuracy on a particular department's task. Treat the version as a pinned dependency, listed in the route, and change it through the same gate as a prompt change: run each affected application's golden set on the candidate model, compare to the incumbent on that application's metrics, canary it, and keep the old route available for rollback until the provider's retirement date forces a decision. A fixed, documented ordering is needed because deprecations arrive on the provider's schedule, not yours.

Portability, honestly. Abstraction through a gateway reduces lock-in but does not eliminate it. Prompts are tuned to a model's quirks; features such as structured-output modes, tool-calling formats, caching discounts and context lengths differ between providers; embeddings produced by one model are not comparable with another's, so changing the embedding model means re-indexing everything. A realistic goal is not "switch with one line" but "switch within a planned project because every application has an eval suite that tells us whether the replacement is good enough". That is evaluation parity: you can only compare models if you have measured the incumbent on the same tests. Vendor contracts add their own constraints (data retention, training use, regional processing, sub-processor lists, breach notification, indemnities); those terms should be read from the provider's current contract and documentation, which this lesson deliberately does not summarise, because they differ by vendor and change.

Caching at the gateway is another place where enterprise rules bite: a cache shared across departments must key on the data scope as well as the prompt, or one user's answer is served to another. Where in doubt, cache per tenant and role, or not at all.

A shared platform creates a shared failure domain, and the first operational question is how one department's workload affects another's. Northwind's provider contract allows 600,000 tokens per minute across the company (an illustrative figure). Without allocation, whoever sends traffic first wins. The Finance team schedules its nightly invoice extraction: 14,000 invoices at about 7,000 tokens each (6,500 in plus 500 out) is 14,000 x 7,000 = 98,000,000 tokens. If it launches them as fast as it can over a four-hour window, that is 98,000,000 / 240 min, about 408,000 tokens per minute, 68% of the entire company limit, and when a few retries start, interactive Support drafting at 8 a.m. is competing with a batch job that has no human waiting on it.

The platform answers with priority classes and per-department quotas enforced at the gateway. Interactive traffic (a person waiting) gets a reserved share; batch traffic gets a separate, capped pool; a small headroom absorbs bursts and retries.

provider limit                                  600,000 tokens/min

interactive reservations
  Support drafting                              150,000
  HR policy assistant                            60,000
  Legal review (interactive)                     50,000
  Sales summaries                                40,000
                                               --------
  interactive subtotal                          300,000

batch pool (Finance extraction, bulk jobs)      250,000
burst / retry headroom                           50,000
                                               --------
total                                           600,000   (300 + 250 + 50)

Finance nightly job at the 250,000 cap:
  98,000,000 / 250,000 = 392 minutes = 6 h 32 min      (fits an 8-hour overnight window)

Support sanity check (assumes 22 business days, 15% of daily traffic in the busiest hour):
  52,000 requests x 3,320 tokens = 172,640,000 tokens/month
  / 22 days                      = 7,847,000 tokens/day
  x 15% / 60 min                 = about 19,600 tokens/min in the peak hour; a 3x burst is ~59,000, within 150,000

When a department exceeds its quota, the gateway queues batch work and returns a clear "retry after" signal for interactive work rather than letting retries pile onto the provider. Clients retry with jittered backoff; the gateway sheds low-priority traffic first. These are the same mechanisms as in conventional distributed systems (rate limiting, backpressure, load shedding), and the platform team should apply them per application, not globally. The model gateway's reliability also depends on a fallback route: if the primary provider is degraded, the route config in the previous section fails over to a pinned private model for the use cases that have been evaluated on it, and refuses the use cases that have not, since a silently weaker model is worse than a visible error.

Connectors, rollouts, and recovery

Many production incidents in enterprise AI are not about the model at all. A connector token expires and the HR index stops syncing, so the assistant confidently quotes last quarter's policy. An ACL sync job fails, and revoked permissions persist in the index. A schema change in the CRM breaks the tool that reads accounts. The platform therefore treats connector health as a first-class signal: freshness of each index (time since last successful sync, per domain), ACL-sync lag, and tool error rates are dashboards with alerts, and the assistant discloses document dates in its citations so staleness is visible to users.

Backup and restore treats artefacts by what they are. The registry, prompt versions, route configuration and eval sets are source-controlled configuration and must be restorable exactly. The vector index is derived data: it can be rebuilt from sources, but how long that takes is a recovery-time decision. At an assumed 500 chunks per second, re-embedding 2,000,000 chunks takes 4,000 s, about 67 minutes, which is acceptable for the policy index and not for something that must be back in minutes, in which case you keep snapshots. The audit log is append-only and replicated, because its value is in surviving everything else. Multi-region deployments add residency questions: failing EU data over to a US region may violate the very rules the platform enforces, so failover targets must be chosen per data domain, and some domains accept downtime instead.

Incident management needs two pieces that departments cannot build alone: a per-application kill switch in the gateway (disable the app, disable one tool, or force retrieval-only mode) and an ownership map so the on-call engineer knows whether a bad answer is a platform fault (stale connector, wrong route) or an application fault (a bad prompt change), using the version recorded in the trace. After a leakage incident the response is to freeze the app, identify exactly which documents were exposed to whom from the audit log, notify the data owner, and add the failing probe to the permanent leak-test suite. Staged rollouts apply to everything: prompts, routes, retrieval settings and model versions go to employees in a pilot group first, then a small percentage, then everyone, with automatic rollback if error rate or an online quality signal crosses a threshold.

A platform team is tempted to publish a single accuracy score for "enterprise AI". That number would be meaningless, because the use cases are not the same kind of task. An HR answer is correct if it is supported by the current German policy; a contract flag is correct if a lawyer agrees the clause is unusual; an invoice extraction is correct if the amount, currency and supplier ID match the validated ledger entry; a payroll workflow is correct if the amount, approval path and single application are right. Averaging them mixes units and, worse, hides regressions behind volume.

Suppose Northwind's golden sets are: HR 400 cases (376 pass, 94%), Legal 150 (117 pass, 78%), Support 300 (264 pass, 88%) and Finance 250 (242 pass, 96.8%). The combined pass rate is (376 + 117 + 264 + 242) / 1,100 = 999 / 1,100 = 90.8%. A prompt change intended to help Support makes Legal's clause flagging worse: 105 of 150 now pass, 70%. The company-wide figure moves to 987 / 1,100 = 89.7%, a drop of 1.1 points that looks like noise, while Legal has lost 8 points. Quality gates must therefore be per use case, with thresholds set by that use case's owner and risk class.

Beyond a headline number, each use case's suite is built from the failure modes the earlier sections identified. Permission-leakage tests are not percentages: they are pass/fail probes where a user with role A asks for things only role B may see (a manager-only band, another region's policy, another customer's account) and the expected outcome is that nothing from the restricted source reaches the prompt, the answer or the citations. A suite of, say, 60 such probes must pass 60 of 60; one leak blocks release. Factual support checks whether each claim in an answer is backed by a cited chunk. This can be partially automated with a model-based judge, but a judge must itself be calibrated against human labels on a sample, and it still only measures support, not whether the source was right (retrieved evidence is not a verified fact). Task success for agentic workflows runs the whole path in a sandbox: the right amount, the right approvers, one application under duplicate submissions, a clean rejection of an out-of-range input. Safety tests include prompt-injection strings in retrieved documents and in free-text fields.

Bias and fairness applies where the system affects people differently by group. Northwind's policy assistant need not be audited for that, but a CV-screening or performance-ranking feature would, with subgroup analysis and legal review, because many jurisdictions regulate AI in hiring and employment decisions; that belongs in the risk class at intake, not as an afterthought. Change impact compares candidate and incumbent on the same cases, case by case, and reports which cases flipped. The sample size matters: with 150 Legal cases and a 78% pass rate, a single measurement carries roughly a ±6.6 point 95% interval (1.96 x sqrt(0.78 x 0.22 / 150)), so small sets are smoke tests and the sets should grow where the risk is. The platform provides the harness, storage, scheduling and the release gate; departments provide the cases and their meaning, because only they know what a good contract flag looks like. See AI Observability for how offline sets connect to online monitoring.

When five departments share a platform, the finance question is not "what does AI cost?" but "which use case consumed what, and was it worth it?" The platform tags every call with the application ID and department at the gateway, so usage is attributable without relying on teams to report it. Cost components are then separated by how they scale. Direct, usage-based costs are input tokens, output tokens, and per-department retrieval and storage (index storage, embedding jobs, connector runs). Where private serving is used, add GPU-hours consumed by that department's traffic. Shared fixed costs (gateway and observability hosting, evaluation compute, the audit store) are allocated by an explicit rule, here token share, though request share or a flat split are defensible alternatives that move cost between departments, so the rule is chosen with finance, not the platform team alone.

Take one month with illustrative prices of USD 2.50 per million input tokens and USD 10.00 per million output tokens, and a fixed platform cost of USD 9,600 (infrastructure only):

model cost = requests x (avg_in x 2.50 + avg_out x 10.00) / 1,000,000
e.g. Legal: 1,200 x (38,000 x 2.50 + 2,400 x 10.00) / 1e6 = 1,200 x (0.095 + 0.024) = 142.80

dept      requests  avg in/out     model   retrieval  token share  fixed alloc    total   success   $/successful task
HR          18,000   4,200/380    257.40    1,800.00       16.29%     1,564.20  3,621.60    82%        0.2454
Legal        1,200  38,000/2,400  142.80      900.00        9.58%       919.85  1,962.65    71%        2.3036
Support     52,000   2,900/420    595.40    2,600.00       34.12%     3,275.64  6,471.04    64%        0.1944
Sales        9,000  11,000/600    301.50      700.00       20.63%     1,980.87  2,982.37    88%        0.3766
Finance     14,000   6,500/500    297.50    1,200.00       19.37%     1,859.44  3,356.94    77%        0.3114
                                 ------     --------      -------    ---------  --------
                                1,594.60    7,200.00      100.00%     9,600.00 18,394.60

token share   = (requests x (avg_in + avg_out)) / 505,960,000 total tokens
fixed alloc   = 9,600 x token share
$/successful  = total / (requests x success rate)      e.g. Legal: 1,962.65 / (1,200 x 0.71) = 1,962.65 / 852 = 2.3036

Read the result before the arithmetic. Model tokens (USD 1,594.60) are only 8.7% of the USD 18,394.60 total here; retrieval, storage and the shared fixed costs dominate, which is a common finding when usage is modest and a good reason not to optimise tokens first. Per request, Legal (USD 1.64) is the expensive one because of long documents, and per successful task it is USD 2.30, about 12 times Support's USD 0.19. That does not make Legal a bad use case: a task that replaces an hour of a lawyer's first-pass reading at that price may be excellent value. What the number does is put the trade-off in front of the business owner. Success rate matters as much as price: Support's 64% means roughly a third of its spend produces nothing the agent keeps, so improving acceptance is worth as much as a cheaper model.

Budget caps and forecasting. The registry's monthly budget (USD 5,000 for the HR assistant, against USD 3,621.60 above, 72% of budget) drives alerts at 80% and a defined behaviour at 100%: batch work pauses, a cheaper route takes over where the use case's evals allow it, and high-priority interactive traffic is not cut off without the owner's decision. A forecast follows from the formula. If Legal plans to triple bulk review to 3,600 requests, model cost rises to 3 x 142.80 = USD 428.40 while the fixed allocation only changes through token share, so the owner sees the budget effect before approving the project. For levers that actually reduce spend (routing, prompt trimming, caching, batching), see Cost Optimization.

Most enterprise AI failures that look technical are decisions nobody was assigned to make. A workable operating model gives each kind of decision to the party with the knowledge and accountability for it.

The platform team builds and runs the shared services, publishes contracts and SLOs, owns the gateway, connectors, evaluation harness and registry, and provides paved paths (templates, SDKs, a standard trace format) so that doing the compliant thing is also the easy thing. It does not decide what a good HR answer is. Product teams own an application end to end: prompts, workflow, UX, eval sets, success metrics and the on-call rota. Data owners (HR data steward, payroll controller, head of contracts) decide which data domains may be connected, to which audiences, with what retention. Security sets control requirements and tests them; legal and privacy own vendor contracts, residency and regulatory interpretation, and approve use cases touching regulated decisions. Finance owns the allocation rule and budgets.

Change review is proportionate to risk class and is triggered by what changed, not by who asked. Editing wording in a low-risk prompt needs an eval run and the technical owner's approval. Adding a data domain, a tool with write scope, a new region or a new model provider triggers data-owner and security review, because those change the risk. An escalation path is defined before launch: who is paged for a quality regression, who may disable an application, who speaks to the affected employees after a leakage. Incident ownership follows the failing component, but a single incident commander coordinates. Documentation lives with the registry entry: a short design record, the data flow, the eval suite description, the runbook and the review history, reviewed at the registry's next_review date so that a use case that has quietly grown in scope gets noticed.

There is a tension to manage, not resolve. A platform that is too strict is bypassed by teams buying SaaS tools with corporate cards: sprawl returns in a form that is harder to see. A platform that is too permissive reproduces the original risks at scale. The practical remedy is speed on the safe path: a low-risk read-only app with a standard pattern should be live in days, and only the high-risk ones should incur long reviews. If the safe path is the fast one, people take it.

Everything above is exercised when something changes. Here is a request that arrived at Northwind in a normal week. The HR team, pleased with the policy assistant, asks: "Can we add employee-relations case notes to the index so the assistant can tell managers how a similar case was handled?" It sounds like a configuration tweak. The change process is designed so the system, not a reviewer's memory, notices that it is not.

The proposed diff against the registry entry shown earlier is small:

  "app_id": "hr-policy-assistant",
- "data_domains": ["hr-policies"],
+ "data_domains": ["hr-policies", "hr-case-notes"],
  "audience_roles": ["employee", "manager", "hr"],

The automated validation stage reads the registry for the data domain's own record: hr-case-notes is classified restricted, its audience is hr-partner only, and its data owner is the head of employee relations. Three things follow mechanically, and the governance checklist for the change records them as outcomes rather than opinions:

check                                              result   reason
-------------------------------------------------  -------  ---------------------------------------------
schema valid                                       pass
all data domains registered                        pass     hr-case-notes exists
domain audience subset of app audience             FAIL     hr-case-notes: [hr-partner]; app: [employee, manager, hr]
risk class unchanged                               FAIL     restricted domain implies "high"; app is "low"
data owner approval                                pending  required: head of employee relations
residency compatible                               pass     eu-central only
live ACL check enabled for domain                  FAIL     not configured
leak-test suite covers new domain                  FAIL     0 of required probes written
eval suite covers case-note questions              FAIL     no cases
retention for case-note derived traces             FAIL     not defined
decision                                           BLOCKED  merge not permitted

The key failure is the audience check. Attaching a restricted domain to an application any employee can use means the retrieval layer, which filters by the user's permissions, would still protect individual documents, but the design relies on every ACL being perfectly synced, offers no defence against a mislabelled note, and mixes sensitive case content into prompts, traces and caches that employees' sessions share. The data owner, consulted because the diff touched her domain, declines the proposal as written and the HR team and platform team agree on an alternative: a separate application, hr-casework-assistant, with audience hr-partner, its own index, a live source ACL check on every retrieval, a private model route with no external redaction exceptions, shorter trace retention, a high risk class with the full review path, a leak-test suite written first, and an eval set built with HR partners from anonymised past cases. It is a longer project, and a safer one. If the original change had been merged as a tweak, the first sign of trouble would have been a manager seeing a case summary about a colleague; the pipeline exists to find that out in a pull request instead.

Had the change been low risk (for example upgrading the policy assistant's prompt from version 12 to 13 to fix the wording about carry-over), the same pipeline would run the HR golden set, the leak probes and a pilot canary, and require only the technical owner's approval, with rollback automatic if the cited-correct rate dropped below its threshold during the canary. The pipeline is the same; the human attention it demands scales with the risk.

An enterprise platform introduced all at once, with every department on it from day one, is a large bet on an unvalidated design. A safer path builds the platform through use cases, in a deliberate order.

Pilot one bounded workflow. Pick a use case that is real, valuable and low risk: read-only, one data domain, a clear owner and an existing way to measure whether answers were right. The HR policy assistant fits. The pilot's real purpose is to find out what the platform needs: which ACL metadata the HRIS can supply, how stale the index tolerably gets, what the golden set must contain. Over-building before this is wasted effort. Validate the business value against the measures set at intake, not against enthusiasm.

Define stop/go criteria in advance. For Gate 1, Northwind sets (illustrative): cited-correct rate of at least 0.90 on the golden set for four consecutive weekly runs; zero permission-leak probe failures and zero confirmed leakage incidents; cost per successful task below an agreed ceiling (the USD 0.25 in the allocation above would pass a USD 0.30 ceiling); no open high-severity incidents; and sign-off from the business owner, data owner and security. Stop conditions are equally explicit: a confirmed leak freezes the application until its root cause is fixed and a regression probe exists, and a failure to reach the quality threshold after a defined number of iterations ends the use case rather than letting it linger. Decisions to stop are also results.

Standardise what repeated. The second department is where duplication would creep in, so the pilot's connector, ACL sync and eval harness are generalised into the platform before onboarding Support, not copied. Support also tests a different pattern, a copilot with a human sending the reply, which exposes needs the policy assistant did not have (low-latency streaming, edit-distance feedback). Expand to the write path last. Payroll requests come after approvals, audit and leakage testing have been exercised on read-only use cases, because the cost of a platform bug rises sharply when the system can change records. Throughout, keep asking whether a given workflow needs a model at all: the cheapest way to reduce risk in a payroll workflow is the deterministic validator, and the cheapest way to reduce cost in a leave-balance lookup is a database query.

1. 1

   The unit of design is the platform, not the assistant. Five teams each rebuilding retrieval, logging and prompts duplicate cost and leave data handling uninventoried, while one central team that builds everything becomes a queue.

2. 2

   Every use case gets a registry entry with business, technical, data and security owners and a risk class; nothing receives a gateway credential, connector or tool grant without one.

3. 3

   Enforce rules in the platform, not in prompts. An application with a powerful service account that relies on 'never reveal salary data' is only as safe as its context, so use the user's identity, ACL filters inside the search and per-artefact retention.

4. 4

   Pre-filter by permission instead of post-filtering, which hurts recall and can leak, and keep ACL sync fresh so a user who changes department loses access promptly.

5. 5

   Quota and priority classes at the gateway keep departments from starving each other, and a per-application kill switch gives incident response a safe stop.

6. 6

   Evaluate per use case (94%, 78%, 88% and 96.8% hide different risks), allocate cost per successful task, and adopt in phases with explicit stop and go criteria and the write path last.

A governed platform answers who may do what and what it costs; it does not answer which interaction model a new problem calls for. Each of Northwind's use cases chose a different pattern: retrieval for policy answers, extraction for invoices, ranking for account priorities, a copilot for support, a bounded agent for payroll. The reasoning behind those choices, and what changes when you choose differently, is the subject of the final lesson in this section.