Intelligent Document Processing (IDP)

Watch a PDF invoice travel through OCR, field extraction, and classification to become a structured, searchable record, then see the same pattern power contracts, resumes, and other enterprise document types.

Intelligent document processing (IDP), also called document AI or document intelligence, is the pipeline that combines OCR with layout-aware extraction, validation and human review to turn scans and PDFs into structured records. OCR only reads characters; IDP also recovers layout, extracts typed fields with evidence, and checks them before anything is trusted. Document intelligence turns files that were made for human eyes (scans, PDFs, photographs, forms, contracts) into records that software can act on. It does this in layers: the system first makes the file legible (rendering, rotation, a text layer or OCR), then recovers its geometry (where each word sits, what is a table, in which order a person would read it), then extracts typed fields with a pointer back to the evidence, and finally runs deterministic validation and, where needed, human review before anything reaches an accounting or contract system. Each layer produces its own artefact and its own failure modes. The record is trustworthy because of the last two layers, not because the model that read the page was clever.

This chapter follows one invoice through the whole pipeline and then widens out to tables, contracts, reliability and evaluation. The model-level foundations are covered elsewhere: how a vision model reads pixels in Vision Models and general visual understanding in Image-to-Text Understanding, how a model can be made to emit schema-shaped JSON in Structured Outputs, and how the same parsing step feeds search in RAG ingestion (Chunking). Here the focus is the system around those pieces, and how a messy page becomes a record you can book.

Kaveri Packaging Supplies (an invented vendor; every value in this chapter is illustrative) emails a PDF to the accounts-payable inbox of a mid-sized manufacturer. It is a phone scan of a printed invoice, and the page is rotated: it was photographed in landscape, so the content runs sideways. The invoice has a letterhead, an invoice number, a date, a two-line item table, a subtotal, tax and a total. Someone at the receiving dock has written a note in ink in the margin. And the printed subtotal is wrong: the two lines add up to ₹900, but the subtotal box says ₹950 (a clerk at the vendor typed over an older figure, or a scan artefact turned a 0 into a 5; we cannot tell from the pixels alone, and that is the point). The CGST and SGST lines follow the Indian GST invoice layout as a teaching example; tax rates and invoice rules differ by country and change over time, so this is not tax guidance.

Running OCR over the raw file and printing the text gives something like this. The system read every character correctly except one, and still handed us nothing we can post to a ledger:

Corrugated carton 5-ply 12x10x8 in   3   200.00   600.00
Subtotal   950.00   Kaveri Packaging Supplies   Invoice No KP/26-27/0873
Packing tape 48mm   2   150.00   300.00
Date 14/10/2026   CGST 9% 81.00   SGST 9% 81.00
Total   1,062.00   rcvd 2 cartons only

Plain text is a stream of characters. A business record is a set of typed claims: this is the vendor, this is the invoice number, this amount is the pre-tax subtotal, these are the line items with quantities and unit prices. Going from the first to the second requires knowing where things were on the page (geometry), what role each piece of text plays (extraction), and whether the claims are consistent with each other and with the world (validation). In the stream above the word "Subtotal" is followed by 950.00, but nothing in the text says that 950.00 contradicts the lines beneath, or that "rcvd 2 cartons only" is a marginal remark by a human rather than a product description. We will resolve all of that in the pipeline. The subtotal conflict ends in a review queue, and the note is preserved as an annotation without being allowed to change the invoice quantity.

"Document AI" bundles several different jobs, and picking the wrong one is the commonest early design mistake. The simplest is searchable text: make a scanned archive findable. Accuracy requirements are modest and a wrong character costs a missed search hit. Next is key-value extraction: the invoice number, the total, the party names. This is where schemas and field-level confidence start to matter. Table extractionrecovers rows and columns, which is much harder than it looks because a table is a two-dimensional relationship that text does not carry. Classification decides what kind of document arrived (invoice, credit note, purchase order, ID card) so that the right schema and validators are chosen. Document comparisonaligns two versions and reports what changed, a task where position-aware text diffing beats naive character diffs. Clause extraction finds named provisions in contracts, and workflow automationuses all of the above to drive a decision such as "pay this invoice".

Three stages are routinely confused, and the architecture separates them. Extraction says what the page contains: "the document states a total of 1,062.00". Interpretation says what it means: "this is a tax invoice in INR, payable on net-30 terms, and the handwritten note suggests short delivery". Business validation asks whether the claim is acceptable for this company: does the vendor exist, is the arithmetic consistent, has this invoice been paid already, does it match the purchase order. A model can do the first well, can propose the second, and cannot be the authority on the third, because the third depends on your database and your policy. If you take only one design principle from this chapter, take this one: each stage outputs a different kind of claim, and the later stages get to overrule the earlier ones.

Everything upstream of OCR is about not trusting the file. A document arrives from an email, a web form or an API; an attacker or a broken scanner can produce anything. The first step is to store the original bytes unchanged in quarantined, access-controlled object storage under a content hash, and only then inspect it. Magic-byte checks decide the real format (a file called invoice.pdf may be a ZIP or an executable). A malware scan runs before any parser touches it, because PDF and Office parsers have a long history of vulnerabilities. Limits protect the pipeline: maximum file size, page count, image dimensions and render time, so that a decompression bomb or a 4,000-page PDF cannot starve other tenants. Encrypted or password-protected files are rejected with a clear reason rather than silently dropped. Parsing and OCR then run in sandboxed workers with no network access to internal systems and with short-lived credentials limited to the one document.

Rendering converts every page to a raster at a fixed resolution (200 DPI is a common starting point, a tunable assumption: higher helps small print but costs time and memory). The raster is what vision models and OCR see, and it is the reference frame for bounding boxes. Text-layer detection is a cheap test with a high payoff. A PDF produced by accounting software already contains exact characters with coordinates; extracting them avoids OCR error entirely. A scan embeds only an image. Some PDFs are hybrids: a scanned page with an OCR layer added by a previous tool, which may be wrong, so the decision is made per page, and a text layer is accepted only if it passes sanity checks (a reasonable ratio of dictionary-like tokens, no strings of replacement characters, geometry that fits inside the page). Otherwise the page falls back to OCR.

Orientation and skew. A phone scan can be rotated by 90° or 180° and skewed by a few degrees. OCR on a sideways page produces plausible-looking garbage, so orientation is classified first (often with a small model, or from OCR confidence at the four rotations) and the page is rotated upright; deskew then straightens small angles from the dominant text-line direction. The rotation must be recorded: the system keeps the transform from raw image to upright page so that a bounding box on the upright page can still be drawn on the original scan if someone asks for it. For a raw landscape raster of width 2,339 px, rotating the content 90° counter-clockwise maps a raw point (x, y) to the upright point (y, 2,339 − x); the raw point (300, 1,000) lands at (1,000, 2,039).

Quality gates. Some pages cannot be rescued: a blurred photograph, a 72 DPI fax, a finger over the total. Measuring sharpness, effective resolution and contrast before OCR lets the system ask for a better upload immediately instead of producing low-confidence fields that a reviewer must squint at later. Language and script detection selects the OCR model and the locale rules used later: the date "03/04/2026" is 3 April in India and 4 March in the United States, and the digit grouping "1,06,200" is lakh-style, not a typo. Ambiguity that cannot be resolved from the document (a date with both parts ≤ 12 and no vendor locale on file) should be flagged, not guessed. Finally, store the original, the rendered pages and every derived artefact in encrypted storage with a retention label attached on day one, since deletion rules (covered in the reliability section) are impossible to honour if derived copies are untracked. Related security controls at the infrastructure level are in AI Security.

OCR has two jobs. Detection finds regions of the image that contain text, as boxes or polygons around lines and words. Recognition converts each cropped region into characters with a per-character or per-word confidence. The result is a list of tokens, each with a string, a bounding box and a score. This is already richer than a text dump, but it is not understanding, because a token list has no notion of columns, tables, headers, or which words belong together.

Coordinates are an interface, and the contract needs a declared convention. Image rasters use a top-left origin with y increasing downwards, measured in pixels. PDF content streams use a bottom-left origin with y increasing upwards, measured in points (1/72 inch). Mixing them is the single most common source of highlights drawn in the wrong place. In the figure the upright A4 page is 595 × 842 pt, which at 200 DPI is a raster of 595 × 200/72 ≈ 1,653 by 842 × 200/72 ≈ 2,339 px (rounded to 1,654 × 2,339 from the exact 8.27 × 11.69 inch size). The total's box on the raster is (x0, y0, x1, y1) = (1180, 2010, 1520, 2070) px. Converting to points divides by the scale 200/72 ≈ 2.778, giving x from 424.8 to 547.2 pt, and the PDF y-range flips: y_bottom = 842 − 2070/2.778 = 96.8 pt and y_top = 842 − 2010/2.778 = 118.4 pt. The same box normalised to the page is (0.713, 0.859, 0.919, 0.885). Storing the box together with the page number and the coordinate system name (px@200dpi,pt, norm) is cheap and prevents a class of bugs that only appear when a second consumer reads the data.

Reading order. Humans read a letterhead, a block of party details, a table row by row, then the footer. OCR emits lines in detection order, which is roughly top-to-bottom but breaks on multi-column pages, side-by-side boxes (the "Bill to" and "Ship to" blocks) and rotated text. That is why our raw output above interleaved a table row with the vendor name. A layout model groups tokens into blocks, assigns them roles (title, paragraph, table, header, footer, figure, handwritten annotation), and orders the blocks. Headers and footers matter more than they look: a page footer "Page 2 of 3" or a repeated letterhead should be recognised as page furniture, not as line items or a vendor change.

Tables are geometry. A table is a grid of cells that may span rows or columns, with a header row that gives meaning to the columns. Table recovery detects the table region, finds the row and column separators (ruling lines where they exist, whitespace alignment where they do not) and assigns each token to a cell. For our invoice that yields a header row (Description, Qty, Rate, Amount) and two data rows, so "3" is a quantity because it sits under Qty, not because it is a small integer. Borderless tables and tables that the camera has curved are where this fails first, which is why table accuracy is evaluated separately from character accuracy.

How layout models differ from text extraction. A text extractor (a PDF library reading the text layer, or OCR) answers "which characters, where". A layout-aware model also answers "which role". There are two common designs. A specialised document model takes the image plus the OCR tokens and their boxes and classifies each token or block, so it is conditioned on position, font size and neighbours. A multimodal LLM (see Multimodal AI Overview) takes the page image directly and answers in language or JSON. The first is cheaper, faster and gives boxes natively. The second generalises to unseen layouts with little setup but returns coordinates less reliably (they are often an additional, separately verified step), may drop a row of a long table, and, being generative, can write a plausible value that is not on the page. That difference drives the architecture comparison later.

Extraction turns blocks into a schema. The schema is a contract between the pipeline and every consumer, so it should be explicit, versioned and typed: dates as ISO 8601 strings, money as a decimal string plus a currency code (never a floating-point number), identifiers as strings (a leading zero is data). The most important design choice is that each field carries its provenance: where on which page the value came from, how sure the system is, and which model produced it. Without provenance a reviewer cannot check a value in five seconds, an auditor cannot reconstruct a decision, and an evaluation job cannot tell whether an error came from OCR, layout or the extractor.

{
  "document_id": "doc_7f3a91",
  "schema_version": "invoice/1.3",
  "pipeline_version": "pl-2026.10.2",
  "pages": [{ "page": 1, "width_px": 1654, "height_px": 2339, "dpi": 200, "rotation_applied_deg": 270 }],
  "fields": {
    "vendor_name": {
      "value": "Kaveri Packaging Supplies", "type": "string",
      "page": 1, "bbox": [120, 140, 900, 230], "bbox_space": "px@200dpi",
      "confidence": 0.98, "source": "ocr+layout", "model": "extractor-v4"
    },
    "invoice_number": {
      "value": "KP/26-27/0873", "type": "string",
      "page": 1, "bbox": [1000, 140, 1540, 230], "bbox_space": "px@200dpi",
      "confidence": 0.99, "source": "ocr+layout", "model": "extractor-v4"
    },
    "invoice_date": {
      "value": "2026-10-14", "type": "date", "raw_text": "14/10/2026", "locale_assumed": "en-IN",
      "page": 1, "bbox": [1000, 250, 1540, 320], "bbox_space": "px@200dpi",
      "confidence": 0.97, "source": "ocr+layout", "model": "extractor-v4"
    },
    "currency": { "value": "INR", "type": "currency", "source": "symbol+vendor_profile", "confidence": 0.99 },
    "subtotal": {
      "value": "950.00", "type": "money",
      "page": 1, "bbox": [1120, 1260, 1540, 1330], "bbox_space": "px@200dpi",
      "confidence": 0.58, "source": "ocr+layout", "model": "extractor-v4"
    },
    "tax_total": {
      "value": "162.00", "type": "money", "tax_breakdown": [
        { "name": "CGST", "rate": "9", "amount": "81.00" },
        { "name": "SGST", "rate": "9", "amount": "81.00" }
      ],
      "page": 1, "bbox": [1120, 1350, 1540, 1420], "bbox_space": "px@200dpi",
      "confidence": 0.96, "source": "ocr+layout", "model": "extractor-v4"
    },
    "total": {
      "value": "1062.00", "type": "money",
      "page": 1, "bbox": [1180, 2010, 1520, 2070], "bbox_space": "px@200dpi",
      "confidence": 0.99, "source": "ocr+layout", "model": "extractor-v4"
    },
    "line_items": [
      { "description": "Corrugated carton 5-ply 12x10x8 in", "quantity": "3", "unit_price": "200.00", "amount": "600.00",
        "page": 1, "row_bbox": [120, 700, 1540, 780], "confidence": 0.95 },
      { "description": "Packing tape 48mm", "quantity": "2", "unit_price": "150.00", "amount": "300.00",
        "page": 1, "row_bbox": [120, 800, 1540, 880], "confidence": 0.96 }
    ],
    "annotations": [
      { "kind": "handwritten", "text": "rcvd 2 cartons only", "page": 1,
        "bbox": [200, 1560, 980, 1700], "bbox_space": "px@200dpi", "confidence": 0.41, "affects_amounts": false }
    ]
  }
}

Read the record as a set of claims. raw_text beside the normalised date preserves what the page literally said so a locale mistake can be diagnosed later. source records whether the value came from the PDF text layer, OCR, or a model's reading of the image; the three have very different error profiles. bbox_space makes the coordinate convention part of the data. The handwritten note is stored as an annotation with affects_amounts: false: the system has decided to preserve the remark and surface it to a person, but has not changed the quantity of the first line from 3 to 2. Reinterpreting a margin note as a correction is a business judgement and belongs to review.

What does confidence mean? It depends entirely on where the number comes from. OCR engines report recognition scores, and a classifier head reports a softmax probability, but a softmax score is not automatically a calibrated probability of being correct. A generative model's own stated certainty is weaker still. Treat any raw score as a ranking signal until you have measured it. The practical method is to run the pipeline on a labelled sample, bucket fields by score, and compute accuracy per bucket. If fields scored 0.9–1.0 are 99.5% correct and fields scored 0.5–0.6 are 55% correct, the score is usable, and a routing threshold can be picked on that data; if the buckets are flat, the score is noise and other signals must be used: agreement between two independent readers (text layer versus OCR), validation failures, and field importance. Thresholds differ by field, because a wrong purchase-order number and a wrong supplier bank account do not have the same cost.

The figure below sweeps a routing threshold over a calibration table to show the trade between review volume and residual error.

For fields the extractor fills with a model that supports schema-constrained output, the schema guarantees the shape (valid JSON, required keys, enum values) and does nothing about truth. A well-formed "total": "1062.00" may still be wrong, which is why the next layer exists. See Structured Outputs for the shape-versus-truth distinction.

The extractor proposes; validation disposes. Validation is ordinary deterministic code that runs after extraction and before anything leaves the pipeline, so it can be unit-tested, versioned and reasoned about independently of the model. It checks different classes of claim. Structural rules: required fields present, date parses and is not in the far future, amounts parse as decimals with at most two fractional digits for INR. Format rules: an Indian GST identifier matches its 15-character pattern (and a checksum, which you should implement from the official specification rather than from memory), a PO number matches the vendor's prefix. Consistency rules: one currency throughout, line amounts equal quantity × unit price, subtotal equals the sum of lines, subtotal plus tax equals total. Referential rules, which need your systems: the vendor exists and is active, the bank account is on file, the PO exists with remaining balance, and the invoice number has not been seen before for this vendor.

Take the worked invoice. Line 1: 3 × ₹200.00 = ₹600.00. Line 2: 2 × ₹150.00 = ₹300.00. The sum of lines is 600 + 300 = ₹900.00. At the illustrative 18% rate the tax is 900 × 0.18 = ₹162.00, which the invoice splits as 9% + 9% = ₹81.00 + ₹81.00 (central and state components). The expected total is 900 + 162 = ₹1,062.00. Now compare with what the page states: tax 162.00 ✓, total 1,062.00 ✓, subtotal 950.00 ✗. Two of the three printed amounts agree exactly with the recomputed figures, and only the subtotal disagrees by ₹50.00. That pattern is informative: the most likely story is a wrong or misread subtotal, not wrong lines. But a pipeline must not rewrite a financial document on a "most likely" story; it records the discrepancy and escalates.

from decimal import Decimal, ROUND_HALF_UP

CENT = Decimal("0.01")

def money(x: str) -> Decimal:
    return Decimal(x).quantize(CENT, rounding=ROUND_HALF_UP)

def validate_totals(lines, printed, tax_rate):
    """lines: [(qty, unit_price)], printed: dict of strings, tax_rate: Decimal percent."""
    issues = []
    line_amounts = [money(str(q)) * money(p) for q, p in lines]
    subtotal = sum(line_amounts, Decimal("0.00"))
    tax = (subtotal * tax_rate / Decimal(100)).quantize(CENT, rounding=ROUND_HALF_UP)
    total = subtotal + tax
    if money(printed["subtotal"]) != subtotal:
        issues.append(("subtotal_mismatch", str(subtotal), printed["subtotal"]))
    if money(printed["tax"]) != tax:
        issues.append(("tax_mismatch", str(tax), printed["tax"]))
    if money(printed["total"]) != total:
        issues.append(("total_mismatch", str(total), printed["total"]))
    return subtotal, tax, total, issues

result = validate_totals(
    lines=[(3, "200.00"), (2, "150.00")],
    printed={"subtotal": "950.00", "tax": "162.00", "total": "1062.00"},
    tax_rate=Decimal("18"),
)
print(result)
# (Decimal('900.00'), Decimal('162.00'), Decimal('1062.00'),
#  [('subtotal_mismatch', '900.00', '950.00')])

Rounding deserves a sentence of its own. Many invoices round tax per line; others round at the total. If line tax is rounded to the nearest paisa and then summed, the result can differ from rounding the total, by a paisa or two. A strict equality check will reject valid documents; an overly generous tolerance will accept real errors. Pick the rule the vendor's jurisdiction uses, apply a tolerance of at most a few paise or a tiny relative value, and make that tolerance configuration with an owner. Likewise a duplicate check should key on normalised (vendor id, invoice number, total, date) rather than on the file hash. The same invoice forwarded twice with different PDF bytes is the typical duplicate; byte-identical resubmission is the easy case.

What validation does with failure. A failed rule is not a binary reject. Each rule has a severity (block, warn) and a suggested remedy, and the outcome is attached to the field and shown to the reviewer. Here the output is a record with status: "needs_review", the single issue subtotal_mismatch (expected 900.00, printed 950.00), the proposed correction, and a pointer to the highlighted crop. Notice what the layer deliberately does not do: it does not ask an LLM to "fix the number". Auto-correction is allowed only where policy permits (for example normalising a date format or trimming whitespace), and each correction is logged. Anything that changes an amount waits for a person. The same philosophy of verifying proposed outputs before acting on them appears in Human-in-the-Loop & Guardrails.

The invoice above had two rows on one page. Real freight statements, bills of materials and bank statements run for pages. Consider a second, separate example: a statement from Sundaram Freight Lines with nine line items. Five rows fit on page 1 (₹4,200, ₹3,150, ₹2,800, ₹5,600 and ₹2,700, summing to ₹18,450) and the page footer reads "Carried forward 18,450.00". Page 2 starts with "Brought forward 18,450.00", does not repeat the column header, and lists four more rows (₹1,800, ₹2,400, ₹3,300 and ₹1,800, summing to ₹9,300). The final block reads "Sub-total 27,750.00; GST 18% 4,995.00; Total 32,745.00". Check: 18,450 + 9,300 = 27,750; 27,750 × 0.18 = 4,995; 27,750 + 4,995 = 32,745.

Several things go wrong here that a one-page invoice never shows. The second page has no header, so its columns must be inherited from page 1; the extractor needs table continuity across pages, matched by column alignment and position near the top of the page, not by text. "Carried forward" and "Brought forward" rows are page furniture that look like line items: if they are extracted as rows, the pipeline counts 18,450 twice and the reconciliation fails by exactly that amount, which is a fast diagnostic when you see it. A row can also straddle a page break (a long description wrapped onto the next page) and has to be merged with its first half. Merged cells and footnotes add further structure: a footnote marker "*" after an amount ties it to a note at the page bottom ("includes fuel surcharge"), an association that exists only in the geometry and the marker, and a merged "Route" cell that spans three rows applies to all three. Row-to-item association is therefore an explicit step that produces a typed line-item list, rather than something the downstream code infers from strings.

The reconciliation step then checks the page-level and document-level totals: each page's rows should sum to the carried-forward figure (18,450 for page 1), the sums across pages should equal the stated sub-total (27,750), and the tax and grand total should follow. When the numbers do not tie, the discrepancy narrows the search: a gap of one row's amount points to a dropped or duplicated row, a gap of one digit position points to an OCR misread of that amount, and a gap equal to the brought-forward figure points to the double-count above.

This is the reason that blindly applying text chunking to tabular documents destroys information. The chunking strategies used for retrieval cut text by token budget or paragraph. Applied to flattened OCR text of the freight statement, a 150-token window may end like this:

--- chunk 7 (ends mid-table, page 1) ---
Description        Weight kg   Amount
Mumbai-Pune        1,200       4,200.00
Pune-Nashik          900       3,150.00
Nashik-Surat         800       2,800.00
Surat-Vapi         1,900

--- chunk 8 (starts mid-row, continues onto page 2) ---
5,600.00
Vapi-Mumbai        1,100       2,700.00
Carried forward 18,450.00
Brought forward 18,450.00
Surat-Mumbai       1,400       1,800.00

A retriever asked "what did the Surat–Mumbai leg cost?" gets a chunk where nothing says that 1,800.00 is an amount and not a weight, and a question about total freight cannot be answered by any single chunk. Documents that will be searched should be parsed with layout first and chunked by structure: a table becomes either one chunk (if it is small), or a series of chunks each prefixed with the header row and the table title, plus a separate structured record of the whole table. The extraction pipeline and the retrieval pipeline should share the same parsed layout, and numeric questions ("sum this column") should be answered by code over the structured table rather than by a model reading chunks, which is both cheaper and verifiable.

No pipeline reaches 100% field accuracy, so the architecture decides who sees the remainder and how fast they can resolve it. The review queue is a first-class product surface, not an exception handler. Routing follows three signals: low field confidence (below the field's calibrated threshold), a validation failure, and field criticality (bank details, total and tax identifiers have lower tolerance than a purchase description). Additional rules route by context: a first invoice from a new vendor, a total above an approval limit, or a document class the system has not seen before is reviewed regardless of its scores. In our example the invoice is routed because the subtotal field conflicts with its recomputed value and has confidence 0.58, and because the handwritten note, at 0.41, might change what was received.

The reviewer does not retype the document. The screen shows only the flagged fields, each beside a zoomed crop of the source region cut from the stored bounding box (this is why provenance was in the schema), with the validator's message and the system's proposed value. The reviewer chooses Accept, Correct or Reject. Here the clerk compares the crop with the line items, sees that the printed 950.00 is inconsistent, and sets the subtotal to 900.00, adding a comment. For the margin note, she keeps it as an annotation and flags the receipt for the buyer to reconcile with the goods-receipt record. She has not amended the invoice amount; that follow-up belongs to procurement. Because the system already knows which fields are in doubt, review effort is proportional to uncertainty, not to the length of the document.

{
  "event_id": "evt_01JQ",
  "document_id": "doc_7f3a91",
  "field": "subtotal",
  "model_value": "950.00",
  "model_confidence": 0.58,
  "validator": { "rule": "subtotal_mismatch", "expected": "900.00" },
  "final_value": "900.00",
  "action": "corrected",
  "reviewer_id": "user_4412",
  "reviewed_at": "2026-10-15T09:41:07Z",
  "time_spent_ms": 21000,
  "pipeline_version": "pl-2026.10.2",
  "schema_version": "invoice/1.3",
  "comment": "Printed subtotal does not match lines; vendor to be informed",
  "training_use": "not_consented"
}

Correction capture serves three audiences. Auditors need the immutable history (append-only events, never updates in place) to show who changed a financial figure and why. Operations needs the metrics that fall out of it: review rate, time per document, and the fields most often corrected, which direct engineering effort to the weakest extractor or the most confusing vendor layout. Machine learning needs the corrected values as labelled examples, and here care is required. Invoices and contracts contain personal and commercially confidential data. Feeding them automatically into model training is not a default: it needs a lawful basis and customer consent or contract terms, tenant isolation (one customer's documents must not teach a model that serves another), redaction of identifiers where possible, and an offline evaluation gate before any retrained model replaces the current one. The safe loop is: corrections update per-vendor rules and profiles (cheap, scoped to one tenant) immediately, and feed periodic, consented, curated training or evaluation sets that go through the same evaluation before release. The training_use flag on the event is the mechanism that makes this enforceable rather than aspirational.

There is also a human factor. Reviewers shown a model's suggestion tend to accept it (automation bias), so a field that reads "950.00, confidence 0.58, conflicts with 900.00" is more useful than a bare value, and a sampled share of high-confidence documents should also be reviewed blindly to measure how many errors escape the thresholds. This quality-audit sample is the only way to learn whether the automation is quietly degrading. The general design considerations for human oversight are in Human-in-the-Loop & Guardrails.

Contracts differ from invoices in the shape of their uncertainty. An invoice field is a short, typed value in a predictable place. A contract provision is a span of prose whose meaning depends on definitions elsewhere in the document. A representative extraction job for a services agreement asks for the parties (including their defined short names, such as "Customer" and "Supplier"), the effective date and term, the renewal and termination clauses with notice periods, payment terms, liability caps, governing law, and the obligations of each party. For each, the output should be a short normalised statement plus the exact quotation and its page and bounding box, as with invoice fields, so a lawyer can read the source in one click.

The hard cases are semantic. An effective date may be "the date of last signature", which is not printed anywhere as a date and depends on signature blocks on the last page. A notice period of "thirty (30) days" may be modified by "except in the case of material breach, where termination is immediate", and an exception may sit three clauses later, or in an amendment that supersedes the original. Cross-references ("subject to Section 9.2") must be resolved by following them, which in practice means building an index of section numbers and linking references to targets, then presenting the referenced text beside the clause. Defined terms change meaning: "Business Day" or "Confidential Information" might be defined narrowly in Schedule 1. A flat chunk of the middle of the document loses all of that context, while a structure-aware parse (sections, numbering, definitions, schedules) preserves it. Scanned contracts add handwritten initials, struck-through text and stamps, any of which can change the reading and none of which a text-only pipeline sees.

Extraction is not a legal determination. A system that outputs "termination for convenience: 30 days' notice" has reported what a clause appears to say. It has not decided whether the clause is enforceable, whether an amendment overrides it, whether it complies with local law, or whether it is acceptable for your organisation. Designs that respect this boundary present results as candidate findings with evidence: each has a quotation, a confidence, a link to cross-referenced text, and a status such as "unreviewed", "confirmed by legal" or "rejected". They support prioritising a lawyer's attention (flag any clause that deviates from the company's standard playbook) rather than replacing the lawyer's reading. Where a missing clause is itself a risk (no liability cap at all), the system should say "not found" together with the places it looked, because absence of evidence is the failure mode most likely to be misread as reassurance. Model errors here are also not random: a generative reader may paraphrase a clause and quietly drop a carve-out, which is exactly why the verbatim quotation, rather than the summary, is the source of truth in the UI. See Hallucinations for why fluent summaries need independent verification.

A document pipeline is a long-running, multi-stage job on unreliable inputs, so it is built as an asynchronous workflow rather than a single request. The upload API responds quickly with a document id and a status URL; a durable queue (see Model Serving for the serving side) feeds workers; each stage records its state and artefacts. Work is split at page level: OCR on 40 pages is 40 independent tasks that can run in parallel, fail separately and be retried separately, and the document completes when its last page does.

Retries and idempotency. OCR services time out, workers are killed, queues deliver a message twice. Each page task therefore has a deterministic idempotency key made from the document content hash, page number, stage and pipeline version, for instance sha256(doc) + ":p3:ocr:pl-2026.10.2". A worker that finishes writes the result with an upsert on that key, so a duplicate run overwrites the identical result instead of creating a second record, and a retry after a crash that happened after the write finds the result and exits. Retries use bounded exponential backoff with jitter (see AI Observability for what to watch, and the System Design lessons on Retry & Backoff and Idempotency for the underlying patterns). Failures are classified: transient errors (timeout, rate limit) retry; deterministic ones (corrupt page, unsupported font) do not, because a thousand retries of a malformed PDF page will not repair it and only burn budget.

Partial results. If page 3 of a 5-page contract fails OCR after the retry budget, the pipeline should not mark the whole document failed, and must not mark it complete either. It keeps pages 1, 2, 4 and 5, records page 3 as unreadable, flags every validator that depends on it (a total that lives on page 3 cannot be checked), and routes the document to review with the missing page called out. Malformed PDFs are expected: broken cross-reference tables, truncated files and invalid fonts are common, so the parser runs under a timeout and memory limit, falls back from the primary parser to a more tolerant one, and then to rendering the pages as images. Deduplication operates at two levels: byte-identical files short-circuit on content hash (return the earlier result), while business duplicates are caught by validation as described earlier. Versioning ties every output to the pipeline, model and schema versions. When the extractor is upgraded, old documents are not silently re-extracted: reprocessing is an explicit job that writes a new version beside the old one, so you can compare them on the same inputs and so that a booked invoice keeps the version it was approved under.

Retention and deletion. Documents commonly contain personal data, bank details and confidential terms. Retention is a property of the document, set at intake from the tenant's policy. Deletion must cover the original, rendered pages, OCR text, extraction records, search indexes and embeddings, review screenshots and caches, and it must propagate on a schedule that you can demonstrate. Logs and traces should carry document ids, not extracted values; a field value in a debug log is a copy nobody will remember to delete. Legal holds and financial-record retention periods may require keeping the structured record while deleting the image, or the reverse, so the stores need separate lifecycles. Access to documents is enforced by the application on every read (tenant id plus role), not by hoping that a prompt tells a model who may see what.

An end-to-end accuracy number hides where the system fails. Evaluation therefore mirrors the pipeline, using a labelled test set of real documents with ground truth (and, for sensitive customers, only their own documents, under their terms). Each metric diagnoses a different layer.

OCR: character and word error rate

Character error rate is CER = (S + D + I) / N, where S, D and I are the substitutions, deletions and insertions in the minimum edit alignment between the OCR text and the reference, and N is the number of characters in the reference. Word error rate has the same form at word level. If the reference vendor name is "Kaveri Packaging" (N = 16 characters including the space) and OCR returns "Kavcri Packaging", one substitution gives CER = 1/16 = 6.25%. For the payment phrase "Net 30 days from invoice date" (6 words), "Net 3O days from invoice date" (letter O for zero) has one word substitution, so WER = 1/6 ≈ 16.7%. One wrong character corrupts the whole word, which is why WER is always higher than CER on the same text, and why for identifiers and amounts it is better to measure exact-match per field than character rates: "1,062.00" read as "1,082.00" has a CER of 1/8 = 12.5%, which sounds small, but the value is wrong in a way that matters.

Fields: precision, recall and F1

Field evaluation compares extracted values to ground truth after normalisation (so "14/10/2026" and "2026-10-14" match). Suppose a test set of 200 invoices with 8 target fields each, which contains 1,600 ground-truth values (illustrative). The system emits 1,520 values (it abstains on 80 fields where it found nothing), of which 1,444 are correct. Precision is the fraction of emitted values that are right: 1,444 / 1,520 = 0.950. Recall is the fraction of true values that were found and right: 1,444 / 1,600 = 0.9025. F1 is their harmonic mean: 2 × 0.950 × 0.9025 / (0.950 + 0.9025) = 0.9256. The distinction matters because abstaining trades recall for precision: a threshold that routes more fields to a person lifts precision of what gets auto-accepted, at the cost of review volume. Report the metrics per field, because a macro average hides that "bank account" at 0.88 is a far bigger problem than "currency" at 0.999, and per vendor or layout, because aggregate numbers are dominated by the large vendors.

Tables, arithmetic, and the operating metrics

Table quality is measured at cell level (with row and column indices): if a test table of 48 cells (12 rows × 4 columns) has 46 cells assigned to the right row and column with the right text, cell accuracy is 46/48 = 95.8%. A stricter metric requires the whole table to be right (any single mistake counts as a failed table), which is the one finance cares about, and the two numbers can be very different. Arithmetic validation pass rate(the fraction of documents where lines, subtotal, tax and total tie out) is a cheap, label-free signal that you can compute on production traffic every day: a drop from 97% to 91% after a deployment is an alarm even with no ground truth, and it is the metric that would have caught our subtotal.

The operating metrics decide whether the system pays for itself. Human review rate is the share of documents (or fields) that reach a person. Cost per document must include that human time. With illustrative unit costs of US$0.0015 per OCR page for a 3-page document (3 × 0.0015 = US$0.0045), US$0.004 for model-based extraction, and an 18% review rate where each review takes 90 seconds at US$12 per hour of labour (0.18 × 90 / 3,600 × 12 = US$0.054), the average is 0.0045 + 0.004 + 0.054 = US$0.0625 per document. The review share is about 86% of the cost, so the lever is review-rate reduction (better calibration, vendor profiles, smarter routing) rather than a cheaper OCR. These unit costs are placeholders, and you should substitute your own. Latency is measured end to end (upload to posted, p50/p95) and per stage; waiting time in the review queue usually dominates it.

Finally, evaluate the thresholds, not only the models. Sweep the confidence threshold on the labelled set and plot the auto-accepted share against the error rate among auto-accepted fields; pick the point where your finance team accepts the residual error, and re-measure it with a blind-reviewed production sample as vendors and templates change. Monitoring for the live system (field-level drift, queue age, failure classes) is the subject of AI Observability.

With the layers understood, the architecture question is who does the reading and extracting. Four families exist and real systems mix them. Template or rule-based extraction anchors on fixed positions, keywords or regular expressions per vendor. It is fast, deterministic, explainable and nearly free, and breaks the moment a vendor moves a box. Specialised document models (layout-aware token classifiers, key-value and table models) generalise across layouts, return native coordinates, and run cheaply, but need labelled training data and maintenance per document family. Multimodal LLMs read the page image and fill a schema from an instruction; they need almost no labelled data, handle novel layouts and free-form text such as contracts well, and cost more per page, add latency, produce non-deterministic output, and can state plausible values that are not on the page, so everything they return should be checked against OCR text, boxes and validation. A hybriduses each where it is strongest.

A defensible default for invoices at moderate scale is therefore hybrid: parse the PDF text layer where it exists; use OCR and a layout model for scans; apply vendor-specific rules where one vendor sends high volume; call a multimodal model only for unrecognised layouts or for fields the cheaper path scored low, then verify its output against tokens (does the claimed value appear in the OCR text at the claimed location?) before it counts. For contracts, where meaning matters more than position, an LLM over a structure-aware parse is the primary reader, with quotation verification. Cheaper approaches go first, and expensive ones are escalated to by routing, an idea shared with Cost Optimization.

type Stage<I, O> = (input: I, ctx: JobContext) => Promise<O>;

async function processDocument(upload: Upload, ctx: JobContext): Promise<DocumentResult> {
  const stored = await quarantineStore(upload);                  // original bytes, content hash
  const safe = await scanAndValidateFormat(stored);              // throws Reject(reason)
  const pages = await renderPages(safe, { dpi: 200 });           // per-page rasters

  const parsed: ParsedPage[] = await mapPages(pages, async (page) => {
    // each page: its own idempotency key and retry budget
    const upright = await normaliseOrientation(page);
    const text = (await trustedTextLayer(upright)) ?? (await ocr(upright));
    const layout = await analyseLayout(upright, text);          // blocks, tables, reading order
    return { page: upright.number, text, layout };
  });                                                            // failed pages => status "partial"

  const docType = await classify(parsed);
  const extracted = await extractFields(docType, parsed);        // schema + evidence + confidence
  const checks = runValidators(docType, extracted, ctx.lookups); // deterministic rules

  const route = decideRoute(extracted, checks, ctx.policy);      // thresholds per field
  if (route === 'auto') return post(extracted, ctx);             // authorised, idempotent write
  return enqueueReview(extracted, checks);                       // human decision required
}

Two decisions in this skeleton are worth defending. First, post sits behind authorisation and idempotency exactly as any other write to a financial system would: the model proposing a record does not give it permission to book one. Second, decideRoute is code with configuration, which means its behaviour can be reviewed, tested against the labelled set and changed without retraining anything. The same separation between a proposal and an authorised action is developed further in AI Agents in Business.

The corollary of "extraction is not interpretation" is knowing when to stop using AI. If your counterparties can send structured e-invoices or an EDI feed, take the structured data and skip OCR altogether. If a form has a fixed layout that you control, a template is better. If a decision is rule-bound (approve if the PO matches and the amount is within tolerance), keep it in rules. Use learned extraction for the unstructured long tail, and let deterministic validation and a human with the evidence at hand be the safety net. Document intelligence also feeds other products: structured records power search and analytics, and parsed documents are the input to retrieval in AI Search Systems and to the grounding data in AI Customer Support Systems. The same extraction ideas also apply when the source is audio, which is the subject of the next lesson.

1. 1

   Extraction, interpretation and validation are separate stages. Plain OCR text is not a record: the pipeline must produce typed, versioned claims with a source region and raw text for each field.

2. 2

   Do not trust the file. Ingestion renders pages at a fixed resolution, corrects orientation and skew, and rejects pages that fail quality gates before OCR runs.

3. 3

   Confidence only means something if it is calibrated. Route fields by a threshold swept against a labelled calibration set, trading review volume against residual error.

4. 4

   Validation is deterministic code with severities: line totals, tax arithmetic and rounding, and format checks catch a wrong printed subtotal (900 against 950) that OCR read perfectly.

5. 5

   Tables spanning pages need inherited headers and page-total reconciliation, and fixed-size text chunking destroys them because a chunk no longer says 1,800.00 is an amount.

6. 6

   Evaluate stage by stage (CER, field accuracy, cell-level table quality, review rate, cost per document), log reviewer corrections append-only, and prefer structured e-invoices or an EDI feed over AI when counterparties can send them.

Document intelligence helps AI understand written information. Another major application category helps AI understand and generate spoken language. In the next module we'll explore AI Voice AI Systems.