Computer Vision Models

Watch an image move from raw pixels through layers of extracted structure and into several downstream capabilities. The animation focuses on one mental model for visual understanding, while the written lesson expands the architecture landscape beyond a single implementation style.

A computer vision model is a function from an array of numbers (an image tensor) to a task-specific output: a class label, a list of boxes, a per-pixel mask, a set of keypoints, or an embedding vector. Inside, it repeatedly transforms those numbers into feature maps or patch tokens that make the relevant structure easier to read: edges, then parts, then objects and layout. Convolutional networks (CNNs) build those features with small sliding filters; Vision Transformers (ViTs) cut the image into patches and let every patch exchange information with every other through attention. CNNs bring a built-in bias toward local structure and tend to be efficient on modest data; ViTs model long-range relationships but cost more as resolution grows. Neither family is categorically better. Which one to use, and whether a vision model is the right tool at all, depends on the output you need, the data you have, and the latency you can afford.

This chapter follows one concrete problem from tensor to deployment: finding the red error badge in payment support screenshots. It stops where language begins. Turning visual features into grounded sentences is the subject of Image-to-Text Understanding, and how images and text are fused in a single system is covered in the Multimodal AI Overview.

A customer sends the screenshot of a failed checkout. The support assistant wants to know about a red badge in the top-right corner of the page. There are at least four distinct questions hiding in that sentence, and each one demands a different output and therefore different labels and a different architecture.

• Classification: does this screenshot contain an error badge? Output: one label (or a probability) per image. Training labels: one tag per image.
• Detection: where is it? Output: a box (x1, y1, x2, y2) in pixels plus a confidence. Labels: a drawn box per badge.
• Segmentation: which pixels belong to it? Output: a mask. Labels: painted outlines, the most expensive kind.
• Reading: what does the badge say? Output: a string. That is optical character recognition (OCR), a text-recognition task, not something a classifier or a box detector produces.

A classifier that is 95% accurate at "contains error" tells you nothing about where to crop, so it cannot feed an OCR step. A detector that returns a box feeds the crop; the OCR step returns a string like "Card declined (code 51)". Choosing the task contract first is not paperwork: it fixes the labels you must collect, the loss function, the head on the model, and the metric you will later be judged on. Starting from "use the best model" and discovering afterwards that you needed boxes is the expensive way to learn this.

A model never receives a "screenshot". It receives an array. A typical preprocessed RGB image is a tensor of shape (3, 224, 224) in channels-first layout (the convention in PyTorch) or (224, 224, 3) channels-last (common in other frameworks and in image libraries). The element count is 224 × 224 × 3 = 150,528 numbers; stored as 32-bit floats that is 150,528 × 4 = 602,112 bytes, about 0.6 MB per image before batching. These dimensions are illustrative: models differ in the input size they were trained on, and nothing about vision requires 224.

The raw file is not what you feed in. Decoding a PNG or JPEG yields integers 0-255 per channel, usually in sRGB colour space with gamma applied. Preprocessing normally does four things, and each must match what the model was trained with:

1. 1

   Decode and fix colour

   Convert to RGB (drop or composite alpha; a grayscale or palette PNG would otherwise give the wrong channel count) and apply the EXIF orientation of photos.
2. 2

   Resize

   Scale to the training resolution. Choose interpolation (bilinear, bicubic, area) deliberately: downscaling a 1170 × 2532 phone screenshot to 224 pixels wide shrinks 12 px body text to roughly 2 px, which no model can read.
3. 3

   Handle aspect ratio

   Squashing a tall screenshot to a square distorts shapes; centre-cropping silently discards the top-right corner where the badge lives; letterboxing (pad to square) preserves content at the cost of empty pixels. Pick one, record it, and apply it identically in training and serving.
4. 4

   Normalize

   Scale to [0, 1] and subtract a per-channel mean and divide by a per-channel standard deviation, (x/255 − μ_c) / σ_c, using the statistics the checkpoint expects.

A subtle consequence: preprocessing is part of the model. Swapping bilinear for nearest-neighbour resizing, or feeding BGR to a network trained on RGB, produces no error message. Accuracy simply degrades, which is why the implementation section later records preprocessing parameters as provenance next to every prediction.

Early vision work hand-designed filters; CNNs learn them. A convolutional layer owns a small grid of weights, the kernel, and slides it across the image. At each position it multiplies the kernel elementwise with the pixels under it and sums the nine products into one number. The grid of results is a feature map: high where the local pattern matches the kernel, low where it does not. A layer has many kernels, so the output has one feature map per kernel (the channel dimension), and a nonlinearity such as ReLU is applied afterwards so that stacked layers do more than one linear operation.

A fully calculated toy example

Take the 5 × 5 grayscale input below (illustrative): the left three columns are dark (10) and the right two are bright (200), a vertical edge. The kernel is the classic Sobel-x vertical-edge detector. Assumptions: stride 1, no padding, and, as deep-learning libraries do, cross-correlation (the kernel is not flipped). The output size follows ⌊(n + 2p − k) / s⌋ + 1 = (5 + 0 − 3) / 1 + 1 = 3, so the feature map is 3 × 3.

For the window covering rows 0-2 and columns 1-3 the pixels are [10, 10, 200] in every row, and the kernel rows are [−1, 0, 1], [−2, 0, 2], [−1, 0, 1]:

row 0: (-1)*10 + 0*10 + 1*200  =  190
row 1: (-2)*10 + 0*10 + 2*200  =  380
row 2: (-1)*10 + 0*10 + 1*200  =  190
                         sum   =  760
Window (row 0, col 0): pixels [10,10,10] per row -> (-1+0+1)*10 = 0 in each row -> 0

This is the central idea. The same nine weights are reused at every location (weight sharing), so the layer detects the pattern wherever it occurs and has only 9 parameters per kernel and input channel, not one per pixel. Stride 2 would skip every other position and halve the spatial size; padding of 1 would keep a 3 × 3 kernel's output the same size as its input. A 7 × 7 convolution with stride 2 and padding 3 maps 224 → ⌊(224 + 6 − 7) / 2⌋ + 1 = 112.

A real learned kernel is not a hand-picked Sobel filter. Training finds the weights that make the final loss go down, and early layers typically end up with edge- and colour-like detectors because those are useful building blocks, a tendency, not a rule.

Stacking convolutions gives growing receptive fields, the region of the original image that can influence one feature. One 3 × 3 layer sees 3 × 3 pixels. Two stacked 3 × 3 layers (stride 1) see 5 × 5; three see 7 × 7; striding or pooling (taking, say, the maximum over 2 × 2 blocks) makes each further layer's view grow faster because one feature now stands for several pixels. Typical backbones are organised as stages: each stage keeps its spatial resolution for a few layers, then halves it while increasing the channel count. Following an illustrative trace for a 224 × 224 input: 112 × 112 × 64, then 56 × 56 × 64, 28 × 28 × 128, 14 × 14 × 256 and 7 × 7 × 512. Space shrinks 32× overall, channels grow, and the representation shifts from "where are the edges" to "which object parts are present".

The part before the head is the backbone or feature extractor; the head is the small, task-specific part. A classification head typically averages the final map over space and applies a linear layer to get class scores. A detection head predicts boxes and confidences from one or several feature scales. A segmentation decoder upsamples features back toward image resolution. This split is why one pretrained backbone can serve many tasks, and why swapping the head is the standard way to adapt it.

The local inductive bias of convolutions, the assumption that nearby pixels matter together and that a pattern means the same thing wherever it appears, is a strong, useful prior. It lets a CNN learn from comparatively little data and run efficiently on accelerators. The limit is long-range relationships: two features far apart (a badge in the corner and the form field it refers to) are linked only after the receptive field grows enough, through many layers or aggressive downsampling that costs fine detail.

The ViT idea (Dosovitskiy et al., arXiv:2010.11929) is to reuse the transformer from language with minimal change: cut the image into fixed-size patches, treat each patch as a token, and let attention mix them. If you need the attention mechanism itself, see Attention Mechanism and Transformer Basics; this section covers only what is specific to images.

With a 224 × 224 image and non-overlapping 16 × 16 patches the grid is (224 / 16) × (224 / 16) = 14 × 14 = 196 patches. Each patch holds 16 × 16 × 3 = 768 raw values. A shared linear layer projects that 768-vector to the model width D, which makes it a patch embedding (the 768 coincidence with some models' width is not required; D is a design choice). Most ViTs then prepend a learned classification token, giving 197 tokens (the 196 count is before special tokens). Because attention by itself ignores order, position information is added to each token, either as learned embeddings or as another scheme; without it the model could not tell the top-right patch from the bottom-left.

The patch layer is, in fact, a single convolution with kernel 16 × 16 and stride 16, a reminder that the families overlap. The rest of the model is a stack of identical blocks: self-attention, an MLP, residual connections and normalisation. The final representation of the classification token (or a pooling of all tokens) goes to the task head. A short runnable check of the shapes:

import numpy as np

H = W = 224; P = 16; C = 3
img = np.random.rand(H, W, C).astype(np.float32)          # (224, 224, 3)
grid = H // P                                                # 14
patches = img.reshape(grid, P, grid, P, C)                   # (14, 16, 14, 16, 3)
patches = patches.transpose(0, 2, 1, 3, 4).reshape(grid * grid, P * P * C)
print(patches.shape)                                         # (196, 768)
tokens = grid * grid + 1                                     # + CLS token
print(tokens)                                                # 197

Attention compares every token with every other. For N tokens, one head of one layer forms an N × N score matrix. At 224 × 224 with 16 × 16 patches, N = 197 and N² = 38,809 scores per head per layer. Double the side length to 448 × 448 and the token count quadruples to 28 × 28 + 1 = 785, so N² = 616,225, about 15.9 times larger, not 4 times. In 16-bit precision (2 bytes) with 12 heads the score matrices alone, if materialised, are 38,809 × 2 × 12 ≈ 0.93 MB per layer at 224 and 616,225 × 2 × 12 ≈ 14.8 MB per layer at 448. That is for the basic implementation: optimised kernels avoid storing the full matrix, but the compute still scales with N².

Resolution matters for exactly the content support teams care about: a 12-pixel font on a 2532-pixel-tall screenshot needs much more than 224 pixels to be legible. This is the tension. Squeeze the image and detail disappears; keep it and attention cost explodes. Several families respond in different ways. Windowed attention (shifted-window hierarchical designs, for example) restricts attention to local groups: with 196 tokens in four 7 × 7 windows of 49 tokens each, the scores fall to 4 × 49² = 9,604 versus 196² = 38,416, and windows are combined across layers, with downsampling between stages. Hybrids put a convolutional stem or early stages in front of attention so the transformer sees fewer, richer tokens. Tiling or cropping processes a large image in pieces. Each trades something (global context, simplicity, or seams at tile borders). None is universally better.

Move the slider below to see how fast the score matrix grows and how windows tame it.

The practical differences follow from inductive bias. A CNN assumes locality and translation structure, so it learns usefully from modest datasets and handles varying input sizes naturally. A plain ViT assumes much less, so it tends to need either substantial data or strong pretraining before it overtakes a CNN; the original ViT results emphasised large-scale pretraining. With that pretraining, its global mixing is a good fit for tasks where relations between distant regions matter, and the same transformer machinery that language models use simplifies sharing infrastructure and scaling recipes. Hierarchical and hybrid designs narrow the gap in both directions, so labels like "CNN" and "ViT" describe a tendency of a design, not a guaranteed behaviour.

Benchmarks reported on public datasets rank architectures under particular training recipes, resolutions and hardware, and the ranking often changes when any of those change. For your decision, three questions are more informative than a leaderboard. How many labelled screenshots do you have, and is there a good pretrained checkpoint close to your domain? What is the highest-detail thing you must resolve, and at what resolution? Where will it run, and what is the latency budget at your batch size? A small pretrained CNN fine-tuned on a few thousand labelled screenshots is a perfectly defensible answer; so is a pretrained ViT when you already have the serving stack for transformers. Measure both on your validation set, as described below.

Because a backbone is shared, the family of tasks is best understood by what comes out. Each contract implies a different label format and a different way of being wrong.

• Classification: a vector of class scores per image, for example {"error_badge": 0.93, "no_badge": 0.07}. Cheap to label, no location.
• Object detection: a variable-length list of (class, confidence, x1, y1, x2, y2). Needs boxes drawn on every training image.
• Semantic segmentation: a class per pixel; two touching objects of the same class merge into one region.
• Instance segmentation: a separate mask for each object, so overlapping badges stay separate.
• Keypoint / pose estimation: a set of named coordinates (corners of a form field, body joints).
• OCR: text strings with positions, usually produced in two stages, text detection then recognition, or by a single model. It reads characters; it does not decide what the text means.
• Image embeddings: a fixed-length vector (for example 512 numbers) used for similarity search and clustering. See Embeddings Explained.
• Visual grounding: given a phrase ("the red badge"), return the region it refers to. It is where vision meets language, covered in Image-to-Text Understanding.

An embedding carries no label and no location: it says an image is like others, which is useful for retrieval of similar failures but not for deciding that this screenshot contains a declined-card message.

A detection model typically proposes many candidate boxes, each with a confidence score, from locations across the feature maps. Boxes are written as corner coordinates (x1, y1, x2, y2) or as centre plus size (cx, cy, w, h) in pixels or normalised to [0, 1]; mixing conventions is a classic bug that silently shifts every prediction. Quality of a box is measured by intersection over union: the area where predicted and true boxes overlap divided by the area they cover together.

def iou(a, b):  # boxes are (x1, y1, x2, y2) in pixels, x2 > x1, y2 > y1
    iw = max(0, min(a[2], b[2]) - max(a[0], b[0]))
    ih = max(0, min(a[3], b[3]) - max(a[1], b[1]))
    inter = iw * ih
    area = lambda z: (z[2] - z[0]) * (z[3] - z[1])
    union = area(a) + area(b) - inter
    return inter / union

A = (40, 20, 140, 80)   # ground truth: 100 x 60 = 6000
B = (60, 30, 160, 90)   # prediction:   100 x 60 = 6000
# intersection: x 60..140 = 80, y 30..80 = 50 -> 4000
print(iou(A, B))        # 4000 / (6000 + 6000 - 4000) = 0.5
print(iou(A, (300, 200, 360, 260)))  # disjoint -> 0.0

Slide the predicted box in the next figure to see IoU fall and the threshold decide what counts as a hit.

A model produces several overlapping candidates for the same object, so a post-processing step called non-maximum suppression (NMS) keeps the highest-confidence box and discards others whose IoU with it exceeds a threshold. A confidence threshold is applied too: only predictions above, say, 0.5 are reported. Both are application choices, not model facts, and they trade missed objects against false alarms. NMS also has a known weakness: genuinely overlapping objects can be suppressed as duplicates, which is one reason instance segmentation, or designs that avoid NMS, are used for crowded scenes.

Segmentation replaces the box by a mask: a per-pixel label. In the toy grid, semantic segmentation would colour the green and red cells identically as "badge", while instance segmentation separates A from B even where they overlap in bounding-box space. A mask is more informative than a box and much more expensive to annotate. For a UI badge, a box is almost always enough to crop for OCR; a mask would earn its cost only if the badge shape itself were needed.

Supervised training adjusts weights to minimise a loss between predictions and labels. The loss follows the contract: cross-entropy for classification; for detection, a combination of a classification loss on each box and a localisation loss on its coordinates (often L1 or an IoU-based term); for segmentation, per-pixel cross-entropy, sometimes combined with an overlap loss such as Dice. For the training/inference distinction see Training vs Inference.

Most of the engineering effort lies in the data, not the optimiser:

• Pretraining and fine-tuning. Start from a backbone pretrained on a large general image set and fine-tune on your labelled screenshots. Freeze early layers if data is scarce; unfreeze more as it grows.
• Augmentation. Random crops, scale, colour jitter, blur and JPEG compression simulate the variation you expect. Choose ones that preserve the label: a horizontal flip turns text into mirror writing, which is wrong for a text-bearing UI and harmless for a street scene.
• Class imbalance. If only 3% of screenshots have the badge, a model that always answers "none" is 97% accurate and useless. Use stratified sampling, class weights, or focal-style losses, and judge with precision and recall.
• Annotation quality. Define what counts as the badge (include the icon? the shadow?) in a written guide, double-label a sample, and measure agreement. Label noise caps achievable accuracy and quietly corrupts the validation numbers too.
• Splits and leakage. Keep training, validation and test sets separate, and split by customer or by session, not by image: ten screenshots from one ticket are near-duplicates, and if some land in training and some in test, the test score is inflated. The same applies to augmented copies, to near-duplicate templates, and to tuning thresholds on the test set.

Labels are expensive, so much of modern vision relies on representations learned with fewer or no human labels. Self-supervised methods create their own training signal, for example hiding parts of the image and predicting them, or making two augmented views of the same image agree. Contrastive image-text training (CLIP, Radford et al., arXiv:2103.00020) trains an image encoder and a text encoder so that matching pairs land close together in a shared vector space and mismatched pairs land far apart. The result is a visual encoder whose features are aligned with language.

Transfer learning is then the default workflow: features learned on large generic data are reused for your narrow task, requiring far fewer labels. And these backbones, CNN or ViT, are the "eyes" of vision-language models, which project the encoder's output into a form a language model can consume. Alignment shows that a screenshot resembles the words "payment error"; it does not verify that a payment failed. The mechanics of turning visual features into grounded text, and where they fail, are in Image-to-Text Understanding, while the general fusion choices are in the Multimodal AI Overview.

All data in this example is illustrative. The goal: given a screenshot uploaded to a support ticket, find the error badge, read its text, and attach both to the case, escalating to a person when the result is uncertain.

Dataset and labels

Collect, with consent and PII redaction, a few thousand historical screenshots stratified over web, iOS and Android, light and dark themes, and several languages. Define one class, error_badge, with a written rule (the coloured container and its text, excluding the browser chrome), and label tight boxes. Split by ticket so near-duplicates never cross splits, reserving a test set that nobody tunes against.

Model, inference and thresholds

With a few thousand labelled boxes, start from a pretrained detection backbone (either family) and fine-tune. Letterbox the screenshot to the training resolution, keeping the aspect ratio so the top-right corner survives. Inference returns, for example, {"cls":"error_badge","conf":0.91,"box":[812,64,1104,132]} in original-image pixels after undoing the letterbox. Apply a confidence threshold (0.6 here, chosen from the validation precision-recall curve to favour not missing badges) and NMS at IoU 0.5. Crop the box with a few pixels of margin and run OCR on the crop at native resolution, which is why detection runs on a downscaled image but OCR must read the original.

Trace of three screenshots (illustrative)

• A: badge conf 0.91, OCR text "Card declined (51)", matches the pattern [A-Za-z ]+ \(\d+\) in the validator. The case gets the observation "screenshot shows text 'Card declined (51)', model confidence 0.91". The assistant still queries the payments API for the actual transaction status through authorised calls before saying anything about it.
• B: best box conf 0.48 (a dark-theme custom skin). Below 0.6, so no OCR result is trusted; goes to a human review queue with the image and the candidate box.
• C: badge conf 0.88 but the OCR output is "Card dec1ined (5l)" with garbled characters; the validator fails, so the case is routed for human review rather than guessed.

Classification. Suppose a held-out set of 200 screenshots has 120 with a badge and 80 without. A classifier gives 108 true positives, 12 false negatives, 10 false positives and 70 true negatives. Accuracy is (108 + 70) / 200 = 0.89, but always answering "badge" would score 120 / 200 = 0.60, which shows how class balance shapes accuracy. Precision = TP / (TP + FP) = 108 / 118 = 0.915 (of the alarms, how many were real); recall = TP / (TP + FN) = 108 / 120 = 0.90 (of the real badges, how many were found). Different denominators, different questions. A confusion matrix lays the four counts side by side and, with more than two classes, shows which pairs get confused.

Detection. A prediction counts as a true positive only if its class is right and its IoU with an unmatched ground-truth box passes a threshold (0.5 here). On the same 200 images, suppose there are 120 ground-truth badges and the detector reports 116 boxes above threshold, of which 102 match: TP = 102, FP = 116 − 102 = 14, FN = 120 − 102 = 18. Then precision = 102 / 116 = 0.879 and recall = 102 / 120 = 0.85, with F1 = 2·102 / (2·102 + 14 + 18) = 0.864. Sweeping the confidence threshold traces a precision-recall curve; average precision (AP) is the area under it, and mAP averages AP across classes (and, in some benchmarks, across several IoU thresholds, so always check which definition a reported number uses).

Segmentation. Overlap metrics for masks: IoU = |P ∩ G| / |P ∪ G| and Dice = 2|P ∩ G| / (|P| + |G|). For a predicted mask of 800 pixels, a ground truth of 1000 pixels and an overlap of 700: IoU = 700 / (800 + 1000 − 700) = 0.636 and Dice = 1400 / 1800 = 0.778. Dice is always at least IoU for the same pair, so never compare one to the other.

An aggregate number can hide the failures that matter. Slice the evaluation by device, theme, language, resolution, compression level and screenshot source, and report per-slice recall. A 0.85 average might be 0.95 on light-theme web and 0.55 on a dark-theme Android skin, and the latter is where a human reviewer is needed. Robustness also means checking performance on a fresh sample from the current traffic, not only on the historical test set.

Latency and memory come from model size, input resolution, batch size and the accelerator. The standard levers, all of which need to be re-evaluated on your own accuracy metric because none is free:

• Quantization stores weights (and sometimes activations) in fewer bits. For an illustrative 25-million-parameter model, 32-bit weights take 25M × 4 B = 100 MB and 8-bit weights 25M × 1 B = 25 MB, a 4× reduction of weight memory, with an accuracy effect that must be measured, especially on small objects.
• Distillation trains a smaller student to imitate a larger teacher's outputs.
• Pruning removes weights or channels; unstructured sparsity only speeds up when the runtime supports it.
• Batching raises throughput on accelerators but adds queueing delay for each request; choose it against your latency target.
• Resolution is often the cheapest lever and the most dangerous one: lowering it can erase small text.
• On-device vs cloud. On-device inference keeps screenshots on the phone and avoids network round trips, but limits model size and ties you to the device's runtime. Cloud inference allows larger models and central updates but moves sensitive images across the network. Details of serving, batching and hardware are in Model Serving and AI Accelerators.

Because performance depends on hardware, runtime and implementation, benchmark your candidates on the target device with realistic inputs and report latency percentiles (p50, p95), not a single average.

Models are trained on a distribution, and fail quietly outside it. Typical causes in the screenshot setting:

• Distribution shift. A redesigned checkout, a new dark theme, or a new phone resolution means pixels that training never contained.
• Small text and downscaling. Reading fails not because the classifier is weak but because the evidence disappeared in the resize.
• Occlusion. A keyboard or a cookie dialog covers the badge; the model cannot detect what is not in the pixels.
• Spurious correlations. If every training positive came from one app version, the model may learn the version's header, not the badge, and the shortcut fails on a different app. Dataset bias of this kind is invisible in an aggregate score.
• Adversarial patterns. Small, deliberate pixel changes or printed patches can flip predictions; random noise does not usually do so, but crafted perturbations can.

To diagnose, work from errors rather than averages. Pull the false negatives and false positives, group them by attribute (theme, device, language, resolution), and look for the slice that dominates. Visualise where the model attends or which pixels drive a score (saliency maps are heuristic aids, not proofs) and check for shortcuts by perturbing candidate cues, for example masking the header and seeing whether the score changes. Then collect targeted data: new screenshots from the failing slice, labelled and added to a frozen, versioned evaluation set so that regressions are caught next release. Production monitoring should track input statistics, confidence distribution and human-review override rate, so drift is noticed before customers notice it (see AI Observability).

Customer screenshots are sensitive data: names, partial card numbers, addresses, order contents, and anything else in the background of the screen. Treat the image pipeline as a data-processing system. Collect only what the task needs, obtain consent where required, redact or blur regions that you do not need (a face, an account number) before storage, set a retention period, restrict access to training and evaluation sets, and make deletion requests reach derived data such as crops, embeddings and training copies.

Image input is untrusted. Adversarial examples can cause misdetections; and in systems that pass images to a language model, visible text inside the image ("ignore previous instructions...") is a prompt-injection channel, as discussed in AI Security. A dedicated detector and OCR step output structured, schema-validated fields, which narrows that surface: you pass a string that matched an error-code pattern, not arbitrary pixels-as-instructions. Even so, treat OCR text as data, never as a command.

Finally, keep the boundary between observation and decision. A badge detected at 0.91 confidence supports a recommendation ("the customer reports a declined card"). Any action that changes an account, such as a refund or a block, must be authorised through the proper service with its own checks and, where stakes are high, a human decision (see HITL and Guardrails). Model confidence is a score, not calibrated correctness, unless you have measured calibration on data that looks like production.

The following is model-neutral pseudocode, not tied to any library. Its purpose is to make shapes and provenance explicit; a real implementation would use your framework's tensors and model loader, and its API names should be taken from that framework's documentation.

interface Provenance {
  modelId: string;          // checkpoint name + version
  preprocess: { size: [number, number]; resize: 'letterbox'; mean: number[]; std: number[] };
  thresholds: { conf: number; nmsIou: number };
  imageSha256: string;      // identifies the input without storing raw pixels in logs
}

interface Detection { cls: string; conf: number; box: [number, number, number, number] } // original-image px

function detectBadge(image: Image, cfg: Config): { detections: Detection[]; prov: Provenance } {
  // 1. preprocess: (H0, W0, 3) uint8 -> (1, 3, 640, 640) float32, record the scale and padding
  const { tensor, scale, pad } = preprocess(image, cfg.preprocess);

  // 2. predict: (1, 3, 640, 640) -> N raw candidates of (cls, conf, x1, y1, x2, y2) in model space
  const raw = model.predict(tensor);

  // 3. postprocess: threshold, NMS, then map boxes back to original-image pixels
  const kept = nms(raw.filter(r => r.conf >= cfg.conf), cfg.nmsIou)
    .map(r => ({ ...r, box: unletterbox(r.box, scale, pad) }));

  // 4. validate: boxes inside image, non-zero area, class known; otherwise reject
  const valid = kept.filter(d => insideImage(d.box, image) && area(d.box) > 0);

  return { detections: valid, prov: provenance(image, cfg) };
}

For real code, the well-known open-source libraries for classification, detection and segmentation (for example PyTorch with torchvision, or an ONNX runtime for deployment) provide these steps; check their current documentation for exact function names rather than copying the pseudocode above. Two bugs account for a large share of silent failures: forgetting to invert the letterbox when mapping boxes back, and mixing coordinate conventions (pixels vs normalised, corners vs centre-size).

A general vision-language model can be asked, in one prompt, "is there an error badge and what does it say?" That flexibility is real and often the right starting point: it needs no labelled data, handles unseen layouts, and can answer follow-up questions. But it also hallucinates, misreads small text, miscounts and is hard to bound in latency and cost; you validate its free-form output rather than a schema-constrained field, and a confident-sounding sentence is not an exactness guarantee.

A task-specific detector gives fast, cheap, repeatable boxes with measurable precision and recall on your distribution, at the price of labelling and maintenance. A dedicated OCR engine is typically more reliable for exact strings such as error codes, amounts, and IDs. And when the UI element is deterministic (a fixed-colour badge at a known position in your own app's screenshots), template matching or colour-region analysis may beat both: no training, no hallucination, and trivially testable. A sensible layering is: deterministic rules where they suffice, a detector plus OCR for the high-volume exact extraction, and a VLM for the long tail of unusual layouts and for explanations, with the VLM's claims about on-screen content cross-checked against the OCR output and authoritative account data. Choose per field by asking what error rate you can tolerate and how you will measure it.

When the pipeline widens to speech and motion, the same logic applies with new constraints: see Video Understanding for what changes once frames form a sequence, and Text-to-Image Generation for the opposite direction, where a model synthesises pixels rather than interpreting them.

1. 1

   Name the task and its output contract before choosing a model: classification (a label), detection (boxes plus confidence), segmentation (masks), OCR (strings) and embeddings (vectors) need different labels and fail differently; a classifier cannot tell you where to crop.

2. 2

   Preprocessing is part of the model. Resize method, colour order and normalisation must match training, because swapping them silently changes outputs.

3. 3

   CNNs bake in locality and weight sharing, so they work well with modest data and are cheap at moderate resolution; ViTs split the image into patches and use full attention, which costs N × N per head, so resolution hurts quickly and small text needs windows, tiling or a higher-resolution path.

4. 4

   Public benchmark rankings reflect a training recipe, resolution and dataset; pick CNN or ViT by measuring on your own screenshots, and start from a pretrained backbone.

5. 5

   Split train, validation and test sets by customer or session to avoid leakage, handle class imbalance (a model that always answers 'none' can be 97% accurate), and write an annotation guide.

6. 6

   Layer the system: deterministic rules where they suffice, a detector plus OCR for high-volume exact extraction, and a general VLM for the long tail, with its claims cross-checked against OCR output and account data.