Text → Image Generation

Watch a prompt become concepts, a noisy latent canvas get refined step by step, and a finished image emerge. The animation illustrates a diffusion-style workflow because that pattern is common, but not every text-to-image system uses the exact same architecture.

Text-to-image AI models work by conditioning a generator on the prompt: the text is encoded into a numerical signal, an image starts as random noise, and a diffusion model denoises it step by step under that signal until a sampled picture matches the words. In other words, a text-to-image model is a conditional generator. It has learned, from a large collection of image and caption pairs, a probability distribution over images given a description, and it produces an image by sampling from that distribution. In the diffusion family, which most of this chapter explains, sampling starts from random noise and repeatedly applies a trained network that predicts how to remove a little of that noise. Each prediction is steered by a numerical representation of your prompt, so the noise settles into an image that is statistically consistent with the words. Nothing is looked up in a library of existing pictures, and nothing is rendered from a scene description; a new picture is constructed from learned visual statistics.

That one sentence hides a lot of engineering: how a prompt becomes a conditioning signal, what denoising actually computes, why most systems work in a compressed latent space, which other architectures exist, what the knobs (seed, steps, guidance, strength) do, how editing differs from regeneration, and what it takes to run all of this behind an API with queues, safety checks and provenance. The rest of the chapter follows a single concrete request through those layers. Publishing workflows that combine generated images with other content are covered in AI Content Generation.

A product designer on the payments team writes: "Create a clean 16:9 illustration of a payment checkout flow, blue accents, no brand logos." It is the kind of request a support-documentation team might make when it needs an illustration for a help article about payment errors (the same fictional support scenario used across the Multimodal AI lessons, but here the task is to produce a picture rather than read one). All prompts, settings and numbers in this chapter are illustrative.

There are three different ways software could answer that request, and keeping them apart clarifies what generation is. Retrieval would embed the request, search an image collection and return the nearest existing picture. It can only return something that already exists, and it cannot honour "no brand logos" unless some stored image happens to satisfy it. Deterministic rendering would take an explicit description (a layout file, an SVG template, chart data) and draw exactly what it specifies, the same way every time. Generation samples a new image from a learned distribution conditioned on the text. It can invent a composition that exists nowhere in a database, but it gives up the exactness and repeatability of rendering. Generation is also not recognition: a classifier maps an image to a label, whereas a generator maps a description to one of many possible images (see Vision Models for the discriminative side).

The request is deceptively demanding. "Payment checkout flow" implies a composition: several screens or steps, in a left-to-right order, with arrows. "16:9" is a constraint on canvas shape. "Blue accents" is a colour constraint that must hold across the whole image. "No brand logos" is a negation, and negations are notoriously weak in text conditioning because the words "brand logos" still push the representation toward logos. And if the designer later adds "with the label 'Pay now' on the button", the model must render legible letters in a specific font, which is among the hardest things for pixel-level generators. Each of these difficulties has a mechanism behind it, and the sections below tie them to the architecture.

Formally, let x be an image and c a conditioning input, here the prompt. The training data is a set of pairs (x, c). The model's goal is to approximate the conditional distribution p(x | c): a function that assigns high probability to images that plausibly match the caption and low probability to everything else. Generating means drawing a sample from that distribution. Two consequences follow directly.

First, the output is not unique. Many different images are consistent with "a clean 16:9 illustration of a payment checkout flow", and the model's randomness (the starting noise, controlled by a seed) selects among them. Asking twice with different seeds yields different pictures that all satisfy the text to varying degrees. Second, the model can only be as specific as its learned distribution allows: if "no brand logos" is rare in captions, the distribution contains little information about how to condition on it. Training data also shapes style and content defaults, so what the model produces when the prompt is silent (people, colours, layout conventions) is a statement about the data, not about the user's intent.

The conditional could be anything: a text prompt, a reference image, a sketch, a depth map, a mask plus a prompt. Text is the most common because captions are plentiful, but the machinery is general. Everything about editing later in the chapter is the same model conditioned on more than text.

The denoising network cannot read text. The prompt must first be turned into numbers, in three steps. A tokenizer splits the string into sub-word tokens and maps them to integer ids (the same idea covered in Tokens Explained). A text encoder then runs those ids through a transformer and produces one vector per token, so a prompt of n tokens becomes an n × d matrix of contextual embeddings; some encoders also emit one pooled vector summarising the whole prompt. Depending on the model family the encoder is a contrastively trained text tower (CLIP-style, see arXiv:2103.00020), a general language-model encoder, or several encoders whose outputs are combined. Finally the denoiser consumes this matrix through a conditioning mechanism.

The most common mechanism in U-Net style denoisers is cross-attention. Inside the network, image positions form the queries, and the text embeddings supply keys and values: Attention(Q, K, V) = softmax(Q·Kᵀ / √d)·V (the general form is explained in The Attention Mechanism). Each image position therefore computes a weighted mix of the prompt tokens: positions in the upper right might attend strongly to "blue" and "button", positions in the lower area to "flow". With our canvas (derived below) there are 3,600 image positions and, for an encoder with a 77-token context (a well known CLIP limit), 77 text positions, so each head's attention map has 3,600 × 77 = 277,200 weights. Transformer-based denoisers often do something different: they concatenate image tokens and text tokens into one sequence and run joint self-attention, and some models inject a pooled prompt vector through learned scale-and-shift modulation of intermediate layers. The right statement is that the prompt conditions the network through some attention or modulation pathway, and the pathway differs between model families.

This explains the behaviour designers observe. A more specific prompt helps because it moves the conditioning vectors toward a narrower region of the learned distribution: "flat vector illustration, three rounded cards left to right connected by arrows" constrains layout in a way "payment thing" cannot. But conditioning is a soft statistical influence, not a program. The network has no explicit slot for "exactly three cards"; it has learned correlations between words and visual structure, and the correlation for counts, spatial relations and negation is weaker than the one for colour or style. Specificity improves the odds; it cannot guarantee compliance.

Diffusion models (the formulation popularised by DDPM, arXiv:2006.11239) are best understood by separating two phases that happen at completely different times.

Training: the forward process is fixed, the reverse is learned

During training, the forward (noising) process is a fixed recipe, not something learned. Take a real training image x₀ and a noise level index t. Sample Gaussian noise ε and mix: x_t = √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε. The cumulative coefficient ᾱ_t falls from near 1 (almost clean) to near 0 (almost pure noise) as t grows; √ᾱ_t is the signal weight and √(1 − ᾱ_t) the noise weight. The network is shown x_t, the timestep t and the caption embedding, and is asked to predict the noise that was added: the loss is the mean squared error between ε and the prediction ε_θ(x_t, t, c). Over millions of examples and random t values it learns, at every noise level, what part of a noisy input is noise and what part is plausible image structure for that caption.

A common training detail matters later: with some small probability (a value such as 10% is typical in published recipes; treat it as a configurable choice) the caption is dropped and replaced by an empty prompt. The same network therefore learns both a conditional and an unconditional prediction, which is what makes guidance possible.

Inference: start from noise, apply the learned reverse step

At sampling time there is no training image at all. The sampler draws a fresh noise tensor from a seed, then loops: ask the network for its noise prediction given the current tensor, the timestep and the prompt embeddings; use a deterministic or stochastic update rule (the scheduler) to remove part of the predicted noise and move to the next, lower noise level; repeat until the noise level reaches zero. This does not reconstruct a particular stored picture. The network has learned a vector field over noisy images, and following it from a random starting point lands on some image the distribution supports. Different starting noise lands on different images. That said, if training data contained many near-duplicates of an image, the learned field can place high density around it, and a prompt closely matching its caption can reproduce something very similar; memorisation is a real, measured risk, not an impossibility (see the rights section).

To make the loop concrete, shrink everything. Suppose the schedule has just four steps and the cumulative coefficients are ᾱ = [1.00, 0.80, 0.45, 0.15, 0.02] for t = 0..4 (toy numbers chosen for readability; a real linear schedule with 1,000 steps ends near ᾱ ≈ 4·10⁻⁵). Taking square roots gives the mix at each level:

t   abar   signal=sqrt(abar)   noise=sqrt(1-abar)   signal/noise power
0   1.00   1.000               0.000                inf
1   0.80   0.894               0.447                4.00
2   0.45   0.671               0.742                0.82
3   0.15   0.387               0.922                0.18
4   0.02   0.141               0.990                0.02

Read the strip from right to left in training (adding noise) and from left to right in sampling. The trace of one sampling run goes like this.

1. 1

   t = 4, pure-looking noise

   The latent is sampled from the seed. The conditioning matters most here but has the least to grip: the network can only commit to coarse structure, such as "a wide bright block roughly in the middle, darker elsewhere", because fine detail is buried under noise (signal power is 2% of noise power).
2. 2

   t = 4 → 3, layout emerges

   The prediction for the noise is subtracted partially. Large-scale arrangement is decided here: where the cards sit, which side the button is on. These early choices are the ones the rest of the run is constrained by; a layout the prompt does not support, such as one card instead of three, is hard to repair later.
3. 3

   t = 3 → 2, objects firm up

   With signal power near 0.8 of noise, shapes with consistent colour appear. The prompt tokens "blue" and "button" shape which regions become blue and button-like. Errors from the previous step are not copied exactly, because the network sees the whole current tensor and can pull a misplaced element toward a more plausible one, but it cannot move something far.
4. 4

   t = 2 → 1, edges and detail

   Texture, edges and small features resolve. Because every step re-predicts noise from the current state, small errors are partly self-correcting at fine scales and mostly permanent at coarse scales.
5. 5

   t = 1 → 0, final cleanup

   The last step removes the remaining grain. The final latent is then decoded into pixels (next sections). No step compared the result to a stored image; the only reference was the network's learned sense of "what a noise-free image looks like given this prompt".

The strip is a schematic: real intermediate latents are not meaningful pictures, and the panels were drawn from the toy weights above, not captured from a model. What the trace does teach accurately is the division of labour across noise levels. High noise decides composition, low noise decides detail, and every step consults the prompt again, which is why prompt influence is not a one-time event at the start.

Running the denoiser on raw pixels is expensive, because a 1280 × 720 RGB image has 1280 × 720 × 3 = 2,764,800 values. Latent diffusion (Rombach et al., arXiv:2112.10752) first trains an autoencoder: an encoder compresses an image into a smaller grid of latent values, and a decoder reconstructs the pixels from them. The diffusion network then operates entirely on latents, and only the final denoised latent is passed through the decoder once.

Take a downsampling factor f = 8 and 4 latent channels (a configuration used by early latent diffusion models; others use different factors and channel counts, so treat these as illustrative assumptions). The latent for our canvas has shape (720 / 8) × (1280 / 8) × 4 = 90 × 160 × 4 = 57,600 values, which is 2,764,800 / 57,600 = 48 times fewer numbers than the pixel tensor. If the denoiser is a transformer that groups latents into 2 × 2 patches, the sequence has (90 / 2) × (160 / 2) = 45 × 80 = 3,600 tokens, the figure used in the attention count earlier. The patching idea is the same one Vision Models develops for image encoders.

The compute implication is the reason the approach spread: the expensive repeated step works on 48 times fewer values (and with a transformer denoiser, on a much shorter sequence). The cost is that the autoencoder becomes part of the quality ceiling. Anything the decoder cannot reproduce, the diffusion model cannot generate. In practice this shows up as lost fine detail, mushy small text, odd high-frequency textures, faces with slightly wrong small features, and occasional colour or banding artefacts. A better decoder can improve these without changing the denoiser, and it is a reason generated text and thin lines are harder than large blobs of colour.

"Diffusion" describes a popular family, not the whole field, and a chapter that implied otherwise would be out of date. The useful distinction is how a model turns noise or a token sequence into an image.

Diffusion variants differ in what the network predicts (noise, the clean image, or a mixture), in the denoiser backbone (convolutional U-Net or transformer), in the scheduler that turns predictions into updates, and in whether they run in pixel space, latent space, or cascaded stages at increasing resolution. Flow-based and rectified-flow-style models (see arXiv:2209.03003) are trained so that the network predicts a velocity that moves a sample along a nearly straight path from noise to data; the sampler integrates that path. The practical appeal is that straighter paths can tolerate fewer steps, but the conditioning, latent autoencoder and guidance ideas carry over, and many recent systems combine a transformer backbone with this objective.

Autoregressive image-token models take a different route. An image tokenizer (a learned discrete autoencoder) turns an image into a sequence of codebook indices, and a transformer is trained to predict the next image token given the text tokens and the earlier image tokens, exactly as a language model predicts words (the early DALL·E, arXiv:2102.12092, used this pattern). Generation is one pass per token rather than one pass per denoising step, and sampling temperature, not noise level, is the randomness knob. Strengths include reuse of language-model tooling and a unified token stream with text; weaknesses include long sequences for high resolution, left-to-right commitment (an early token error cannot be repaired by later global refinement), and the quantisation loss of the tokenizer.

Which family a product uses is an implementation decision, and sometimes hybrids exist. For the rest of the chapter the diffusion loop serves as the running example because its knobs (steps, guidance, strength) are the ones most APIs expose, and the guidance and editing ideas translate to the others with adjustments. When a vendor does not document the family, an application should not assume it.

Only the controls the specific API exposes exist for you; many hosted services hide steps, guidance or scheduler choice entirely. For those that expose them, each knob has a precise meaning.

Seed. The seed initialises the random generator that draws the starting noise (and any noise injected by stochastic samplers). The same seed, prompt, settings, model version and scheduler reproduce the same image in principle. In practice reproducibility is weaker than people expect: different GPU architectures, library versions, kernel choices and reduced-precision arithmetic change floating-point rounding, and a model or scheduler update changes the function entirely. Store the seed and the full parameter set with every generated asset for audit and for "regenerate like this", but never promise bit-identical output across deployments.

Resolution and aspect ratio. They set the latent shape. 16:9 at 1280 × 720 gives the 90 × 160 latent above. Models are trained on particular size ranges; requesting a shape far from them may produce repeated or cropped content, and the cost grows with the number of latent positions (see the compute section). Generating at a supported size and upscaling, or generating on the nearest supported ratio and cropping, is common.

Steps. More sampling steps give the scheduler finer integration of the learned reverse path, with diminishing returns: structure is mostly settled early, and extra steps polish detail. Latency is almost linear in steps. The useful number depends on the scheduler and model, so it should be measured against an evaluation set, not taken from a rule of thumb.

Image-to-image strength. Instead of starting from pure noise, the source image is encoded to a latent, noised up to an intermediate level, and then denoised. Strength s ∈ [0, 1] selects how far to noise: with S = 30 steps, s = 0.2 runs about 6 steps (a light restyle), s = 0.4 runs 12, s = 0.7 runs 21, and s = 1.0 runs all 30 and discards essentially all of the source. Lower strength preserves composition; higher strength gives the prompt more freedom. It is the same loop entered part-way down the noise schedule.

Slide the strength to see how many of the 30 steps run.

Negative prompts. Where supported, the negative prompt replaces the "unconditional" branch of guidance (below) with a prediction conditioned on text to move away from. This is why "no logos" in the positive prompt is unreliable while "logo, watermark, text" in a negative prompt can work in systems that implement it: the negation is expressed geometrically, not linguistically. It still is a steering heuristic, not a filter, and it does not exist in every model family.

Classifier-free guidance

Recall that the network was trained to predict noise both with and without the caption. At each step the sampler can run it twice: once with the prompt, ε_c, and once with an empty (or negative) prompt, ε_u. Classifier-free guidance (Ho and Salimans, arXiv:2207.12598) extrapolates in the direction the prompt pushes the prediction:

eps_hat = eps_u + w * (eps_c - eps_u)

Toy 2-value example: eps_u = [0.20, -0.10], eps_c = [0.50, 0.10]
  w = 1.0 -> [0.50, 0.10]   (just the conditional prediction)
  w = 4.0 -> [1.40, 0.70]
  w = 7.5 -> [2.45, 1.40]

Drag the guidance scale to watch the guided prediction move along that direction.

Intuitively, ε_c − ε_u isolates what the prompt adds to the prediction. Using w = 1 trusts the conditional prediction as trained; w > 1 amplifies the prompt's effect. Higher guidance usually improves adherence to the text and raises apparent contrast, at the price of reduced diversity across seeds and, pushed too far, oversaturated colours, harsh contrast and artefacts, because the extrapolated prediction leaves the region the network was trained on. Practical scales depend on model, scheduler and parametrisation, which is why published defaults vary. Guidance also costs compute: the two evaluations per step are either two forward passes or one batch of two. Some distilled or flow-based models are trained with guidance built in and expose no such parameter, so treat the formula as a description of one common mechanism, not of every generator.

The designer likes the first draft but wants the checkout card on the right to show a green "payment received" tick instead of an error symbol. Re-running the original prompt with a changed word would almost certainly change the entire layout, because the seed-to-image map is chaotic: a small prompt change shifts the conditioning at all noise levels, including the high-noise ones where composition is decided. Editing exists precisely to avoid that.

Inpainting supplies a source image and a binary mask. The region outside the mask must be kept; the region inside is regenerated according to the prompt. In a latent model the mask is downsampled to the latent grid (so it should align to multiples of the downsampling factor: our 880..1200 by 120..360 rectangle is exactly 110..150 by 15..45 latent cells), and at each denoising step the unmasked latent area is replaced with a suitably noised copy of the source latent, so that only the masked area is free. Dedicated inpainting models take the mask and the masked source as extra input channels instead. Either way the masked fraction is 76,800 / 921,600 = 8.33% of the pixels, and the other 91.67% is meant to remain unchanged.

Two limits are worth knowing. The unmasked area is preserved in latent space, so after decoding it can differ slightly from the original pixels, and a seam or colour shift at the mask boundary is a classic artefact; compositing the original pixels back outside a slightly feathered mask removes the first problem. Second, the model sees context only through the surrounding latent, so large masks give it a lot of freedom and a high risk of style drift. Outpainting is the same operation with the mask placed in a newly added border region.

Other control signals work as extra conditioning rather than a mask. A reference image can be encoded and injected to carry style or subject identity; structural control inputs such as edge maps, depth maps or pose skeletons constrain layout while the text supplies content; and region-level control assigns different prompts to different parts of the canvas. They differ in strength and in how well they preserve identity across a series of images. Consistent characters or a consistent illustration style across ten help articles remain a weak point: the standard workaround is to pin the seed and reference image, record every setting, and review each asset, not to assume the model remembers.

The worked edit request is therefore: source image I (1280 × 720), mask M (rectangle above), prompt "green circular checkmark icon on the card, same flat blue style", strength near 1.0 inside the mask only. What we expect is a changed 320 × 240 region and an otherwise identical canvas. What we must still check by eye is that the icon is legible, the colours match, and the boundary has no visible seam; the edit's success cannot be read from the parameters.

The weaknesses of text conditioning are not random; they follow from how conditioning works. Cross-attention and joint attention bind words to image regions through learned correlations, so properties that appear on whole regions (colour, style, material) are bound well. Counts require representing "exactly three" discrete objects, and the model has only statistical hints about numerosity. Spatial relations("left of", "behind") require a composition-level binding that was decided at high noise from an embedding in which relational words are weak. Attribute binding ("blue card, red arrow" versus "red card, blue arrow") can swap, because the embedding of the whole prompt does not strongly separate which adjective attaches to which noun. Negation, as noted, is poorly represented in the positive prompt.

Legible text is the hardest case. Letters are thin, high-frequency, exact structures: a single wrong stroke changes the word. The generator produces pixels from a statistical sense of what text looks like; the latent autoencoder may blur thin strokes; and the text encoder may carry subword-level information rather than a guarantee about spelling. Large, short, common words in common fonts often come out right, long strings or small print often do not. Exact brand constraints are the same problem in another form: a logo or brand colour must match a specification pixel for pixel or hex for hex, and a model can only approximate it, may invent a near-miss logo, and may reproduce protected marks.

Prompt writing is constraint engineering: each sentence should remove a degree of freedom the model would otherwise fill with defaults. Consider three revisions of the designer's request. None of the expected-element lists below is a claim about what a model outputs; they define what an inspector should look for.

v1 (underspecified)
  "payment checkout illustration"
  Free to decide: aspect ratio, style, number of steps, colours, people,
  devices, brands, text. Likely to inherit data defaults.

v2 (constrained)
  "Flat vector illustration of a three-step payment checkout flow shown as
   three rounded cards from left to right connected by arrows, white
   background, blue accent colour, generous margins, wide 16:9 composition."
  Adds: style, count, order, background, colour, aspect.

v3 (constrained + exclusions + deterministic overlay plan)
  prompt:    (v2) + "cards are empty with simple shapes instead of text"
  negative:  "logo, brand name, watermark, photo, 3d render, readable text"
  size:      1280 x 720      seed: 4242 (recorded)
  overlay:   step titles and button label added later as SVG text

v1 produces whatever the data's most common "checkout" imagery is, often a hand holding a card or a shopping cart, and has no reason to match 16:9. v2 states layout, count, order and style, which the model can mostly bind because they are region- and style-level properties. v3 acknowledges the weak points: it asks for no text (because text will be added by code), it routes exclusions through the mechanism that handles them best, and it records the seed so a reviewer can regenerate and compare.

Whether v3 actually succeeded is a question for inspection, and automated tooling alone cannot answer it. A practical manual checklist, applied to every candidate, is:

• Canvas is 16:9 and nothing important is cropped by the edges.
• Exactly three cards, ordered left to right, with arrows pointing in the same direction.
• Colours: blue accents dominate; no unintended colours, and background is plain white or near white.
• No logos, brand marks, watermarks, or pseudo-letters that look like real words.
• No malformed shapes: broken arrow heads, merged cards, uneven corners, artefacts at card edges.
• Style is flat vector throughout; no photographic textures or random shading.
• Safe for the help article: no depiction of a real person's payment details or of a real organisation.

The result is a decision (accept, regenerate with another seed, edit a region, or draw it by hand), and the record of that decision belongs in the asset's metadata.

Image generation takes seconds, not milliseconds, needs scarce accelerator capacity, and can fail in several ways, so a request cannot be handled like an ordinary synchronous API call. A typical architecture (a representative design, not any particular vendor's) is a pipeline with stages that can be made independently observable, scaled and retried: request validation and a policy check on the prompt, prompt normalisation, a job queue, a model worker, a safety check on the output, storage with metadata, and delivery. The sequence below shows why the API returns a job identifier instead of an image.

Job identifiers and idempotency. Because the call returns before the work is done, the client polls (or receives a webhook). The submit endpoint should accept a client-supplied idempotency key, so a retry after a network timeout returns the existing job rather than starting a second billable generation. A minimal illustrative contract:

type GenerationRequest = {
  idempotencyKey: string;        // client-chosen, unique per intent
  prompt: string;
  negativePrompt?: string;       // only if the model supports it
  width: number; height: number; // must be a supported size
  seed?: number;                 // stored for audit, not a determinism promise
  n: number;                     // images per job
};

type JobStatus =
  | { state: 'queued' | 'running'; jobId: string }
  | { state: 'succeeded'; jobId: string; assets: { url: string; sha256: string; model: string; params: object }[] }
  | { state: 'blocked'; jobId: string; reason: 'prompt_policy' | 'output_safety' }
  | { state: 'failed'; jobId: string; retryable: boolean };

Retries and rate limits. Worker failures (out-of-memory on a large size, preempted node) are retryable, whereas policy blocks are not and must not be retried with the same input. Retries should use bounded attempts with backoff, and the queue should expose depth and age so the autoscaler reacts before users wait minutes. Per-user and per-tenant rate limits protect shared capacity and cap spend; admission control can reject or deprioritise oversized requests (large resolution times many images times many steps) at the edge.

Safety in two places. The prompt is checked before spending compute, and the finished image is checked after generation, because a benign prompt can still produce a disallowed image and an automated prompt filter can be bypassed with creative phrasing. Policies and classifiers are application decisions layered on a model, with their own error rates; see AI Security for the broader threat model and Human-in-the-Loop and Guardrails for review patterns when automated checks are not enough.

Asset lifecycle. Store the image under a content-addressed or job-scoped key, with metadata (prompt, settings, seed, model and scheduler versions, safety outcome, creator, timestamps, hash). Decide retention explicitly: prompts and reference photos can contain personal data, and generated images can be sensitive derivatives, so retention windows, deletion on request and access control are system requirements. Serve through a CDN with signed URLs where assets are private, and keep the original separate from derived sizes and compressions. Serving mechanics for the model workers are covered in Model Serving.

Cost and latency come from a small number of multipliers. Sampling steps S set how many times the denoiser runs. Guidance usually doubles the passes per step (P = 2), unless the model has guidance built in. The resolution sets the number of latent positions N, which sets the cost per pass; for attention-based denoisers the cost of self-attention grows roughly with N², while the other layers grow roughly linearly. Model size sets both memory (weights, plus activations that grow with N and batch size) and per-pass time. Numeric precision (fp32, fp16, bf16, or lower-bit quantisation) changes memory and speed, and sometimes quality. Batch size raises throughput by sharing weight reads but increases memory and per-request latency.

Apply that to our canvas. With f = 8 and 2 × 2 patches, 1280 × 720 gives 3,600 tokens, against 1,024 for a 512 × 512 image, i.e. 3.52 times as many. The attention part scales as the square of that ratio, 3.52² ≈ 12.4 times, so doubling the side of an image is far worse than doubling the cost. A labelled illustrative request-cost formula, with timings that are assumptions chosen for arithmetic only and not measurements of any hardware, is:

T_request = T_text_encode + S x P x T_pass(N) + T_decode + T_safety

Assumed: T_text_encode = 15 ms, S = 30, P = 2, T_pass = 45 ms,
         T_decode = 120 ms, T_safety = 150 ms

S x P x T_pass = 30 x 2 x 45 = 2,700 ms
T_request      = 15 + 2,700 + 120 + 150 = 2,985 ms

Throughput per worker = 1 / 2.985 s = 0.335 requests/s
For 10 requests/s peak: 10 x 2.985 = 29.85 busy workers
At 70% target utilisation: 29.85 / 0.7 = 42.6 -> 43 workers

The denoising loop is 2,700 / 2,985 ≈ 90% of the request, so the levers that matter are the ones that reduce S or P: fewer steps with a better scheduler, distillation to a few-step model, built-in guidance, smaller resolution followed by upscaling, lower precision, and caching. Caching helps less than in text systems: because outputs are sampled, an exact-match cache only helps for identical prompt, settings and seed, which is useful for re-delivery and regeneration of a recorded asset but not for general reuse. See Inference Infrastructure and Cost Optimization for capacity and spend, and AI Accelerators for the hardware side; this chapter deliberately gives no fixed GPU numbers because they vary with model, kernel and precision.

Runnable check of the arithmetic

The following Python script reproduces every number used in the toy trace, the guidance example, the latent sizes, the cost model and the mask geometry. It uses NumPy only and was executed to produce the values quoted in this chapter. It is a calculator for the text, not an image generator.

import numpy as np

# 1. Toy forward noising schedule: x_t = sqrt(abar) * x0 + sqrt(1 - abar) * eps
abar = np.array([1.00, 0.80, 0.45, 0.15, 0.02])
for t, a in enumerate(abar):
    snr = a / (1 - a) if a < 1 else float("inf")
    print(f"t={t} signal={np.sqrt(a):.3f} noise={np.sqrt(1 - a):.3f} snr={snr:.2f}")

# 2. Classifier-free guidance on one predicted-noise vector
eps_u, eps_c = np.array([0.20, -0.10]), np.array([0.50, 0.10])
for w in (1.0, 4.0, 7.5):
    print(f"w={w}", np.round(eps_u + w * (eps_c - eps_u), 3))

# 3. Latent shape, token count, cost
H, W, f, C, p = 720, 1280, 8, 4, 2
lat = (H // f, W // f, C)
print("pixels", H * W * 3, "latent", lat, int(np.prod(lat)), "ratio", H * W * 3 / np.prod(lat))
n_tok = (H // (f * p)) * (W // (f * p))
n_ref = (512 // (f * p)) ** 2
print("tokens", n_tok, "vs", n_ref, "ratio", round(n_tok / n_ref, 3), "attention", round((n_tok / n_ref) ** 2, 2))
S, P, t_pass, t_txt, t_dec, t_safe = 30, 2, 45, 15, 120, 150  # ms, assumed
total = t_txt + S * P * t_pass + t_dec + t_safe
print("request ms", total, "req/s per worker", round(1000 / total, 3))
print("workers at 10 req/s:", round(10 * total / 1000, 2), "at 70% util:", round(10 * total / 1000 / 0.7, 2))

# 4. Mask geometry and image-to-image strength
x0, x1, y0, y1 = 880, 1200, 120, 360
print("mask px", (x1 - x0) * (y1 - y0), "frac", round((x1 - x0) * (y1 - y0) / (H * W), 4))
print("latent cells", (x1 - x0) // 8 * ((y1 - y0) // 8), "of", (H // 8) * (W // 8))
for s in (0.2, 0.4, 0.7, 1.0):
    print("strength", s, "steps", round(s * S))

There is no single number that says whether an image generator is good, and anyone who proposes one is collapsing several different questions. A prompt-adherence score computed with a vision-language model measures whether the image appears to match the text, and it inherits that model's blind spots about counts and small details. A distribution-level quality or diversity metric compares statistics of many generated images with a reference set, and says nothing about whether one particular image is usable. A human preference study captures taste, but its result depends on the prompts and raters chosen. None of these detects a leaked logo, a subtle face distortion, or a seam at the boundary of an edit. For that reason a serious evaluation uses several measurements chosen to match the downstream use, on a fixed prompt set that includes the hard cases from the previous sections.

The rubric below summarises the dimensions developed in this chapter. It is a checklist for an evaluation harness, not a scoring formula, and each row names an observable method.

Edit preservation is one of the few places where an objective measure is easy: compare source and result outside the mask and require the mean absolute difference to be below a threshold you choose (zero is not achievable through a latent round trip unless you composite the original pixels back). Acceptance rate, the fraction of candidates a reviewer accepts without regeneration, links the technical settings to the downstream cost, because every rejected candidate costs another 2,985 ms of the assumed model above.

A generator is a content-production tool, and the risks that follow are properties of the system, not of one output. Copyright and memorisation. Models learn from large datasets, and training on a work does not mean the model stores a copy, but studies have shown that models can reproduce training images in some circumstances, particularly for duplicated, distinctive content. Generated images are therefore not guaranteed to be unique or non-infringing, and they do not become so by being "AI made". Whether a particular training practice or output is lawful is a legal question that varies by jurisdiction and is still developing; an organisation should obtain legal guidance, prefer models and datasets with documented licensing for commercial use, and avoid prompts that ask for a named artist's style or a protected character.

Privacy and reference photos. When a user supplies a photo for identity or style reference, the image may contain faces, children, locations or documents. Collect consent, minimise what is stored, set short retention, and keep the photo out of any training or evaluation corpus unless explicitly agreed. Editing a real person's photo can create convincing but false depictions, so likeness-related requests deserve stricter policy and sometimes human review.

Misleading synthetic media. Realistic generated images can be used to deceive, for instance to fabricate a "screenshot" of a payment confirmation or an error to support a fraudulent claim. In the multimodal support scenario this means an uploaded image is never evidence of an account fact on its own, and a generated image shown to customers should be labelled as an illustration.

Watermarks and provenance. Two approaches exist. An invisible watermark embeds a signal in the pixels (or in the generation process) that a detector can later find. Content provenance metadata, for example signed manifests in the style of the C2PA specification, records how an asset was made and edited. Both help, and neither is tamper-proof: metadata can be stripped when an image is re-encoded, screenshotted or passed through a platform that discards it; watermarks can be degraded by compression, cropping or deliberate removal, and detectors have false positives and false negatives. Treat them as signals that raise the cost of misuse and aid good-faith attribution, never as proof of authenticity or its absence, and design review processes that do not depend on them alone.

The previous sections point to a clear rule: generation is for content where plausibility is enough and an approximate answer is acceptable. It is the wrong tool when exactness is the requirement. An illustration of the actual checkout interface for a help article should be a real screenshot or rendered from the interface's own design file, because a generator will invent plausible but incorrect buttons, labels and layout. A chart of payment-failure rates must be drawn from data by a charting library, since a generative model has no access to the numbers and will happily produce attractive nonsense. Legally constrained brand assets (logos, regulated disclosures, trademarks) must come from approved files. A diagram with precise arrows, aligned text and consistent geometry is a vector-graphics job. In each case deterministic rendering gives repeatability, accessibility (real text and alt descriptions), testability and clear provenance for free.

The pragmatic hybrid is often best: generate the decorative background or concept illustration, then composite deterministic text, data and brand assets on top, and record both parts in the asset metadata. This is also the safest shape for the support scenario, where a model may produce an illustrative graphic but must not be the source of a number or a status message shown to a customer.

Generation is half of the picture. The same customer-support incident includes the opposite task: a user uploads a screenshot of a payment error and the system must work out what it shows. That task reverses the data flow (pixels in, language out) and brings a different set of risks: misread small text, hallucinated objects, and untrusted instructions embedded in the image. A generator learns p(image | text); a reader must produce text that is grounded in a specific image, which is why the engineering concerns shift from sampling and guidance to grounding, OCR and verification. The next lesson develops those ideas.

1. 1

   The prompt does not draw the image; it is encoded into a conditioning signal (commonly read through cross-attention) that steers a denoiser. Output is not unique, so the same prompt gives different valid images.

2. 2

   Diffusion sampling starts from seeded noise and repeatedly removes it. Latent diffusion runs the expensive loop on a compressed latent (about 48 times fewer values in the worked example) and decodes once at the end.

3. 3

   Know which knob does what: the seed fixes the starting noise, resolution sets the latent shape, steps trade quality for time with diminishing returns, image-to-image strength sets how many steps run, and the guidance scale pushes along ε_c − ε_u (too high trades diversity and naturalness for adherence).

4. 4

   Edit rather than regenerate when only a region should change: inpainting keeps the unmasked area, but latent-space preservation means it can still differ slightly after decoding.

5. 5

   Prompt adherence breaks predictably on counts, spatial relations, brands and legible text; generate the artwork with an empty area and add exact text, logos and brand colours in your own layer.

6. 6

   Write prompts as constraints, verify against a checklist (aspect ratio, count, colours, no pseudo-letters, safe content) and decide accept, reseed, edit or hand-draw. Most of the cost is the denoising loop, so steps and resolution are the levers.

7. 7

   Run generation as an asynchronous job with idempotency, retry rules, safety checks before and after generation, and stored metadata (prompt, settings, seed) for provenance.