How to design text-to-image generation

A text-to-image system takes a sentence and returns images.

The generator that turns random noise into an image is a settled engineering choice. What decides product quality is how the sentence reaches that generator: which model reads the text, how its output enters the image generator, what the guidance knob does at sampling time, how to measure any of it, and what one image costs.

In this lesson, we’ll draw the system end to end, derive classifier-free guidance from Bayes’ rule, price an image, and explain why “a red cube on a blue sphere” often comes out purple. By the end you’ll be able to say which model should read the text and why, defend guidance as an operating point instead of a fix, and gate quality on the two axes that actually trade off.

The contract: a sentence goes in, say "a red cube on top of a blue sphere, in a sunlit room". Four PNG images at 1024 × 1024 pixels come out about six seconds later, all four different from one another. Everything below exists to make the images match the sentence.

The machinery this chapter stands on

The generator is a latent diffusion model, which has three parts: a compressor, a denoiser, and a loop that runs the denoiser.

The VAE compresses the image. A variational autoencoder (VAE) is a pair of small networks trained together. Its encoder squeezes a 1024 × 1024 image down to a small grid of numbers, here 128 × 128 positions with 4 numbers each. That compressed grid is a latent. Its decoder expands a latent back into pixels. Working in the latent instead of in pixels is what makes the whole thing affordable: 128 × 128 is 16,384 positions against a million pixels, about 64x fewer.

Patching shrinks it again. Before the denoiser sees the grid, a patch size of 2 groups each 2 × 2 block of latent cells into a single token: 128 × 128 cells become 64 × 64 = 4,096 tokens, a further 4x reduction. That token count, not the cell count, is what every FLOP and dollar figure in this chapter is computed from.

(Public checkpoints ship c = 4 numbers per position; c = 16 is a known improvement at no change in token count, so nothing here depends on the choice. The image synthesis chapter covers that.)

The denoiser does the generating. This is the large network. Training takes a real image’s latent, corrupts it with a known amount of random Gaussian noise, and asks the denoiser to predict the noise that was added. How much noise is indexed by a timestep t: near 1 means almost pure noise, near 0 means almost a clean image.

The sampler runs the denoiser over and over. Generation starts from pure noise and subtracts a little predicted noise at a time, about 28 rounds, until a clean latent falls out; the VAE decoder then turns it into an image. Each round is one forward pass through the denoiser, and the count of forward passes is the entire compute bill.

Those three parts leave exactly one thing undecided: how a sentence gets into the denoiser. That path is the conditioning path (“conditioning” is the ML word for extra information a model gets alongside its main input, here the prompt alongside the noisy latent). The questions that decide product quality all live on it: which model reads the text, why the text reaches the denoiser through cross-attention, what guidance does to the sampled distribution, and why attributes swap between objects.

Problem framing

• Input: a free-text prompt, 1–200 words, from an open-domain population. Open-domain means no fixed vocabulary and no category list; you do not get to restrict what people ask for.
• Output: 4 images at 1024 × 1024, in under ~6 seconds.
• Constraints: cost per image far below revenue per image; nothing illegal or defamatory leaves the box; the same prompt is reproducible on request but the four samples are not identical.

Why it is hard: there is no label. There is no “correct” image for “a lonely lighthouse in a storm.” So the objective is a distribution match: produce images that look like plausible draws from the space of images that fit the sentence, not reproduce one target. Every metric you can compute is a proxy for a judgment you cannot.

Prompt adherence is the term this chapter uses constantly: how faithfully the image contains what the sentence asked for, the right objects, counts, colours on the right objects, and spatial arrangement. It is separate from aesthetics, how good the image looks regardless of what was asked. The two are measured differently and they trade off, which is the single most repeated idea here.

Three reframes carry the design:

Three terms there: FID (Fréchet Inception Distance) is the standard one-number image-quality score, derived under Metrics below. Recaptioning means discarding the text that came with your training images and writing new descriptions with another model. Capacity is loosely how many parameters a network has, and therefore how much it can represent.

The load-bearing claim of this stage is that no single number can express quality, because there is no correct image. Everything downstream depends on it: two-axis human evaluation, gating on per-prompt (VQA) adherence, not FID, and splitting the regenerate metric by whether the prompt was edited.

ML objective

Latent diffusion trains one network to predict the noise added to a latent, given the text. Writing the target as the noise itself instead of the clean image is the epsilon parameterization (“epsilon”, eps, is the conventional symbol for that noise):

z_0    = E(x)                                     VAE encoder, image -> latent
z_t    = sqrt(a_t)·z_0 + sqrt(1 - a_t)·eps        eps ~ N(0, I)
loss   = E_{z_0, c, t, eps}  || eps - eps_theta(z_t, t, c) ||^2

x is a training image and E the encoder, so z_0 is its clean latent. Line 2 corrupts it: eps ~ N(0, I) draws each number independently from a standard Gaussian, and a_t is a fixed schedule saying how much clean signal survives at timestep t. Line 3 is the loss: eps_theta is the denoiser (theta its weights), handed the corrupted latent, the timestep, and the conditioning c; E_{...} averages over images, captions, timesteps and noise draws. In words: on average, the denoiser’s guess at the noise should match the noise actually added.

c is the text conditioning. The entire text-to-image problem is what c is and how eps_theta reads it; everything else is the unconditional picture-maker that ignores text.

Two properties of this loss cause most later problems:

• It is a regression, not a likelihood. The loss measures distance between a prediction and a target; it does not tell you how probable the data is under the model. So the loss value says almost nothing about sample quality. A model with lower validation loss can produce visibly worse images, because the loss is dominated by high-noise timesteps where every model predicts roughly the mean. Never gate a release on training loss.

• It has no term for prompt adherence. The model is asked to denoise; the caption is a hint that makes denoising easier. If the caption is wrong, ignoring it is the correct thing for the loss to learn. That single fact is why caption quality is the dominant data lever, and why classifier-free guidance exists at all: sampling has to force the conditioning to matter more than training did.

(Flow-matching and v-prediction are common alternatives to the epsilon parameterization. They change the arithmetic inside the sampler and shift guidance scales by a point or two, and change nothing about the conditioning argument that is this chapter’s subject.)

Conditioning: the centerpiece

The model that reads the prompt matters more than the model that draws the image. This section gives the mechanism by which text reaches the denoiser, and rules out the alternatives.

Why the text encoder outranks the denoiser

A text encoder turns a sentence into a sequence of vectors, one per token (a chunk of text a bit smaller than a word). Frozen means its weights are downloaded, never updated during your training, and used as a fixed function.

The denoiser never sees your prompt. It sees vectors a frozen encoder produced. Structure the encoder discarded is structure the denoiser cannot recover, at any parameter count, because no path in the architecture could.

The candidates:

• CLIP (Contrastive Language-Image Pretraining) is a pair of models trained to make an image and its caption land near each other in one shared vector space; its text half is the “text tower.”
• OpenCLIP bigG is a larger open reimplementation of the same idea.
• T5 (Text-To-Text Transfer Transformer) is a general language model; its encoder half is what gets used, and “XXL” is its largest size.
• Decoder LM hidden states are the internal activations of an ordinary text-generating model, such as a 7B chat model.

Vocabulary the argument turns on. Context is the maximum tokens the encoder accepts; CLIP’s 77 is a hard truncation, so a long prompt loses its tail. A contrastive objective trains a model to score matching pairs higher than mismatched ones. Pooled means the whole sentence is collapsed into one summary vector before the loss sees it, by averaging or by reading one designated position. Span corruption is T5’s task: blank out random runs of words and reconstruct them. Attachment is which word modifies which; in “a red cube on a blue sphere,” attachment is what says red belongs to cube.

The size argument turns on scale. The denoiser is 2.6B, so swapping a 123M CLIP tower for a 4.7B T5 encoder adds 4.7 / 2.6 ≈ 1.8x the denoiser’s entire parameter count in conditioning capacity. That is why the modern default is a large frozen LM encoder with a comparatively modest denoiser, not the reverse.

The objective argument goes deeper. CLIP’s contrastive loss is computed on a single pooled vector per caption. Nothing in it requires the pooled vector to distinguish these two captions:

"a red cube on a blue sphere"
"a blue cube on a red sphere"

Both contain the same words. A bag of words representation (which words appeared, order forgotten) matches both images about equally well, and a pooled vector is close to a bag of words. So the objective is nearly indifferent between encoding the binding (which adjective goes with which noun) and not, with no gradient pressure either way. CLIP is bad at composition because its training objective never asked for composition.

T5’s span corruption is the opposite: you cannot fill a deleted word without knowing what the surrounding phrase is about, so its token features carry attachment as a side effect of being trained at all.

Cross-attention is the mechanism

Attention is the operation by which one set of vectors looks up information in another. Each vector on the asking side emits a query (Q); each on the answering side emits a key (K) and value (V). Every query is scored against every key, the scores become weights that sum to 1 via a softmax, and each query gets back the weighted average of the values. When a set attends to itself it is self-attention; to a different set, cross-attention.

Conditioning enters through cross-attention layers interleaved with the denoiser’s self-attention: the image asks, the text answers. Its one asymmetry is that Q and K,V come from different places, and there are far fewer of the latter:

Q    from image latent tokens     (64 · 64 = 4,096 at 1024px)
K, V from text encoder outputs    (up to 512)

attention(Q, K, V) = softmax( Q · K^T / sqrt(d) ) · V

Q · K^T scores every query against every key; d is vector width and dividing by sqrt(d) keeps scores from growing with width. Each latent token is a small patch of the image asking the prompt what should be drawn there.

flowchart LR
    P["Prompt<br/>a red cube on<br/>a blue sphere"] --> ENC["Frozen text encoder<br/>FIXED CAPACITY<br/>structure lost here<br/>is lost forever"]
    ENC --> KV["K, V<br/>one vector<br/>per text token"]
    NZ["Noisy latent z_t<br/>4,096 tokens"] --> QQ["Q<br/>one vector<br/>per spatial location"]
    QQ --> XA["Cross-attention<br/>softmax over TEXT<br/>independently per location"]
    KV --> XA
    XA --> OUT["eps prediction"]

    style ENC fill:#1d3557,color:#fff
    style XA fill:#bc6c25,color:#fff
    style OUT fill:#2d6a4f,color:#fff

The prompt reaches the denoiser through exactly one component, and it is frozen. Three consequences fall straight out of the shape:

• Every spatial location queries the whole prompt independently. Latent token i computes its own softmax over the text tokens, with no coupling to location j. The architecture has no representation of “these two regions are the same object.”

• The attention map is a soft segmentation. For each text token, its column of softmax weights reshaped onto the 64 × 64 grid is a heatmap of where in the picture it is applied. This is your primary debugging instrument for the failure modes below.

• Cost is linear in prompt length, not quadratic. Q · K^T has shape 4,096 · 512 (image tokens by text tokens); only one factor is the prompt, so doubling the prompt doubles the work, and in absolute terms costs almost nothing. Doubling the resolution is expensive, because that number grows on both sides of every self-attention product. This is the opposite of the language-model intuition, where every token attends to every other and cost grows with the square of the length. (The denoiser’s own self-attention over its 4,096 image tokens is still quadratic in the image; that is the 5.5 TFLOP attention term under Serving.)

Alternatives to cross-attention, and why they lose

The timestep embedding is the small vector the denoiser already gets telling it how noisy z_t is; broadcast means adding the same vector at every position, which is why it cannot say different things in different places. MMDiT (Multimodal Diffusion Transformer) glues text tokens onto the image sequence so both get updated by the same self-attention; the catch is cost, since the sequence becomes 4,096 + 512 = 4,608 and self-attention costs its square, (4608/4096)^2 ≈ 1.27x per attention layer. FiLM (Feature-wise Linear Modulation) scales and shifts each channel from one summary vector, right for “how noisy” and “how pretty” and useless for “red on the left.” A class label is the one-of-N conditioning older models used (“dog,” “cathedral”), where a single vector genuinely is the whole message.

MMDiT is not a bad idea; it is a real competitor rejected on cost, not capability.

The load-bearing claim here is that the encoder is frozen, so structure it discarded is unrecoverable. That carries three later arguments: spending parameters on the encoder instead of the denoiser, the explanation for attribute binding, and the top rejection in the alternatives table.

Classifier-free guidance, derived

Classifier-free guidance (CFG) is a sampling-time trick that makes the image follow the prompt harder than the trained model would on its own. It has one dial, the guidance scale w; turning it up buys prompt adherence and sells realism, diversity, and colour fidelity. The whole method rests on a classifier you never trained already sitting inside the model.

The setup

Start with one fact. Diffusion’s noise prediction is a scaled score, the gradient of a distribution’s log-density with respect to the data, i.e. the direction the data would move to become more probable. With grad_z the gradient in z, p(z_t | c) the density of noisy latents given the caption, and sigma_t the noise standard deviation:

eps_theta(z_t, t, c)  ≈  -sigma_t · grad_z log p(z_t | c)

So the denoiser, trained only to predict noise, also reports which direction increases probability. The rest is algebra.

Rearrange Bayes’ rule. Take logs, differentiate in z, and move the classifier p(c | z) alone to the left; log p(c) has no z, so its gradient drops out:

grad_z log p(c | z)  =  grad_z log p(z | c)  -  grad_z log p(z)

Convert scores to denoiser calls. Multiply by -sigma_t and replace each score using the fact above. The first term becomes the denoiser run with the caption (eps_cond); the second, the same run with no caption (eps_uncond):

-sigma_t · grad_z log p(c | z)   =   eps_cond  -  eps_uncond

The difference between the conditional and unconditional predictions is exactly the score of an implicit classifier p(c | z), “how well does this image match this caption”, that you never trained and never have to. That is where classifier-free gets its name.

The extrapolation

Now sample not from p(z|c) but from a sharpened distribution that overweights that classifier:

p_w(z | c)  proportional to  p(z | c) · p(c | z)^(w - 1)

Taking logs, differentiating, and substituting the result above gives the formula everyone quotes:

eps_cfg  =  eps_uncond + w·(eps_cond - eps_uncond)

• w = 1 recovers plain conditional sampling (the correction term is zero).
• w = 0 gives unconditional sampling.
• w > 1 raises the implicit classifier to a power, sharpening it: probabilities near 1 stay near 1 while everything else is pushed toward 0.

eps_uncond is a training-time trick, not a second model. During training, replace the caption with a learned null embedding (a single trained vector standing for “no caption”) with probability ~10%. One network learns both behaviours; at sample time you run it twice, once with c and once with null. That habit is conditioning dropout. The 10% is a balance: below ~5% the unconditional branch is undertrained, so the difference vector is noise instead of a score and guidance is unstable at high w; above ~20% you pay real adherence for a term you only subtract.

The tradeoff, derived rather than asserted

Three separate things go wrong as w rises, for three unrelated reasons, which is why no single fix addresses all three.

Diversity falls, because the target is mode-seeking. A mode is a peak of a distribution. p(c|z)^(w-1) concentrates mass wherever the classifier is most confident; raising a number below 1 to a large power shrinks it fast, so the gap between “very confident” and “somewhat confident” widens with every increment of w. At w = 12, four samples from “a golden retriever” are four near-copies of the most prototypical one.

Realism falls, because the tilted distribution is not the data distribution. The data manifold is the thin, curved sheet inside the space of all latents on which real images live; almost every other latent decodes to garbage. Multiplying two densities (tilting one by the other) produces peaks that need not sit on that sheet, so the sampler walks off it.

Saturation appears, because the step size assumes a calibrated eps. The sampler was built assuming eps_theta has roughly unit variance per component. CFG adds the difference vector w times, so its magnitude grows with w:

|| eps_cfg ||  ≈  || eps_uncond ||  +  w · || eps_cond - eps_uncond ||

An over-large eps makes each step over-shoot toward the predicted clean latent, repeated over-shooting drives the latent’s numbers out of the range the VAE decoder saw in training, and the decoder maps those to clipped, blown-out colours. The “deep-fried” look at w = 15 is a decoder running out of distribution, not an aesthetic choice.

One measured sweep of w on a fixed 2,000-prompt set (read it for the disagreement between columns, not the absolute values). VQA adherence decomposes the prompt into yes/no questions and scores the fraction a vision-language model answers correctly; the last column is the fraction of raters preferring that setting over w = 5:

Three metrics, three optima: FID at 2.0, CLIPScore at 12.0, humans at ~4–5. Ship the FID-optimal w = 2 and only 31% of raters prefer it to w = 5; ship the CLIPScore-optimal w = 12 and only 9% do. That table is the argument against reporting one number, and it is also the answer to “what guidance scale should I use”: the one that maximizes the metric you are actually paid for.

flowchart TD
    W["Raise guidance w"] --> S1["p of c given z raised to w-1<br/>-> mode-seeking"]
    W --> S2["tilted product density<br/>-> modes off the data manifold"]
    W --> S3["norm of eps grows ~linearly in w<br/>-> sampler over-shoots"]
    S1 --> D1["DIVERSITY falls<br/>4 samples become 4 copies"]
    S2 --> D2["REALISM falls"]
    S3 --> D3["SATURATION<br/>VAE decodes out-of-range latents"]
    W --> A1["ADHERENCE rises<br/>up to a plateau"]

    style W fill:#1d3557,color:#fff
    style A1 fill:#2d6a4f,color:#fff
    style D1 fill:#bc6c25,color:#fff
    style D2 fill:#bc6c25,color:#fff
    style D3 fill:#9d0208,color:#fff

Raising w fires all four arrows at once, so there is no setting that gives the green box without the orange ones. The choice is an operating point, not a fix.

The pattern across the first three rows of the table below is one idea: the weaker the conditioning signal, the more you have to amplify it. Flow-matching learns a straight-line velocity field instead of a noise estimate and needs less guidance; guidance distillation trains a model to imitate a guided model in a single pass.

Two mitigations before lowering w:

• Guidance rescale. After computing eps_cfg, rescale it to match the standard deviation of eps_cond. This cancels the norm growth above without touching the direction, so you keep adherence and lose the saturation. Blending at ~0.7 instead of replacing outright is empirical: full rescaling costs a little adherence.
• Guidance interval. Apply CFG only in the middle band of timesteps. At very high noise the conditional and unconditional predictions barely differ, so guidance is wasted; at very low noise it mostly amplifies texture into crunch. Restricting CFG to t in [0.10, 0.85] leaves 20 guided steps of 28, so NFE drops from 56 to 48, roughly a 14% saving. Serving below prices the default (CFG at every step), so turning the interval on is a saving you re-derive.

One guided step, with the rescale correction applied, is the teaching point:

# two denoiser calls, batched as one forward of size 2
eps_c, eps_u = model(z_t, t, [c_text, c_null])
eps = eps_u + w * (eps_c - eps_u)          # the CFG extrapolation

# guidance rescale: eps.std() grows with w, but the sampler was built for
# eps_c.std(), so divide the growth back out (blend at 0.7)
if eps.std() > 0:                          # guard the divide-by-zero below
    scaled = eps * (eps_c.std() / eps.std())
    eps = 0.7 * scaled + 0.3 * eps

One caution: the ML objective says ignoring a bad caption is loss-minimizing, so eps_c == eps_u is an input the model is trained to produce. That makes eps.std() == 0 a real input, not an edge case to wave away, hence the guard above (skip it, since rescaling is a no-op then anyway).

What CFG costs

CFG doubles the number of function evaluations. The NFE is the count of forward passes to make one image, the standard unit of cost because everything else is rounding error. 28 sampling steps become 56 forward passes, the single largest line item under Serving.

You do not pay double latency. The conditional and unconditional passes are independent, so they batch into one forward of size 2: exactly 2x in FLOPs, nearly free in wall-clock on an under-utilized GPU.

The real fix for the FLOPs is guidance distillation: train a one-pass student that takes w as an input to imitate the two-pass teacher. Combined with step distillation (a student trained to take one big sampling step where the teacher took several), it is how 4-step models exist.

The load-bearing claim is that eps_cond - eps_uncond is a usable estimate of the implicit classifier’s score. That is what makes guidance a principled sharpening, not a hack, and what predicts all three failure directions at once. If the unconditional branch is undertrained, the difference is noise, guidance amplifies it, and none of this holds; that is why the 10% dropout floor exists.

Data and labels

Training data has two halves: which image-text pairs survive filtering, and what their text says. The second decides model quality.

The pairs

The raw material is image-text pairs scraped from the web, usually the image plus its alt-text (the description an HTML page supplies for screen readers). On the order of 2–5 billion pairs after collection, filtered hard. The filters, in order of how much they pay off, compound: ~55% then ~60% already cuts to about a third of the corpus.

An aesthetic predictor is a small model trained on human ratings that scores how good-looking an image is. Perceptual dedup is deduplication by visual similarity instead of exact file match, so a photo republished at three sizes counts once; the failure it prevents is memorization, a model reproducing a training image recognizably. CSAM is child sexual abuse material, removed by matching against hashes of known material, not any classifier. A stock overlay is the semi-transparent watermark stock sites stamp across previews.

Aesthetic filtering is a fine-tuning tool, not a pretraining tool. Filter hard at pretraining (the long, broad first run) and you delete most of the world’s visual diversity, diagrams, product photos, ordinary rooms, and the model loses the ability to render anything unglamorous. The standard recipe is broad pretraining then a short high-aesthetic fine-tune, because you cannot recover coverage (the range of things the model can render) that you never trained on, but you can always add polish later.

Caption quality is the dominant lever

Raw alt-text is terrible. It is written for accessibility, for SEO (writing text to rank in search instead of describing anything), or for nothing:

alt="IMG_20190412_113255.jpg"
alt="Click here to buy"
alt="dog"
alt="Golden retriever puppy sitting on a red picnic blanket in a park,
     shallow depth of field, late afternoon light"      <- rare

Since the loss has no adherence term, an uninformative caption makes ignoring it optimal. A model trained on alt-text learns to condition weakly, because weak conditioning was loss-minimizing on most of its data.

The fix is synthetic recaptioning: run a VLM (vision-language model, image in, text out) over the whole corpus and have it write a dense, structured description of each image. Sweeping the mixture ratio on the same architecture and compute, the first two columns rise with more synthetic captions, but the third goes the other way, which is the whole point. Rare proper-noun recall is the fraction of prompts naming a specific landmark, brand, or character that the model renders correctly:

Keep a slice of original alt-text. VLM captions describe what is visible and rarely name it: alt-text says “Eiffel Tower,” a VLM says “a tall iron lattice tower at dusk.” Drop alt-text entirely and proper-noun recall falls from 0.72 to 0.31, losing exactly the vocabulary users type. The 10% slice also keeps the short, keyword-ish style of real prompts inside the training distribution. The mix works only because the two sources are not nested: alt-text is the only place rare proper nouns live, synthetic captions the only place dense grounded description lives.

What recaptioning costs

A forward pass through a transformer costs about 2 · parameters · tokens (each parameter does one multiply and one add per token); training costs roughly 3x that, adding a backward pass. Applying that rule to both sides gives an order-of-magnitude estimate:

• Captioning: 500M images through a 7B VLM (~600 input + 120 output tokens ≈ 10 TFLOP each), with a 3x penalty because text decode is memory-bound, is roughly 14,000 GPU-hours, ~$35,000, about 6 days on 96 H100s.
• Pretraining it feeds: a 2.6B denoiser over 2 billion image-steps at ~80 TFLOP each (3 × the 26.8 TFLOP forward pass priced under Serving) is roughly 149,000 GPU-hours, ~$372,000.

$35,000 / $372,000  ≈  9%

Recaptioning is about 9% of the pretraining budget, and it moves prompt adherence more than any architecture change the same money could buy. The ratio stays near 8–12% across a wide range of assumptions. (Pricing the training run on the parameter term alone understates it: attention is 5.5 / 26.8 ≈ 21% of the real per-pass number, so the honest run is ~1.26x larger, which if anything flatters the recaptioning ratio.)

Two questions have no technical workaround and delete a section instead of adjusting it: whether you have the legal right to train on the crawled images at all, and whether the captioner’s licence permits training a competing model on its output. A captioner that writes only English also silently narrows the model to English prompts.

Training

The image synthesis chapter covers the shared latent-diffusion mechanics: noise schedules, samplers, the autoencoder. Five parts are specific to text-to-image:

• Resolution curriculum. A curriculum presents easy work before hard. Train at 256px until the model reliably puts the right kinds of things in the picture, then 512, then 1024. Cost scales with token count, which scales with area: 256px is 256 tokens against 4,096 at 1024px, 16x per step. Spending 80% of steps at 256 and 20% at 1024 costs 0.8·(1/16) + 0.2·1 = 0.25, a 4x saving at no measured quality cost, because composition is learned at low resolution and only texture needs the high one.

• Aspect-ratio bucketing. Center-cropping a square out of a wider image teaches the model that heads are cropped and text is cut off, because that is what its training images looked like. Instead, bucket the data: sort every image into one of a handful of aspect-ratio groups sized for roughly constant token count (1024×1024, 1152×896, 1344×768 all near a million pixels), and draw each batch from a single bucket so the GPU never pads.

• Conditioning dropout at 10% for CFG, plus independent dropout of any secondary conditioning signals so each can be supplied or omitted separately at generation time.

• EMA of the weights. An exponential moving average is a second copy of the weights continuously nudged toward the live weights, here with decay ~0.9999 (each update moves it one ten-thousandth of the way), so it averages over the last several thousand steps. Sample quality is visibly better from the EMA copy; budget the VRAM (the card’s onboard memory) for a second copy.

• Micro-conditioning on nuisance variables. A nuisance variable is a property of a training image you do not want associated with the caption. Feed original resolution, crop offset, and aesthetic score in as extra conditioning. Then the model can explain away “this is blurry because it was upscaled” instead of learning that blur is a property of the caption, and you can ask for high aesthetic at generation time without having filtered the training set for it.

The 4x curriculum saving rests on composition being learned at low resolution. If high-resolution training taught composition too, the curriculum would be a quality regression, and the pretraining budget above would roughly quadruple.

Metrics

The measurement stack has three layers: what you can compute offline, what only humans can judge, and which online signals separate the two failure axes.

Offline: two axes, never one number

FID (Fréchet Inception Distance) is the standard headline number. Push 50,000 generated and 50,000 real images through Inception-v3 (an old classifier) and take its feature vectors; fit a Gaussian to each set (mean mu, covariance S); measure the distance between the two Gaussians. Lower is better; 0 means statistically indistinguishable.

FID = || mu_g - mu_r ||^2  +  Tr( S_g + S_r - 2·(S_g · S_r)^(1/2) )

(g generated, r reference, Tr the trace, the sum of a matrix’s diagonal.) Four things are wrong with it:

1. No per-prompt semantics. It cannot tell whether this image matches this prompt. A model that ignores prompts and generates beautiful images from the reference distribution scores excellently.
2. It moves with the reference set. FID against COCO and against a curated aesthetic set rank models differently. An FID without its reference set is not a number.
3. Inception features are ImageNet-shaped. They over-weight object texture and under-weight layout, faces, and text.
4. It prefers low guidance (the sweep above), so optimizing FID actively degrades what users notice.

CLIPScore is 100 · cos(CLIP_img(x), CLIP_txt(c)): run the image and caption through CLIP’s two halves and take the cosine similarity (1 when the vectors point the same way, 0 when unrelated). It is cheap and correlates with adherence, but it is the same model family whose pooled objective could not represent binding, so it is close to blind to exactly the errors you most need to catch: attribute swaps, red cube and blue sphere trading colours. It also saturates; in the sweep above, 33.4 and 33.6 are a tie, and everything above ~33 is noise.

VQA-based adherence is the one to lead with. Decompose the prompt into atomic yes/no questions, ask a VLM each, and score the fraction correct. “Atomic” means each question checks one claim, so a wrong answer localizes the failure:

prompt    "a red cube on top of a blue sphere, in a sunlit room"
questions is there a cube?              yes
          is the cube red?              no    <- caught
          is there a sphere?            yes
          is the sphere blue?           no    <- caught
          is the cube above the sphere? yes
          is the room sunlit?           yes
score     4/6 = 0.67

Both caught errors are attribute-binding errors, the objects all present and only the colours wrong, exactly what CLIPScore would pass. This catches binding, counting, and spatial relations. Cost is one VLM call per question, so run it on a fixed few-thousand-prompt eval set, not on every user image.

Human preference is the decision metric. Show a rater two images and force a choice (pairwise forced choice), then aggregate with Bradley-Terry, the standard model for turning many pairwise wins into a per-model strength score (an Elo rating is that score on the chess scale). Size the study instead of guessing: the standard sample-size formula for detecting a 55/45 win rate against 50/50 (at the usual 5% significance and 80% power) gives about 800 comparisons. At 3 raters and ~$0.05 each, that is about $120 per model comparison, cheap enough to be the release gate. People skip human evaluation assuming it is expensive; at this scale it is cheaper than the GPU time that produced the samples.

Always collect preference on two separate questions, “which follows the prompt better” and “which looks better,” because those have different optima. A single “which do you prefer” collapses the axes. This is also the trap reward models walk into: a model trained on “which looks better” comparisons raises the aesthetic score while lowering adherence, because raters comparing two images rarely re-read the prompt, so “better” quietly means “prettier.”

Online metrics and A/B

Live traffic teaches what the offline stack cannot. An A/B test sends some users the current system and some the new one and compares outcomes. The two middle rows below are the same event split two ways, and the split is what makes it diagnostic.

Splitting regeneration by whether the prompt changed separates the two failure axes using a signal you already log. It tells you whether to work on the encoder or the sampler.

A few design notes: randomize by user, not request (within-session outcomes are correlated; per-request assignment manufactures significance). A new model changes the style prior, so existing users regress on contact; read the new-user cohort separately, and if new users prefer it and existing users do not, that is a migration problem, not a quality one. Novelty effects (the temporary lift from anything changing) run about two weeks, so do not call a win before then. Treat safety metrics as a gate, not a tradeoff: blocked rate (requests the filters refuse) and leak rate (prohibited content that got through) are pass/fail, not part of the preference arithmetic.

Everything offline is calibrated against human preference collected on two questions. If raters cannot tell the two questions apart, or the rater pool does not resemble the user base, the whole stack is anchored to nothing.

Serving

The request path runs from prompt to delivered image. It has two independent safety checks, one before generation and one after.

flowchart TD
    P([Prompt]) --> TXTMOD{Text moderation<br/>classifier}
    TXTMOD -->|block| REJ([Refuse + reason])
    TXTMOD -->|pass| ENH[Optional prompt<br/>expansion · small LM]
    ENH --> ENC["Text encoder service<br/>T5-XXL · 9.4 GB<br/>cache by prompt hash"]
    ENC --> Q[[Request queue<br/>priority by tier]]
    Q --> GPU["Sampler pool<br/>2.6B DiT · 28 steps<br/>CFG batched as 2"]
    GPU --> VAE[VAE decode<br/>latent -> 1024px]
    VAE --> IMGMOD{Image NSFW<br/>classifier}
    IMGMOD -->|block| REJ
    IMGMOD -->|pass| FACE{Public-figure<br/>face match}
    FACE -->|match| REVIEW[Hold / degrade]
    FACE -->|clear| WM[C2PA manifest +<br/>invisible watermark]
    WM --> CDN([CDN])

    style TXTMOD fill:#bc6c25,color:#fff
    style IMGMOD fill:#2d6a4f,color:#fff
    style GPU fill:#1d3557,color:#fff
    style REJ fill:#9d0208,color:#fff
    style CDN fill:#2d6a4f,color:#fff

A request queue holds arriving work until a card is free, with priority by paying tier. Prompt expansion is an optional small-LM rewrite of a terse prompt into a richer one. Hold or degrade is the middle option between shipping and refusing: send to human review, blur, or return without the face. C2PA is an industry standard for a signed record of how a file was made, and a CDN is a fleet of caches near users that serves the finished image.

The text encoder is a separate service, for VRAM, not latency. Encoding 128 tokens through T5-XXL is 2 · 4.7e9 · 128 ≈ 1.2 TFLOP, about 4 ms, negligible against 5 s of sampling. The real reason is memory: the encoder is 9.4 GB in bf16 (16-bit floats, 2 bytes per parameter), 12% of an 80 GB card. Splitting it off leaves 80 − 5.2 = 74.8 GB free for the sampler’s batch size instead of 80 − 9.4 − 5.2 = 65.4 GB, so the sampler fleet gets 74.8 / 65.4 ≈ 14% more batch for free.

The image classifier is the control; the text classifier is a mitigation. A control checks what actually happens; a mitigation checks what someone appears to intend. The prompt filter catches stated intent and is defeated by euphemism; the output filter looks at what was actually produced, one CLIP-class pass at ~0.6 ms against 5 s of sampling. There is no cost argument for skipping it.

Cost per image

The reference backbone is a 2.6B-parameter DiT (Diffusion Transformer: the denoiser is transformer blocks instead of the older U-Net convolutional design), width d = 2048, 40 layers, patch 2, turning the 128 × 128 latent into 4,096 tokens.

parameter term per pass   2 · 2.6e9 · 4,096                 =  21.3 TFLOP
attention term per pass    4 · T^2 · d over 40 layers        =   5.5 TFLOP
per pass                                                     =  26.8 TFLOP
NFE                        28 steps · 2 (CFG)                =  56
per image                  56 · 26.8                         =  1,501 TFLOP

H100 bf16, ~300 TFLOP/s effective  ->  ~5.0 s/GPU  ->  ~$0.0035/image at $2.50/GPU-hr

The parameter term touches every parameter once per token, so it grows linearly in tokens; the attention term grows with the square of the sequence length T, which is why resolution is expensive.

The rest of the pipeline is rounding error. The VAE decode is 2.2 TFLOP (0.15% of the denoiser: a ~50M-parameter convolutional decoder run once against a 2.6B transformer run 56 times); the safety classifiers are ~0.012%; egress (the charge for data leaving the cloud) is ~1.5 MB per image at $0.09/GB, about $0.000135. All in it is ~$0.0036 per image, and at a realistic 60% fleet utilization (the fraction of rented GPU-seconds doing useful work) every figure grows by 1/0.6 ≈ 1.67x to ~$0.0060, so four images cost under three cents, and 96% of it is the denoiser.

Now the levers. Read the third column against $0.0060 (the number you actually pay), not against the 100%-utilization $0.0036. fp8 is an 8-bit float, half the width of bf16, kept accurate with per-tensor scaling (each weight matrix gets its own multiplier); p99 latency is the time by which 99 of 100 requests finish.

The 768 row is the one people get wrong: tokens fall from 4,096 to 2,304, only 1.78x, but the attention term falls quadratically, so per-pass cost falls 26.8 / 13.7 ≈ 1.95x, more than the token count alone suggests.

The right architecture is two tiers. A 4-step distilled model renders a preview grid in ~0.7 s at $0.0011 each, the user picks one, and the full 28-step model renders the keeper:

two-tier    4 previews · $0.0011  +  1 keeper · $0.0060  =  $0.0104
single-tier 4 full renders · $0.0060                     =  $0.0240   -> 2.3x cheaper

It is also 7x faster to first pixel (0.71 s against 5.0 s), which matters because most generations are discarded before anyone looks closely.

That the denoiser is 96% of the cost is load-bearing: every lever targets it, and the two-tier design exists only because of it. If the classifiers or the VAE were a comparable share, the lever ranking inverts and the “no cost argument for skipping the output classifier” claim under Safety loses its arithmetic.

Failure modes

The four characteristic failures are all consequences of the conditioning path, and each can be explained mechanistically.

Attribute binding

Attribute binding is attaching each property to the right object: red to the cube, blue to the sphere. The mechanism is exactly the cross-attention architecture above.

PROMPT   "a red cube on top of a blue sphere"

OBSERVED across 8 samples
  3x   blue cube, red sphere            attributes swapped
  2x   purple cube, purple sphere       attributes merged
  2x   red cube, blue sphere            correct
  1x   red cube, red sphere             one attribute duplicated

cross-attention mass, averaged over layers, step 12 of 28:
  token "red"      0.31 on cube region   0.27 on sphere region
  token "blue"     0.29 on cube region   0.33 on sphere region
  token "cube"     0.58 on cube region   0.06 on sphere region
  token "sphere"   0.04 on cube region   0.61 on sphere region

Nouns localize; adjectives do not. “Cube” puts 0.58 on the cube and 0.06 on the sphere; it knows where it goes. “Red” puts 0.31 and 0.27, spread across both, essentially indifferent. Two mechanisms compound:

1. The encoder never bound the adjective. A contrastively-trained encoder’s feature for “red” barely encodes what it modifies, so its value vector is roughly “redness,” unattached to any noun.
2. The architecture cannot bind it either. Each spatial location softmaxes over the prompt independently, so nothing prevents the sphere region from drawing on “red.”

The failure is not a bug in the denoiser. It is the absence of any component whose job is binding. Four mitigations, in descending order of how much they help:

1. A token-level-trained encoder (T5-style). The largest effect by far.
2. Attention-map regularization at sample time, pushing tokens in the same noun phrase to overlap and different noun phrases to separate. (Regularization is an extra penalty that steers a model away from an unwanted behaviour.)
3. Explicit region conditioning: the user or a layout model supplies bounding boxes, each with its own text. Sidesteps the problem by giving spatial structure its own channel.
4. Prompt rewriting into separate clauses. Helps least; it does not change the mechanism.

Counting

PROMPT   "exactly seven apples on a wooden table"
OBSERVED 5, 6, 6, 8, 6, 9, 6, 7    (one correct in eight)

Nothing on the conditioning path carries a count the denoiser can check, and nothing in the sampling loop can count; the model generates a texture of apples at a plausible density and stops. Training data compounds it, since captions are rarely numerically accurate. Reliable counting needs either an external layout stage that places N boxes, or a verifier loop that counts the output and resamples. Both are system design, not model design, which is why this is harder to fix than binding even though it is simpler to describe.

Text rendering

Text rendering fails for two independent reasons. The image synthesis chapter traces the autoencoder half; this chapter owns the conditioning half.

PROMPT   "a neon sign that says OPEN LATE"
OBSERVED "OPEN LATF"  "OPFN LATE"  "OPEN LAIE"  "0PEN LATE"

The errors are not random noise; they are plausible letters in the right slots. The model knows a sign has eight glyph-shaped things in a row and roughly which shapes, and is wrong on one or two. That is what you get from conditioning that carries the idea of the string but not its characters: a subword tokenizer (the standard scheme that splits words into common fragments, not letters) turns LATE into one or two IDs, and the embedding for each encodes a word, not a sequence of letter shapes. The denoiser is not misdrawing letters it was given. It was never given letters.

There is also a resolution floor. A sign at 200px of a 1024px image with 8 characters is 25px per character; the VAE downsamples 8x, so ~3.1 latent cells per character, against a threshold of roughly 4:

• Under ~4 cells per character, the autoencoder ceiling binds and nothing on the conditioning path can help.
• Above it, the errors are conditioning errors, fixed by a character-aware encoder run alongside the semantic one, a byte-level model such as ByT5 whose tokens are individual bytes, so the letters really are in the conditioning.

Diagnose which regime you are in first: encode a real photo of the sign through the VAE and decode it straight back, no diffusion, and measure the character error rate. If that round trip is already high, the text encoder is not your problem.

Spatial relations

PROMPT   "a cat to the left of a dog"
OBSERVED correct in ~55% of samples; near chance for "behind", "under"

Position reaches the denoiser only through the content of text-token values; there is no spatial channel. “Left” is a word whose embedding must somehow bias a softmax over 4,096 positions into a half-plane, and nothing trained it to. Layout-conditioned variants fix this by construction; prompt engineering does not.

The rest

A pose estimator locates body joints, and its confidence collapsing is a cheap detector for hand and limb errors. Style collapse is the diversity loss above showing up as four near-identical images in one grid. NaN (“not a number”) is what a float computation produces on overflow or divide-by-zero; at fp8’s narrow range a guidance-inflated eps can overflow, and one NaN spreads, so you check before decoding.

These are architectural absences, not training deficiencies: no component whose job is binding, none that carries a count, none that carries position. The attention-map evidence is what proves it; an undertrained model would put attention mass in the wrong place, not spread it evenly across both regions.

Safety

Three problems get grouped under “safety,” and only the first has a clean technical answer.

Prohibited content. Runs at two points. At ingest: classifier plus hash matching for known illegal material, before the data reaches a training node, logged and auditable. At inference, on both sides: a text classifier on the prompt and an image classifier on the output. The prompt filter is a mitigation; the output filter is the control. A prompt filter is walked around with euphemism, misspelling, or a foreign language; an output classifier looks at the pixels that would actually ship. Measure them separately: a rising image-filter catch rate means the text filter has been figured out, which a combined number would hide.

Likeness is generating a recognizable image of a real person without consent. Prompt-side: detect public-figure names. Output-side: run a face embedding on any detected face (a vector built so two photos of the same person land close) and match against a gallery of known public figures. The output check is the one that matters, because “the 45th president” and a plain physical description both route around a name blocklist. The threshold is a business decision with arithmetic attached:

Lowering it catches more and annoys more people; raising it does the reverse. State the operating point and who chose it. The headshot generation chapter builds this machinery out, since its entire purpose is generating one specific person’s face.

Style mimicry. The architecture cannot resolve this. You can blocklist artist names, and it accomplishes little:

blocked   "in the style of <living artist>"
works     "thick impasto brushwork, swirling cobalt night sky,
           cypress silhouette, heavy visible palette-knife texture"

Removing the name does not remove the learned style, because the name was never the mechanism; it was an index into a region of the model that description also indexes. The only intervention that works is exclusion at training time, honored via an opt-out registry, and even that misses reproductions in the corpus the registry does not cover. It is a licensing question, not one a classifier resolves.

Provenance is the record of where a file came from. A C2PA manifest is signed metadata saying an image was machine-generated, so it dies on the first screenshot. An invisible watermark is a pattern in the pixels a detector can read but a viewer cannot; it survives resize, crop, and JPEG re-compression, but dies against a screenshot re-encoded through another generative model. Provenance is a supply-chain signal for good-faith platforms, not an adversarial defense, and overclaiming it is a bad look.

The output classifier being a control while the prompt classifier is only a mitigation is why the output check is never cut under cost pressure: cutting a control to save 0.012% is not a cost decision.

Alternatives considered and rejected

Each rejection is a number, not a preference.

A GAN (generative adversarial network) trains a generator against a discriminator that tells real from fake, in a single forward pass; mode coverage (how much of the real variety it produces) is where it fails at open-domain scale. An autoregressive image model turns the image into discrete tokens via a VQ tokenizer (vector quantization: map each patch to the nearest entry in a learned codebook) and predicts them one at a time; conditioning each token on all previous ones is why it composes well, and doing it 4,096 times in strict sequence is why it is slow. A cascade generates a small image, then enlarges it through a chain of super-resolution models.

Conclusion

The generative backbone, latent diffusion, is a settled choice. What decides the product lives on the conditioning path, and three facts carry the whole design:

• The text encoder is frozen, so structure its training objective discarded is gone before the denoiser sees the prompt. That is why you spend parameters on a large token-level encoder (T5-XXL) instead of on the denoiser, and why CLIP’s pooled contrastive objective cannot bind “red” to “cube.”
• The training loss has no adherence term, so a model learns to ignore weak captions. That is why recaptioning is ~9% of the budget and the highest-leverage spend, and why classifier-free guidance has to exist at all: sampling forces conditioning to matter more than training did. Raising w buys adherence and sells diversity, realism, and colour, for three unrelated reasons, so guidance is an operating point chosen against the metric you are paid for.
• No single number measures quality, so you gate on two-axis human preference and per-prompt VQA adherence, never FID, which prefers the wrong guidance scale.

The serving diagram above is the system glued together: two safety checks around a sampler that is 96% of the cost, an encoder split off for VRAM, and a two-tier preview/keeper design that follows directly from where the cost is. Four images land under three cents, and every failure mode, binding, counting, text, and spatial position, traces back to an architectural absence on that same conditioning path.

One line to remember: the model that reads the sentence decides adherence, the loss never asked for adherence, and no single number can score it, so design and measure on the conditioning path, not the denoiser.

Further reading

• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (2022), the latent-diffusion / Stable Diffusion architecture.
• Ho & Salimans, Classifier-Free Diffusion Guidance (2022), the original derivation of CFG.
• Saharia et al., Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen, 2022), evidence that a large frozen text encoder outranks a larger denoiser.
• Betker et al., Improving Image Generation with Better Captions (DALL·E 3, 2023), the synthetic recaptioning result.
• Podell et al., SDXL (2023), micro-conditioning and aspect-ratio bucketing.
• Esser et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (Stable Diffusion 3, 2024), MMDiT joint attention and flow-matching.
• Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (2017), the FID metric.
• Lin et al., Common Diffusion Noise Schedules and Sample Steps Are Flawed (2023), guidance rescale and the saturation fix.

Next: Personalized headshot generation, what changes when the subject is a specific person and the model is built per user.