How to design text-to-image generation

A sentence goes in, 4 images at 1024×1024 come out ~6s later. The generator (latent diffusion) is a settled choice; product quality lives entirely on the conditioning path, how the text reaches the denoiser.

The machinery

• Latent diffusion = VAE compressor + denoiser + sampler loop.
• VAE: encoder squeezes 1024×1024 → 128×128×4 latent (~64x fewer positions); decoder expands back. Patch size 2 groups 2×2 cells into one token → 64×64 = 4,096 tokens (a further 4x). Token count, not cell count, drives every FLOP/dollar.
• Denoiser (large net, a DiT): trained to predict the noise added to a latent at timestep t (near 1 = pure noise, near 0 = clean image).
• Sampler: start from noise, subtract a little predicted noise per forward pass, ~28 rounds, then VAE-decode. NFE (forward passes/image) is the whole compute bill.

ML objective

• Epsilon parameterization: loss = E || eps - eps_theta(z_t, t, c) ||^2, with z_t = sqrt(a_t)·z_0 + sqrt(1-a_t)·eps. c is the text conditioning; the whole problem is what c is and how the denoiser reads it.
• It is a regression, not a likelihood: loss says almost nothing about sample quality (dominated by high-noise steps). Never gate a release on training loss.
• It has no adherence term: ignoring a bad caption is loss-optimal. This is why caption quality is the top data lever and why CFG must exist.

Conditioning: the centerpiece

• The model that reads the prompt outranks the one that draws. The denoiser sees vectors from a frozen encoder; structure the encoder discarded is unrecoverable at any parameter count.
• CLIP’s contrastive loss runs on one pooled vector ≈ bag of words → cannot bind “red” to “cube.” T5’s span corruption is token-level → carries syntax, attachment, negation.
• Swapping a 123M CLIP tower for a 4.7B T5 adds ~1.8x the 2.6B denoiser’s params in conditioning capacity. Modern default: large frozen LM encoder + modest denoiser.

• Cross-attention is the mechanism: image latent tokens emit Q (4,096), text emits K,V (≤512). attention = softmax(Q·K^T / sqrt(d))·V. Each location queries the whole prompt independently, so there is no “these two regions are one object.”
• Cost is linear in prompt length (only one factor of Q·K^T is text); resolution is quadratic. The attention map (a text token’s softmax column reshaped to 64×64) is the primary debug tool.
• Rejected alternatives: pooled-vector-to-timestep (one vector can’t say “red here, blue there”), FiLM (global only, used alongside for timestep/aesthetics). MMDiT (concat text into self-attention) is a real competitor rejected on cost (~1.27x/attention layer), not capability.

Classifier-free guidance (CFG)

• Noise prediction ≈ a scaled score: eps_theta ≈ -sigma_t · grad_z log p(z_t|c). Bayes gives eps_cond - eps_uncond = the score of an implicit classifier p(c|z) you never trained → “classifier-free.”
• Formula: eps_cfg = eps_uncond + w·(eps_cond - eps_uncond). w=1 = plain conditional, w=0 = unconditional, w>1 sharpens.
• eps_uncond is one model run with a learned null embedding; train with ~10% conditioning dropout (below ~5% the branch is undertrained and guidance is unstable; above ~20% you lose adherence).
• Raising w fires all four arrows at once, three unrelated causes:

raise w ─┬─ p(c|z)^(w-1) mode-seeking ─→ DIVERSITY falls (4 samples → 4 copies)
         ├─ tilted product off manifold ─→ REALISM falls
         ├─ ||eps|| grows ~linearly in w ─→ SATURATION (VAE decodes out-of-range)
         └────────────────────────────────→ ADHERENCE rises (to a plateau)

• Three metrics, three optima on a fixed sweep: FID best at w=2, CLIPScore at w=12, humans at ~4–5. Use the w that maximizes the metric you are paid for. Weaker conditioning (CLIP + epsilon) needs w≈6–9; sharp conditioning (LM + flow-matching) ≈3–5; distilled = 1.
• Mitigations before lowering w: guidance rescale (rescale eps_cfg to eps_cond’s std, blend ~0.7; guard eps.std()==0, a real trained input) and guidance interval (apply CFG only for t∈[0.10,0.85], ~14% NFE saving).
• CFG doubles NFE (28→56) but the two passes batch into one forward of size 2: 2x FLOPs, ~free in latency. Real FLOP fix = guidance distillation.

Data: captions are the lever

• Image-text pairs from web alt-text, 2–5B, filtered hard (resolution ~55% → CLIP-similarity ~60% → dedup removes 20–30% → NSFW/CSAM → watermark ~5%).
• Aesthetic filtering is fine-tuning, not pretraining: filter hard early and you delete coverage (diagrams, ordinary rooms) you can never recover. Broad pretrain → short high-aesthetic fine-tune.
• Raw alt-text is terrible → synthetic recaptioning with a VLM. Best mix ~90% synthetic / 10% alt-text: keep the slice because alt-text is the only source of rare proper nouns (“Eiffel Tower”); dropping it drops proper-noun recall 0.72→0.31.

• Recaptioning costs 9% of the pretraining budget ($35k vs ~$372k) and moves adherence more than any architecture change the same money buys.

Training specifics

• Resolution curriculum 256→512→1024 (cost scales with area, 16x). 80% steps at 256 + 20% at 1024 = 0.25, a 4x saving; composition is learned low-res, only texture needs high-res.
• Aspect-ratio bucketing instead of center-cropping (which teaches cropped heads); group images into ~constant-token buckets, one bucket per batch.
• 10% conditioning dropout for CFG; EMA of weights (decay ~0.9999, visibly better samples, budget a second VRAM copy); micro-conditioning on nuisance vars (original res, crop, aesthetic score) so blur isn’t blamed on the caption.

Metrics: two axes, never one number

• FID: distance between Gaussians fit to Inception features of 50k gen vs 50k real. Flaws: no per-prompt semantics, moves with the reference set, ImageNet-shaped features, prefers low guidance. Never gate on it.
• CLIPScore = 100·cos(img,txt): cheap, but same family that can’t bind → blind to attribute swaps, and saturates above ~33.
• VQA adherence (lead with this): decompose prompt into atomic yes/no questions, score fraction a VLM gets right; localizes binding/counting/spatial errors. Run on a fixed eval set.
• Human preference is the decision metric: pairwise forced choice → Bradley-Terry/Elo; ~800 comparisons ≈ $120/model, cheaper than the GPU time. Always ask two questions (“follows prompt” vs “looks better”) — collapsing them is the trap reward models fall into.
• Online: keeper rate (headline), regenerate rate split by prompt-edited (blames adherence) vs unchanged (blames sampling variance) — separates the two axes from a signal you already log. Randomize by user; novelty effects run ~2 weeks; safety metrics are a gate, not a tradeoff.

Serving and cost

• Two safety checks: text moderation before, image NSFW + public-figure face match after. Text filter = mitigation (beaten by euphemism); output filter = control (looks at real pixels, ~0.6ms, never cut).
• Text encoder split into its own service for VRAM, not latency (9.4GB in bf16; splitting frees ~14% more sampler batch).
• Cost/image: 2.6B DiT, per pass = 21.3 (param) + 5.5 (attention) = 26.8 TFLOP; NFE 56 → 1,501 TFLOP → ~5s → ~$0.0036, ~$0.0060 at 60% util. Denoiser is 96% of cost; VAE 0.15%, classifiers 0.012%.
• Levers (vs $0.0060): fp8 → $0.0035; guidance distillation (56→28) → $0.0031; step distill to 4 → $0.0011 (diversity loss); gen@768 upscale → $0.0032 (attention falls quadratically, ~1.95x/pass).
• Two-tier: 4 distilled previews + 1 full keeper = $0.0104 vs 4 full renders $0.0240 (2.3x cheaper, 7x faster to first pixel). Follows directly from the denoiser being 96%.

Failure modes = architectural absences

• Attention mass spread evenly across both regions (not in the wrong place) proves these are missing components, not undertraining.

Safety

• Prohibited content: hash + classifier at ingest; text (mitigation) + image (control) at inference, measured separately.
• Likeness: face embedding vs public-figure gallery; the output check matters because names route around blocklists. Threshold is a stated business choice (0.65 catches ~94% / flags ~2%; 0.75 catches ~81% / flags ~0.3%).
• Style mimicry: name blocklists fail (description indexes the same region); only training-time exclusion works — a licensing question.
• Provenance: C2PA manifest dies on screenshot; invisible watermark survives resize/crop/JPEG but not a re-generative pass. A good-faith signal, not an adversarial defense.

One line

The model that reads the sentence decides adherence, the loss never asked for adherence, and no single number can score it — design and measure on the conditioning path, not the denoiser.