How to design high-resolution image synthesis

Run diffusion straight on pixels at 1024 × 1024 and a single image costs about eight hours of H100 time. That is not slow, it is infeasible. Compressing the image by a factor of eight per side cuts the dominant cost by a factor of about four thousand, and that one decision drives the rest of the design.

In this lesson, we’ll take a small-image generator up to 1024 × 1024 and serve it at scale, a different problem from generating one small image. We’ll work through why pixel-space diffusion is impossible here, how latent diffusion removes the cost, how to choose the network shape and the number of sampling steps, why the standard quality metric stops working above about 300 pixels, and the mechanisms behind the four failures users report most (garbled text, wrong hands, tile seams, and duplicated subjects at wide aspect ratios). By the end you’ll be able to price each design choice in FLOP and dollars, defend latent diffusion over a cascade, and name the mechanism behind each of those failures.

The starting point

Three terms carry the chapter.

A diffusion model is a network trained to remove a small amount of noise from a picture. You apply it over and over: generation starts from pure random noise and walks it down into an image over some number of steps. This family was chosen over the alternatives (see the face-generation chapter) chiefly because it covers the full variety of the training data instead of quietly abandoning parts of it, and none of those reasons change with resolution.

The denoising network is a transformer, the same architecture language models use. It chops its input into a sequence of tokens, fixed-size pieces each represented by a vector of numbers, then runs attention, an operation in which every token compares itself against every other token.

Attention is the constraint. With n tokens there are n × n comparisons, so the cost of attention grows with the square of the number of tokens. Double the tokens, quadruple the attention cost.

The input is a condition describing the image: a set of attributes, or a sentence (the text case is handled in the text-to-image chapter). The output is one 1024 × 1024 image, delivered in under five seconds and for well under a tenth of a cent.

Framing

Pin down the interface and constraints first, because between them they force the structural question the rest of the design answers.

• Input: a condition (attributes, or text).
• Output: 1024 × 1024, and a product that will ask for 1536 × 640 and 2048 × 2048 within a quarter.
• Why it is hard: the compute of the naive design grows faster than the pixel count, and the metric everyone quotes is computed after shrinking the image to 299 × 299.

The three constraints, each with the thing it rules out:

p95 is the value 95% of requests come in under; p50 is the median. Latency is stated as a percentile, not an average, because an average hides the slow tail: one request in twenty taking 30 seconds barely moves the mean, and is exactly what users notice.

The resolution question is really a question about where the compression lives. Attention over pixels is quadratic in pixel count, so at 1024 the model must operate on a compressed representation. There are exactly three places to put the compression: inside the diffusion model, in front of it as a learned autoencoder, or spread across a chain of models. Picking one is the whole design.

Five terms appear in the diagram and are used for the rest of the chapter:

• Downsampling: reducing the resolution of a grid, so fewer positions carry the same picture.
• U-Net: the convolutional network diffusion models originally used, shaped like a letter U: it downsamples through several stages, then upsamples back, with shortcut links across the middle.
• Autoencoder: a pair of trained networks. One compresses an image into a compact numeric code; the other expands the code back into pixels.
• Cascade: a chain of models where a small one produces a low-resolution image and later ones enlarge it.
• Patching: grouping neighbouring cells of a grid into a single token before the transformer sees them.

The three leaf boxes state the whole argument in three numbers: 142,798 TFLOP per forward pass for pixel-space, 5.82 TFLOP per forward pass for latent diffusion, and 525.8 TFLOP per image for the cascade (which runs several smaller forward passes across three models). The rest of the chapter is the argument between those three numbers.

flowchart TD
    Q{"1024 x 1024 target.<br/>Attention is O n^2 in tokens.<br/>Where does the compression live?"}
    Q -->|"inside the model"| A["Pixel-space U-Net<br/>conv downsampling stages,<br/>attention only at low res"]
    Q -->|"in front of the model"| B["LATENT DIFFUSION<br/>a trained autoencoder<br/>compresses 8x per side"]
    Q -->|"across models"| C["Cascade<br/>small base + super-res stages,<br/>each conditioned on the last"]
    A --> A1["still 1,048,576 pixels at<br/>the top of the U-Net.<br/>142,798 TFLOP per forward"]
    B --> B1["16,384 latent cells<br/>4,096 tokens after patching.<br/>5.82 TFLOP per forward"]
    C --> C1["3 models, 3 training runs,<br/>errors compound down the chain.<br/>525.8 TFLOP per image"]

Why pixel-space diffusion at 1024 is hopeless

The obvious design fails on arithmetic, and the obvious escape fails right behind it. That is what makes compression a requirement here, not an optimization.

Take the DiT-XL backbone. DiT is a diffusion transformer: the denoising network is a plain transformer, not a U-Net. XL is the size, described by two numbers used in every estimate below: the width d = 1152 (numbers per token) and the depth L = 28 (stacked transformer blocks). That gives about 445.9 million parameters acting on every token.

Run that network directly on pixels at one token per pixel and you have 1,048,576 tokens. A forward pass has two costs. The token-linear term (every parameter touching every token) grows in proportion to the token count and comes to about 935 TFLOP. The attention term (every token against every other) grows with the square of the token count and comes to about 141,863 TFLOP.

Attention is 99.3% of the cost, 142,798 TFLOP per forward pass. The two terms are equal at roughly n = 6d tokens; at d = 1152 that crossover is 6,912 tokens, and 1,048,576 tokens is 152× past it. The same backbone at 1,024 tokens sits well below the crossover, which is why “attention is quadratic” is true there but not the dominant cost. Four times the resolution per side moves you 152× past the crossover.

Turn that into a bill. Generation runs the network once per sampling step, and twice per step if classifier-free guidance (CFG) is on. CFG makes the output obey the request more strongly by running the network once with the request and once without and extrapolating away from the unconditioned prediction; the only thing that matters here is that it doubles the forward passes. A 30-step sample with CFG is 60 forwards:

60 forwards × 142,798 TFLOP  ≈  8.6e18 FLOP
÷ 300 TFLOP/s (H100 effective)  ≈  7.9 hours per image
× $2.50/GPU-hour ÷ 0.60 utilization  ≈  $33 per image

Eight hours and thirty-three dollars for one picture. The 300 TFLOP/s is the effective rate an H100 sustains end to end, not its 990 TFLOP/s peak; the 60% utilization is the fraction of paid-for GPU-seconds that do useful work, since you pay for the idle time too.

That eight-hour figure already assumes the best available implementation. FlashAttention computes attention in small tiles so the giant table of token-to-token scores never exists in memory all at once; it removes the memory problem but not the arithmetic. Without it the design does not merely cost too much, it does not fit: one score table at n = 1,048,576 in 16-bit numbers is 1,048,576² × 2 bytes ≈ 2.2 TB, for a single attention head of a single layer, against 80 GB of GPU memory.

Why you cannot just delete the attention

The obvious escape is to drop attention and build the network entirely out of convolutions, whose cost grows only in proportion to the pixel count. It does not work, and seeing why pins down what attention buys.

A convolution applies a small window of weights (a kernel) at every position; a 3 × 3 kernel lets each output pixel see only its immediate neighbours. Stack such layers and the receptive field (the region of input an output can depend on) grows only about 2 pixels per layer. Connecting one corner of a 1024 image to the opposite corner would take roughly 1023 / 2 ≈ 512 layers.

A generative model needs exactly that connection: one nose, two eyes, a single light source, one horizon are all long-range dependencies between distant parts of the image. So you need one of two things: a stride greater than 1, which is downsampling and therefore the thing we are choosing where to put; or attention, which is quadratic. There is no third option, and that is why the framing question drives the design.

Latent diffusion

The idea in one sentence: train an autoencoder once, run the entire diffusion process on its compressed output instead of on pixels, and decode to pixels a single time at the very end.

An encoder E maps an image of height H, width W, and 3 colour channels down to a grid of height H/f, width W/f, and c channels. A decoder D maps that grid back to pixels. The compressed grid is the latent, one position in it is a latent cell, f is the downsampling factor per side (at f = 8, a 1024 × 1024 image becomes a 128 × 128 grid), and c is the channel count (numbers per cell). Both networks are trained once and then frozen, so every later diffusion model reuses them unchanged and their training cost amortizes away.

One subtlety decides everything downstream. The compression is 8× per side, but the number of stored values falls by far less than 64×, because the latent trades spatial resolution for channels. At the c = 16 configuration this chapter uses, a 1024 image has 3,145,728 pixel values and 1,048,576 pixel positions; its latent has 262,144 values but only 16,384 cells. So values fall 12× while positions fall 64×, and attention charges you for positions, not for values. That is why c can be raised almost for free and f cannot.

The diagram shows the full path. The grouped box is the autoencoder, trained once and frozen; everything to its right runs on every request. The count falls twice (1,048,576 pixels, then 16,384 latent cells, then 4,096 tokens) and the loop runs around the DiT only. The decoder runs once, at the end.

flowchart LR
    subgraph OFF["Trained once, frozen"]
        IMG(["1024 x 1024 x 3<br/>3,145,728 values"]) --> ENC["Encoder<br/>f = 8"]
        ENC --> LAT["128 x 128 x 16<br/>16,384 cells<br/>262,144 values"]
    end
    LAT --> PATCH["Patchify p = 2<br/>64 x 64 = 4,096 tokens"]
    PATCH --> DIT["DiT<br/>28 blocks, d = 1152<br/>5.82 TFLOP per forward"]
    DIT --> LOOP{"30 steps"}
    LOOP -->|"loop"| DIT
    LOOP -->|"done"| DEC["Decoder<br/>2.2 TFLOP"]
    DEC --> OUT(["1024 x 1024 x 3"])

The 64×, and the 4,096×

The autoencoder removes a factor of f per side, so token count falls by f² = 64×. Attention is quadratic in token count, so its cost falls by the square of that: 64² = 4,096×. Checked exactly, the pixel attention term of 141,863 TFLOP becomes 34.6 TFLOP, precisely 4,096× smaller.

An 8× spatial downsample is a 64× reduction in tokens, and because attention is quadratic, a 4,096× reduction in attention cost. That single line is the reason latent diffusion exists.

Patchification stacks a second factor on top. At patch size p = 2, each 2 × 2 block of latent cells becomes one token, so tokens fall by another 4× (to 4,096) and attention by another 16×. But the token-linear term falls only by the token factor, not its square, so the end-to-end saving lands between the two: per forward, pixel space is 142,798 TFLOP and latent space is 5.82 TFLOP, a 24,546× reduction.

Notice what happened to the balance between the two terms. In pixel space the split is 935 : 141,863, attention-dominated 152 to 1. In latent space it is 3.65 : 2.17, dominated instead by the token-linear term. The compression did not just shrink the bill. It changed which term you are paying, and that matters beyond the money. When cost is proportional to parameters, scaling laws are clean: adding capacity has a knowable payoff, because doubling the parameters doubles the cost instead of doing something quadratic and unpredictable.

What the autoencoder costs, and where a GAN loss belongs

A GAN (generative adversarial network) trains one network against a critic, a second network whose job is to tell generated output from real data. The face-generation chapter rejected that training scheme outright. Inside the autoencoder, the same loss is not merely acceptable but necessary, a good example of a loss being safe or dangerous depending entirely on what else sits in the objective.

The autoencoder is trained with four terms:

• L1: sum of absolute pixel differences between input and round-tripped output. Straight pixel fidelity.
• LPIPS (learned perceptual image patch similarity): compares images through the internal features of a pretrained classifier (VGG, an older convolutional network) instead of pixel by pixel. It tracks human judgements of similarity far better than raw pixel distance.
• PatchGAN: a small adversarial critic that judges local patches for realism instead of scoring the whole image.
• KL: the Kullback-Leibler divergence, pulling the latent distribution toward a standard bell curve. Weighted so lightly (about one part in a million) that it does little beyond keeping the latent’s scale from drifting.

The adversarial term does a job L1 structurally cannot. A loss built on absolute or squared differences is minimized by an average over everything still consistent with the input (the median for L1, the mean for L2) not by any single plausible answer. When several fine textures are equally plausible for a patch (the exact arrangement of pores, threads, gravel), their average is smooth, so the loss actively prefers blur. LPIPS moves the comparison into feature space, which helps, but it is still a regression loss with the same averaging behaviour one level up. Only an adversarial critic supplies pressure to commit to one plausible texture, because an averaged texture is exactly the artifact a critic learns to spot.

The adversarial loss is dangerous when it is the only signal and safe when it is a texture prior on top of a reconstruction loss. In the face-generation chapter the GAN was rejected because mode collapse (the generator abandoning whole kinds of output) carries no penalty. Here the reconstruction term makes ignoring the input impossible: the autoencoder is handed the image and must return it, so drifting away from it costs L1 immediately. Same loss, opposite risk, entirely because of what else is in the objective.

The ceiling the autoencoder sets

Compression is lossy, and how lossy is a hard ceiling on everything built on top: the diffusion model can never produce detail the decoder cannot reconstruct. So measure it before training anything downstream: push the validation set through the encoder and straight back out with no diffusion, then compare originals against round-tripped copies.

The measure is reconstruction FID (rFID). FID compares two collections of images through the internal features of a pretrained classifier; lower is better, zero means indistinguishable. Because both collections here are the same images differing only by the round trip, rFID isolates exactly what the compression destroyed.

Rows 2 and 3 have identical compute and very different ceilings, and understanding why is the highest-leverage idea at this layer. Raising the channel count c does not change the token count at all: tokens come from positions, and c is depth per position. It only widens the projections at the very input and output of the transformer, a rounding error against the 12·L·d² parameters in the body. So raising c from 4 to 16 quadruples the information in each latent cell at essentially zero extra diffusion cost, and cuts rFID from 0.74 to 0.28, a 62% reduction. That is why recent systems use 16-channel latents.

Latent scaling, the one detail that silently breaks everything

The diffusion noise schedule is written in absolute terms. The blending rule x_t = sqrt(abar_t)·x_0 + sqrt(1 - abar_t)·eps mixes the clean input x_0 with noise eps in proportions set by the schedule, and it assumes x_0 has roughly unit variance (variance being the standard measure of spread; its square root is the standard deviation).

Nothing forces a freshly trained encoder to obey that. Suppose it emits latents with a standard deviation of 5 instead of 1. The signal is then 5× larger than the schedule expects while the noise is exactly the size the schedule chose, so the SNR (signal-to-noise ratio) is 5² = 25× what was intended, at every step. The model then never experiences a genuinely high-noise step, even its noisiest step still has the signal shouting through it. High-noise steps are exactly where global layout gets decided, so the model never learns to produce a global layout at all.

The fix is one line: measure the latent standard deviation over the corpus once, and divide every latent by it. That is the origin of the famous 0.18215 constant in public checkpoints, one specific encoder’s measured scale, not a universal number. Skip it and the symptom is distinctive: locally beautiful texture, globally incoherent composition, with nothing erroring and the loss curve looking fine.

Cascaded super-resolution, the alternative

Latent diffusion has one serious competitor, and the decisive objection to it is operational, not arithmetic.

The cascade generates a small image, then enlarges it with a chain of super-resolution (SR) models. Each SR model is itself a conditional diffusion model, taking the previous stage’s output as its condition and producing a larger version. The resolutions climb 64, 256, 1024 while the models get smaller, because the top stage only has to add local detail. Every stage runs entirely in pixel space, so there is no autoencoder anywhere and no compression ceiling. That is the cascade’s whole selling point. To keep the top stage out of hopeless territory, each stage patchifies more aggressively: at p = 2/4/8 the three stages have 1,024, 4,096, and 16,384 tokens.

flowchart LR
    Z(["noise"]) --> B["Base model<br/>64 x 64 pixel space<br/>1,024 tokens · 1.05 TFLOP<br/>60 forwards"]
    B --> L1(["64 x 64"])
    L1 --> NCA1["Noise-conditioning aug<br/>corrupt the LR input,<br/>tell the model how much"]
    NCA1 --> S1["SR 64 -> 256<br/>4,096 tokens · 2.75 TFLOP<br/>30 forwards"]
    S1 --> L2(["256 x 256"])
    L2 --> NCA2["Noise-conditioning aug"]
    NCA2 --> S2["SR 256 -> 1024<br/>16,384 tokens · 12.68 TFLOP<br/>small model, 30 forwards"]
    S2 --> OUT(["1024 x 1024"])

Noise-conditioning augmentation

Every cascade has one unavoidable problem. Each SR stage is trained on real low-resolution images (genuine photographs shrunk down) but deployed on generated ones, which carry the base model’s characteristic artifacts. That is a distribution shift: at run time the model is handed something statistically unlike anything it saw in training, and it compounds down the chain because each stage’s own artifacts become the next stage’s unfamiliar input.

The fix, noise-conditioning augmentation, has two halves: during training, deliberately corrupt the low-resolution input with a random amount of noise; and tell the model how much you added, by passing the level in as an extra condition. The SR model then learns a family of behaviours indexed by “how much should I trust this input,” and at generation time you dial in the level that matches how artifact-laden the base output actually is. The general name for the problem is exposure bias (a model trained only on clean inputs and deployed on dirty ones), and the general fix is the one used here: make the training distribution contain the corruption the model will actually see.

Cost, and the real objection

Sized for their jobs, the three stages cost 62.9 + 82.5 + 380.4 = 525.8 TFLOP per image, against 351.2 for the single latent model, 1.5×. The largest single contribution (380.4 TFLOP) is the top SR stage even though it uses the smallest model, because its token count is fixed by the output resolution and cannot be designed away. Size the SR stages generously instead of shaving them and the gap goes past 4×.

But the FLOPs are not the argument. Compare the two designs on everything else:

The decisive argument is the retrain coupling. Latent diffusion has one training run with one loss curve you can read. A cascade has three, and the second and third are conditioned on a distribution that changes whenever you touch the first. Improve the base model and you now have two SR stages tuned for artifacts that no longer exist, an operational cost you pay every quarter, and it appears in no FLOP table.

The cascade does win at 4K. At 4096 pixels per side, f = 8 gives a 512-per-side latent and p = 2 gives 65,536 tokens, 16× the count at 1024, so 256× the attention cost, and attention is back to dominating. The standard answer there is a hybrid: latent diffusion up to 1024, then a refiner in latent space or a lightweight pixel-space SR stage to reach 4K. That is a cascade whose base happens to be a latent model, and it is what production systems ship.

Architecture: U-Net vs DiT

The U-Net was designed to solve the pixel-space cost problem internally: put attention only where the token count is small enough to afford it. Convolution stages carry the high-resolution work using a locality prior (the built-in assumption that nearby pixels matter most, which is true of images and saves the model from learning it), attention turns on only at 32 × 32 and below, and skip connections carry each downward stage’s detail straight across to the matching upward stage so it survives the trip through the narrow middle.

The DiT does none of that: 4,096 tokens go in, pass through 28 blocks of identical shape, and 4,096 tokens come out. Once a trained autoencoder does the compression externally, the U-Net’s main purpose is already handled, which is why the field moved to transformers.

flowchart TD
    subgraph U["U-Net — compression inside the model"]
        U1["128x128 conv"] --> U2["64x64 conv"] --> U3["32x32 conv + attention"]
        U3 --> U4["16x16 conv + attention"] --> U5["32x32 up"] --> U6["64x64 up"] --> U7["128x128 up"]
        U3 -.->|"skip"| U5
        U2 -.->|"skip"| U6
        U1 -.->|"skip"| U7
    end
    subgraph D["DiT — compression already done by the autoencoder"]
        D1["4,096 tokens"] --> D2["Block 1<br/>attn + FFN + adaLN-zero"]
        D2 --> D3["Block 2 ... Block 28<br/>identical shapes"]
        D3 --> D4["4,096 tokens"]
    end

Four supporting reasons follow. Four terms first: MFU (model FLOP utilization) is the fraction of the hardware’s peak arithmetic rate the model achieves, mostly a matter of shapes, since GPUs run large uniform matrix multiplies far more efficiently than small ragged ones; a power law means the loss falls by a steady fractional amount per doubling of compute, which lets you predict the outcome of a budget; cross-attention is how a U-Net takes a text condition (image features attend to the text but never the reverse); and MMDiT (multimodal diffusion transformer) is the alternative once everything is one sequence, with text and image tokens attending in both directions.

The bottom row is the U-Net’s one honest advantage: in the small-data regime its locality prior genuinely wins, and the honest answer there is a U-Net. Every DiT advantage above assumes enough data to pay for the missing prior.

adaLN-zero, and why the zero matters

Layer normalization (LN) rescales each token’s vector to a standard size, then applies a learned scale and shift, fixed once training ends. adaLN makes those adaptive: a small network produces them on the fly by reading the conditioning (here the diffusion timestep and the class). That is how the condition gets into the network at all. The DiT adds a third produced value, a gate, which multiplies the block’s entire output:

h  =  x  +  gate · Block( scale · LN(x) + shift )

The zero is the crucial part: the network producing gate is initialized to output zero. At training step 0 every gate is zero, so every block contributes nothing and h = x: the whole 28-block network is exactly the identity function, with the residual stream passing straight through. The network then switches blocks on one at a time as each earns its gradient, instead of starting as 28 layers of random noise stacked on each other. That is the difference between “trains” and “diverges in the first 2,000 steps.”

One accounting note, since two parameter counts float around. The adaLN networks are 6·d² per block, about 223M in total, but they act on a single conditioning vector per image, not on every token, so they add nothing to the per-token FLOP count. That is why the model gets called ~675M parameters while every FLOP figure here uses the 445.9M (12·L·d²) that actually touches each token.

Sampling: DDPM, DDIM, and the step-count curve

How many denoising steps should the sampler run? The two names in the heading are the samplers being compared: DDPM (denoising diffusion probabilistic models), the original random-walk sampler, and DDIM (denoising diffusion implicit models), the deterministic one that replaced it. Beyond both is distillation, which removes most of the remaining cost in exchange for output diversity.

Why DDPM needs many steps

DDPM generates by ancestral sampling: running the noising chain backwards one link at a time, each link a Markov step whose result depends only on the immediately preceding state. Each update is partly a denoise and partly a re-randomization (a fresh random draw is re-injected every step).

The limit is mathematical, not a weakness of the network. The reverse of a Gaussian noising step is itself Gaussian only in the limit of infinitesimally small steps. Over a large step the true reverse distribution is a more complicated shape that a Gaussian approximates badly, yet a Gaussian is exactly what the update rule assumes. So the step size is bounded by the accuracy of a Gaussian approximation, which is why the original formulation uses T = 1000 and why a better-trained network does not buy you larger steps.

DDIM: drop the stochasticity, gain a solver

DDIM removes the randomness. It rewrites the update in terms of the predicted clean image: first estimate what the finished image is, then re-noise that estimate down to the next noise level. Nothing samples anything, so the trajectory from noise to image is deterministic and a given seed maps to exactly one output.

That changes what the sampler is. The sequence of steps becomes the numerical approximation (the discretization) of a smooth continuous path described by a probability-flow ODE, an ordinary differential equation carrying pure noise continuously into an image. The steps are no longer links in a chain that must be followed one at a time; they are sample points on a curve. Two consequences follow:

1. You can take any evenly spaced subset of the timesteps. Going from 1000 steps to 50 to 20 is just solving the same equation on a coarser grid.
2. The remaining error is discretization error, so numerical-method accuracy applies. The naive Euler method (step along the current slope) has error O(1/N) for N steps. A second-order method such as Heun or DPM-Solver++ 2M, which corrects each step using slope from more than one point, has error O(1/N²) for essentially the same number of network evaluations.

The step-count / quality curve

Total error splits into a part that shrinks as you add steps and a part that does not:

FID(N)  =  FID_inf  +  C / N^k        k = solver order
           model error    discretization error

FID_inf is the irreducible quality the model would reach with infinitely many steps; no sampler change touches it. Fit this to measured points and you get FID_inf = 7.3, with C = 60 at first order and C = 240 at second order.

Two readings. A second-order solver at 20 steps matches a first-order solver at 100, a 5× compute saving from the sampler alone, with no retraining and no quality loss. And past ~50 steps, cost grows linearly in N while benefit shrinks as 1/N²: going from 30 to 250 steps is 8.3× the cost for 0.3 FID, which is below the noise floor of the metric (two runs of the same model differ by that much).

Distillation, and what it costs

Distillation trains a cheaper student to reproduce in few steps what the trained teacher produces in many. Three variants, in increasing aggression:

• Progressive distillation: a student takes one step landing where two teacher steps land, then repeats with itself as teacher; each round halves the step count (50, 25, 12, 6, 3).
• Guidance distillation: a student takes the guidance scale as an input and reproduces the guided prediction in a single pass, removing the unconditional pass entirely: a clean 2× saving with no change in step count.
• Adversarial / consistency distillation: pushes down to one to four steps by adding a critic that judges the student’s output directly.

The cost has a specific mechanism. The teacher’s ODE trajectory is a smooth curve; the student approximates it with N straight chords. As N falls, the map from starting noise to final image gets smoother and lower-entropy (entropy measures how much variety a distribution contains), so fewer distinct starting points reach distinct images and the range of producible images narrows. That loss shows up in recall, not precision (precision is the fraction of generated images that look real; recall is the fraction of the real variety the model still covers). A few-step model stays sharp and gets less diverse.

Row 3 is the default: 3× cheaper than row 1 and better on FID, because upgrading the solver more than pays for the small regression guidance distillation introduces. Rows 5 and 6 are interactive-preview modes, not the delivered image. At 4 steps recall falls from 0.62 to 0.48, 23% of the coverage gone, not a quality wobble but the product quietly ceasing to show certain kinds of output, and nobody reports the absence of an image they never saw. The FID and recall columns tell the story only when read together: FID barely moves down the first four rows while recall falls steadily, and that divergence is the whole argument.

Serving: memory is the binding constraint

Running this model in production inverts two intuitions carried over from serving language models: batching stops helping, and the memory ceiling is set by the decoder, not the transformer.

flowchart TD
    REQ(["Request<br/>1024 x 1024"]) --> Q["Queue"]
    Q --> PACK["Pack the CFG pair<br/>into one batch of 2"]
    PACK --> GPU["DiT sampler<br/>20 steps, 2nd order<br/>4,096 tokens"]
    GPU --> LAT(["128 x 128 x 16 latent"])
    LAT --> TILE{"output side<br/>> 1536?"}
    TILE -->|"no"| DEC["VAE decode<br/>2.2 TFLOP<br/>~2.5 GB peak"]
    TILE -->|"yes"| TDEC["Tiled decode<br/>512-px tiles, 64-px overlap<br/>blend seams"]
    DEC --> SAFE["Safety cascade + watermark"]
    TDEC --> SAFE
    SAFE --> CDN(["CDN"])

A diffusion step is a prefill, not a decode

A language model serves in two phases. Prefill processes the whole prompt at once, in parallel. Decode then emits one token at a time, each depending on the last, and is slow per token because it drags the whole model’s weights out of memory to do a tiny amount of arithmetic. It is memory-bandwidth-bound. The cure is batching many users so one trip through the weights serves all of them, aided by a KV cache that stores earlier tokens’ keys and values so they need not be recomputed.

Which regime you are in is decided by arithmetic intensity, the FLOPs an operation performs per byte it moves, compared against the hardware’s ridge point, the ratio at which peak compute and peak bandwidth balance. Above the ridge point an operation is compute-bound; below it, memory-bound. For the dominant per-token projection here at batch 1, arithmetic intensity is about 505 FLOP/byte against an H100 ridge point of 296, so the model is compute-bound even at batch size 1.

The reason is structural: a diffusion step processes all 4,096 tokens in parallel, so it is a prefill, repeated 20 times. Nothing is autoregressive (no token depends on a previously emitted one, and nothing carries between steps except the latent image) so there is no decode phase and no KV cache to amortize. This is the opposite of LLM decode, where batching is the entire optimization.

The consequence is the opposite serving strategy. Batching a diffusion model buys perhaps 1.1–1.2× (from amortizing kernel-launch overhead) and then goes flat, because there was never bandwidth waste to recover, while it adds latency in proportion to batch size. So: use batch 2 (the guidance pair you were going to run anyway, free); scale out with replicas, not batch size; and reserve batching for the decode, a small convolutional network where launch overhead is a larger share.

The memory budget, and what actually binds

For an 80 GB H100 serving at 1024: fixed costs (DiT weights, VAE, CUDA context) are about 3 GB. Per-sample diffusion activations, with autograd off and FlashAttention on, are only about 0.3 GB. But the VAE decode at 1024 peaks at about 2.5 GB per sample: a single full-resolution feature map is 1024 · 1024 · 128 · 2 bytes ≈ 268 MB and the decoder holds several at once.

Peak memory is the decoder, not the transformer, by roughly 8×. The diffusion model works on 4,096 tokens; the decoder works on 1,048,576 pixels with 128 channels. This scales with output pixels: ~10 GB per sample at 2048, ~40 GB at 4K, at which point one image no longer fits. That is why the tiling threshold is a resolution, not a batch size.

Tiled decoding decodes the latent in overlapping square pieces and blends the overlaps, so no piece ever needs the memory of a full-resolution image. It costs two things. Seams: a seam appears wherever the decoder’s receptive field reaches beyond the overlap into a neighbouring tile decoded separately, most visible in smooth gradients like skies; the usual settlement is a 64-pixel overlap on 512-pixel tiles, cosine-blended. Redundant compute: with 512-pixel tiles overlapping by 64, consecutive tiles start 448 pixels apart, so the decoder processes (512/448)² ≈ 1.31, about 31% redundant in the limit (worse at finite tile counts, ~56% at 2048, because edge tiles have nothing to amortize against).

Cost per image, end to end

Everything converges on one number. Diffusion is 20 steps × 2 forwards × 5.817 TFLOP = 232.7 TFLOP; add ~2.2 for the VAE decode and ~0.02 for safety screening, for about 235 TFLOP per image.

235 TFLOP ÷ 300 TFLOP/s effective       ≈  0.78 s of GPU
× $2.50/GPU-hour ÷ 0.60 utilization     ≈  $0.00091 per image
                                        ≈  $0.91 / 1,000 images
at 2M images/day                        ≈  $1,813/day
steady-state load                       ≈  18.1 GPUs
provision for the 2.2× diurnal peak     ≈  40 GPUs

Two easy mistakes hide in that last pair of lines. The 18.1 GPUs is the average load; 40 is what you have to own: the daily peak is 2.2× the mean, and a GPU cannot be conjured for the busy hour, so provisioning follows the peak while the bill follows the mean. And two utilization figures appear in this chapter that must never be multiplied together: the 300 TFLOP/s here is the end-to-end effective rate (~30% of the H100’s 990 TFLOP/s peak), the right number for money because it absorbs attention kernels, normalizations, the decoder, and kernel-launch gaps; the 55% from the architecture section is the DiT’s matmul MFU, used only as a ratio against the U-Net’s 35%. Use the effective rate for money and the matmul MFU for architecture comparisons.

The full end-to-end latency at batch 2 is about 0.8 s of GPU path plus ~0.7 s of queue, transfer, WebP encode, and CDN put, for a p50 near 1.5 s and a p95 near 3.1 s. The sampling loop is ~94% of the GPU path and the decode ~1%, the reverse of the memory picture.

The lever table

Every row is the finished design with exactly one decision changed:

The latent-space decision is worth 36,500×. Every remaining lever is under 10×, and the two cheapest of those are paid for in coverage, not dollars. Once you are in latent space the cost problem is solved, and the remaining engineering is about not spending the savings on diversity.

Metrics at high resolution

The cost is settled; measurement is not, because the metric the field quotes stops carrying information above about 300 pixels.

FID stops working at high resolution. The Inception-v3 network it is computed through accepts only 299 × 299 input, so a 1024 image is shrunk 1024/299 ≈ 3.42× per side before the metric ever sees it, 89,401 of 1,048,576 pixels survive, and 91.5% are discarded. You spent about 25× the compute of a 256-pixel model (token count rises 16×, attention 256×, total ~24.6×) to produce detail the metric cannot see. So a model with perfect global structure and mushy skin pores scores identically to one with both, and the number everyone reports cannot tell them apart.

What to report instead. Two terms: VQA is visual question answering. You ask a VLM (vision-language model) questions derived from the request and score the answers.

Three of the five are free; the expensive one, human raters counting named defects, is the only one that catches failures no feature extractor was trained to notice.

Failure modes

The four failures users report most (text, hands, tile seams, duplicated subjects) each trace to a mechanism, and one pattern recurs: each is a discrete global constraint over a tiny area, and the loss is per-element and knows nothing about either.

Text rendering

Illegible text is the most-reported failure, with three independent causes stacked, only the last of which is really the generator’s fault, which is why “train harder” does not fix it.

Cause 1: the autoencoder destroys the letter shapes before diffusion is involved. A glyph is the drawn shape of a character. At f=8, c=4, one latent cell must represent an 8 × 8 pixel block, 8 · 8 · 3 = 192 subpixel values compressed into 4 numbers. A 12-point glyph in a 1024 image has strokes about 2 pixels wide, a quarter the width of the cell that has to encode it. There is nowhere for it to live. Encode-then-decode of rendered text, with no diffusion anywhere, shows this directly (OCR character error rate, the fraction of characters misread):

f=8  c=4    36 pt text     4% error
f=8  c=4    12 pt text    31% error
f=8  c=16   12 pt text     9% error   (same token count as row 2)
f=4  c=4    12 pt text     3% error   (8.5x the diffusion cost)

The 31% is damage the compression did on its own. Sixteen channels fix most of it at no change in token count.

Cause 2: the conditioning never contained the letters. A subword text encoder splits text into common fragments, so STARBUCKS arrives as two or three chunks with no character-level structure. And CLIP (contrastive language-image pretraining, the usual source of a diffusion model’s text conditioning) was never trained with character-level supervision. The model is not failing to render the letters. It was never told what they are.

Cause 3: the objective does not care. The loss is squared error in latent space, so turning an E into an F moves a handful of latent numbers by a small amount. Nothing demands an exact glyph sequence, and gradient pressure toward correct spelling is proportional to the area affected, a fraction of a percent of the image.

The fixes, in order of effect: 16-channel latents (takes error from 31% to 9%, free in token count); a character-aware encoder such as ByT5 (which works directly on raw bytes, so individual letters survive) run alongside the semantic one; and glyph-conditioned training or a loss that weights text regions specially.

Hands and anatomy

Wrong hands are the second most-reported failure, same shape as text with different constants. A hand in a 1024 image is roughly 100 × 100 pixels, about 0.95% of the image, about 12 × 12 of the 128 × 128 latent cells, or 0.88% of the latent grid. A completely wrong hand costs under 1% of the reconstruction loss, while a slightly-off sky costs 40%: the loss weights defects by area, and this defect is a discrete global constraint (“exactly five fingers”) over a tiny area.

Add uncertainty on top: hands appear in enormously varied poses with heavy self-occlusion (fingers hiding fingers), so what belongs in a hand-shaped region is genuinely uncertain, and under uncertainty a regression model hedges. Hedging over a continuous variable gives a blur; hedging over a discrete count gives six fingers. The fixes are all on the data side, because the loss cannot be talked out of averaging: oversample crops with hands; weight the loss in hand regions; or condition on pose keypoints (explicit joint coordinates) so the count is supplied instead of inferred.

Aspect-ratio artifacts

The third failure appears the moment you serve a shape you did not train on, and the symptom is specifically duplicated subjects. Train at square 1024, serve 1536 × 640, and the token count moves from 4,096 to 3,840. Two things break:

1. Position embeddings go out of distribution. A position embedding tells the transformer where in the grid a token sits, since attention is otherwise blind to order. A learned 64 × 64 grid simply has no entry for column 95; interpolating is a guess, extrapolating past the edge is worse.
2. Attention entropy shifts. Attention weights come from a softmax over the candidate keys; running it over 3,840 keys instead of 4,096 changes how sharply the weights concentrate, the effective temperature of every attention distribution.

The signature symptom, duplicated subjects (two heads, two horizons), has a sharper cause. The model learned local statistics consistent with “one subject fills a square frame.” At double the width, those same statistics are equally well satisfied by two subjects side by side, and nothing prefers one: there is no global “exactly one” constraint, for the same reason there is no “exactly five fingers.” The fix has two parts: aspect-ratio bucketing during training (group real images into a handful of aspect-ratio buckets, keep total pixel area roughly constant so token count barely moves, and train on all of them); and RoPE (rotary position embedding), which encodes position as a rotation applied to the attention vectors instead of a table lookup, so an unseen position is handled by the same rule as every other instead of being a missing entry.

Prompt adherence versus aesthetic quality

The fourth failure is not a bug but an optimization working exactly as specified against the wrong target, the clearest case of reward hacking here. Fine-tuning a generator on human preference data maximizes expected reward, coming either from a reward model (a network trained to predict which of two images a human prefers) or from DPO (direct preference optimization, which tunes on preference pairs directly). Either way the preference data was collected as pairwise aesthetic comparisons, and aesthetic preference correlates with saturated colour, shallow depth of field, symmetry, and centred composition, not with “included all five requested objects,” because a rater comparing two images rarely re-reads the prompt.

after aesthetic fine-tuning, same 500 prompts:
  aesthetic score       6.21 -> 6.63   (+0.42)
  VQA prompt adherence  0.71 -> 0.68   (-3 pp)
  recall (coverage)     0.62 -> 0.52   (-0.10)

This is reward hacking: maximizing the stated objective by a route that defeats the purpose of stating it. The model found the region the reward model likes and moved there, exactly what you asked for, and not what you wanted. The control is to report the two numbers as a pair and gate on adherence, never to ship on the aesthetic number alone.

The rest

The remaining failures, each with its detection and control. The detection column carries the most weight here, because it is the difference between a failure you catch on a dashboard and one a user reports. Three terms: std is standard deviation; an FFT (fast Fourier transform) decomposes an image into its repeating patterns, so a faint regular grid shows up as a sharp spike; OOM is out of memory.

Alternatives rejected

Each rejected alternative names the assumption that would have to change to reverse it. The first is 36,500×; the rest are single digits.

Conclusion

The whole design hangs on a single lever and a handful of consequences.

flowchart TD
    D1["Attention is quadratic in tokens,<br/>pixels are tokens → pixel diffusion<br/>at 1024 costs ~8 h, $33/image"] --> D2["Compress 8x per side in a<br/>frozen autoencoder (f=8, c=16)"]
    D2 --> D3["64x fewer tokens → 4,096x less<br/>attention → 24,546x cheaper end to end"]
    D3 --> D4["Denoiser is a DiT, not a U-Net:<br/>the U-Net's job is now done by the AE"]
    D4 --> D5["Sample with a 2nd-order solver,<br/>20 steps ≈ 100 Euler steps"]
    D5 --> D6["Serve compute-bound at batch 2,<br/>scale with replicas, tile the decode<br/>above 1536 → ~$0.00091/image"]

The load-bearing pieces, and what breaks if each is wrong:

• The denoiser needs genuine long-range dependencies. This rules out a pure-convolution model and makes compression unavoidable, not merely economical. A generator that only had to produce locally plausible texture could be pure convolution, and none of this would apply.
• One autoencoder, trained once and frozen, serves every future generator. That amortizes its cost to nothing and lets the diffusion model be retrained freely. It is also the decisive advantage over the cascade, whose stages re-tune every time the base changes.
• Latent scaling is applied before training. Skip it and the model trains to a low loss and produces locally beautiful, globally incoherent images, with no error anywhere.
• A diffusion step is compute-bound at batch 1 (505 FLOP/byte against a 296 ridge point). This inverts every serving instinct from language models: scale with replicas, not batches, and there is no KV cache.
• Recall is measured, not just FID. Drop it and few-step distillation reads as a free 9× while the model has silently stopped producing a fifth of the distribution.
• FID is measured at native resolution (patch-FID). Standard FID discards 91.5% of a 1024 image before scoring, so it cannot see the detail you paid ~25× the compute to produce.

Above ~2K the design becomes a hybrid: latent diffusion to 1024, then a refiner stage, a cascade whose base is a latent model. Everything else is choosing where to spend the savings the latent decision already earned, and the honest answer is to spend them on quality, not on further cost, since every remaining lever is under 10×.

One line to remember: compress 8× per side in a frozen autoencoder first, and every other choice (DiT over U-Net, 20 second-order steps, batch 2 with replicas) is just deciding where to spend the 24,546× the compression already bought.

Further reading

• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (2022), the latent-diffusion / Stable Diffusion paper.
• Peebles & Xie, Scalable Diffusion Models with Transformers (2023), the DiT and adaLN-zero.
• Esser et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024), MMDiT and 16-channel latents.
• Ho et al., Denoising Diffusion Probabilistic Models (2020), and Song et al., Denoising Diffusion Implicit Models (2021), the two samplers compared here.
• Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models (2022), and Lu et al., DPM-Solver++ (2022), second-order samplers and the step-count trade.
• Saharia et al., Photorealistic Text-to-Image Diffusion Models (Imagen, 2022), cascaded super-resolution and noise-conditioning augmentation.
• Dao et al., FlashAttention (2022), tiled attention.

Next: the text-to-image chapter.