08 — High-Resolution Image Synthesis

“Take the generator from chapter 07 to 1024 × 1024 and serve it at 2M images a day.”

Generating a large image is not the same problem as generating a small one, and the gap between the two is the whole subject here.

The centrepiece is one derivation you should be able to produce on a whiteboard: compressing the image by a factor of eight per side cuts the dominant cost by a factor of four thousand. That single decision is worth more than every other optimization in the chapter combined.

By the end you should be able to:

• price a 1024 × 1024 image down to five decimal places of a dollar;
• choose between the two competing architectures for getting there, and defend the choice;
• pick a sampling step count from a fitted curve instead of copying a number out of a paper;
• explain why the standard quality metric stops carrying information above 299 pixels;
• name the mechanism behind each of the four failures users complain about most — garbled text, wrong hands, tile seams, and duplicated subjects at wide aspect ratios.

The starting point, restated so this chapter stands alone

Three terms carry the whole chapter. Get them straight before anything else.

A diffusion model is a network trained to remove a small amount of noise from a picture. You apply it over and over: generation starts from pure random noise and walks it down into an image over some number of steps. Chapter 07 picked that family over the alternatives — chiefly because it covers the full variety of the training data instead of quietly abandoning parts of it — and none of those reasons change with resolution.

The denoising network itself is a transformer, the same architecture language models use. It chops its input into a sequence of tokens — fixed-size pieces of the input, each represented by a vector of numbers — and then runs attention, an operation in which every token compares itself against every other token.

That last word is the whole problem. Every token compares against every other token, so with n tokens there are n × n comparisons. The cost of attention grows with the square of the number of tokens. Double the tokens, quadruple the attention bill.

Input and output. The input is a condition describing the image you want — a set of attributes, or a sentence, though chapter 09 is where the text case is handled properly. The output is one 1024 × 1024 image, delivered in under five seconds and for well under a tenth of a cent.

And the number that organizes everything. Naive diffusion run directly on pixels at 1024 × 1024 costs eight hours of H100 time per image. That is not “slow”, it is architecturally impossible. Everything in this chapter is a consequence of that number, and the centrepiece is the derivation of the compression factor that makes it go away.

1. Framing

Pin down the interface and the constraints first, because between them they force the single structural question that the rest of the design answers.

• Input: a condition (attributes, or text — chapter 09 takes the text case).
• Output: 1024 × 1024, and a product that will ask for 1536 × 640 and 2048 × 2048 within a quarter.
• Constraints: three of them, listed below.
• Why it is hard: the compute of the naive design grows faster than the pixel count rather than in proportion to it, and the metric everyone quotes is computed after shrinking the image to 299 × 299.

The three constraints, each with the thing it rules out:

On percentiles, since they appear throughout: p95 is the value 95% of requests come in under, and p50 is the median. Latency promises are stated as percentiles rather than averages because an average hides the slow tail — one request in twenty taking 30 seconds barely moves the mean and is exactly what users notice.

Say this first: “The resolution question is really a question about where the downsampling lives. Attention over pixels is quadratic in pixel count, so at 1024 the model has to operate on a compressed representation. There are exactly three places to put the compression — inside the diffusion model, in front of it as a learned autoencoder, or between a chain of models — and picking one is the whole design.”

Five terms appear in the diagram below and are used for the rest of the chapter:

• Downsampling — reducing the resolution of a grid, so fewer positions carry the same picture.
• U-Net — the convolutional network diffusion models originally used. It is shaped like a letter U: it downsamples through several stages, then upsamples back, with shortcut links across the middle.
• Autoencoder — a pair of trained networks. One compresses an image into a compact numeric code; the other expands the code back into pixels.
• Cascade — a chain of models where a small one produces a low-resolution image and later ones enlarge it.
• Patching — grouping neighbouring cells of a grid into a single token before the transformer sees them.

The diagram lays out the three places compression can live, with the per-forward-pass cost of each attached. Read the three leaf boxes against each other: 142,798 against 5.82 against 525.8 TFLOP. The rest of the chapter is the argument between those three numbers.

flowchart TD
    Q{"1024 x 1024 target.<br/>Attention is O n^2 in tokens.<br/>Where does the compression live?"}
    Q -->|"inside the model"| A["Pixel-space U-Net<br/>conv downsampling stages,<br/>attention only at low res"]
    Q -->|"in front of the model"| B["LATENT DIFFUSION<br/>a trained autoencoder<br/>compresses 8x per side"]
    Q -->|"across models"| C["Cascade<br/>small base + super-res stages,<br/>each conditioned on the last"]
    A --> A1["still 1,048,576 pixels at<br/>the top of the U-Net.<br/>142,798 TFLOP per forward"]
    B --> B1["16,384 latent cells<br/>4,096 tokens after patching.<br/>5.82 TFLOP per forward"]
    C --> C1["3 models, 3 training runs,<br/>errors compound down the chain.<br/>525.8 TFLOP per image"]

Assumptions in this section.

• State out loud: a 1024 × 1024 target, a p95 latency budget of 5 seconds, and a cost ceiling of a tenth of a cent per image. Those three are the specification, and the rest of the chapter is derived against them.
• Ask: what resolutions and aspect ratios the product will demand within a year. That single answer decides whether the design needs the cascade in Cascaded super resolution the alternative, and whether tiled decoding in Serving memory is the binding constraint is optional.
• Load-bearing: that the model must fit on one GPU. That rules out sharding the denoiser across devices, which would otherwise be a legitimate answer to the memory problem. It is a fleet fact rather than a modelling one, which makes it the assumption most likely to be wrong for a different organization.
• Also load-bearing: that there are exactly three places to put the compression. If a fourth existed, Why pixel space diffusion at 1024 is hopeless through Architecture u net vs dit would be arguing over an incomplete list.

2. Why pixel-space diffusion at 1024 is hopeless

The obvious design falls to arithmetic, and the obvious escape from it falls right behind — which is what makes compression a requirement here rather than an optimization.

Take the DiT-XL backbone from Training. DiT is a diffusion transformer: the denoising network is a plain transformer rather than a U-Net. XL is the size.

Two numbers describe that size, and both appear in every formula below:

• d = 1152 is the width — how many numbers represent each token.
• L = 28 is the depth — how many stacked transformer blocks the network has.

Those two give 445.9 million parameters acting on every token. The count is 12·d^2 parameters per transformer block times L blocks — the standard accounting from Parameter arithmetic you should be able to do in your head — so 12 · 28 · 1152^2 = 445,906,944.

Now run that network directly on pixels, one token per pixel.

Two units before the numbers. A FLOP is a single floating-point operation. A TFLOP is a trillion of them. The standard accounting for a forward pass is two FLOPs per parameter per token — one multiply and one add.

Here is what one forward pass costs at 1024 × 1024. The first line counts the tokens; the next two price the two kinds of work the network does.

tokens          n = 1024 · 1024                          =  1,048,576

token-linear    2 · N · n = 2 · 445.9e6 · 1.049e6        =      935.1 TFLOP
attention       4 · n^2 · d · L
                = 4 · (1.049e6)^2 · 1152 · 28            =  141,863.4 TFLOP
                                                            ------------
per forward                                                 142,798.5 TFLOP

The cost splits into two terms, and which one dominates is the whole story.

• The token-linear term is the ordinary matrix work — every parameter touching every token. It grows in proportion to the token count.
• The attention term is every token comparing itself against every other. It grows with the square of the token count.

Read the split: attention is 99.3% of the cost (141,863.4 / 142,798.5 = 0.9935).

The two terms are equal at roughly n = 6d tokens, a crossover derived in Parameter arithmetic you should be able to do in your head. At d = 1152 that crossover is 6 · 1152 = 6,912 tokens. You are at 1,048,576 tokens, which is 1,048,576 / 6,912 = 152× past it. The quadratic term has stopped being a footnote and become the entire bill.

For contrast, chapter 07 ran the same backbone at 1,024 tokens — well below that crossover — which is why “attention is quadratic” was true there and not what you were paying for. Four times the resolution per side moves you 152× past the same line.

Turning FLOPs into hours and dollars

Generation is not one forward pass. It runs the network once per sampling step, and twice per step if classifier-free guidance (CFG) is on.

CFG is the standard technique for making the output obey the request more strongly: run the network once with the request, once without it, and extrapolate away from the unconditioned prediction. It is derived in Classifier free guidance. The only thing that matters here is that it doubles the forward passes.

30 sampling steps × 2 forwards (CFG)  =  60 forwards
60 × 142,798.5 TFLOP                  =  8,567,910 TFLOP  =  8.57e18 FLOP
H100 at 300 TFLOP/s effective         =  8.568e18 / 300e12
                                      =  28,560 s  =  7.93 hours per image
at $2.50/GPU-hour                     =  7.93 × 2.50   =  $19.83
at 60% fleet utilization              =  19.83 / 0.60  =  $33.06 per image

Eight hours and thirty-three dollars for one picture.

Two notes on that block. The 300 TFLOP/s is the effective rate an H100 sustains end to end, not its peak — Cost per image end to end explains the difference and why costing uses the effective number. The 60% utilization is the fraction of the fleet’s paid-for GPU-seconds that do useful work; you pay for the idle 40% too, so dividing by 0.60 is what turns compute time into a bill.

And the eight hours is the number you get with the best available implementation. FlashAttention is the standard trick that computes attention in small tiles, so the giant intermediate table of token-to-token scores never has to exist in memory all at once. It removes the memory problem but not the arithmetic — the FLOP count above already assumes it.

Without it the design does not merely cost too much, it does not fit. One n × n score table at n = 1,048,576 in 16-bit numbers is 1,048,576^2 × 2 bytes = 2.2 TB, and that is for a single attention head of a single layer, against 80 GB of GPU memory.

Why you cannot just delete the attention

The obvious escape is to throw attention away and build the network entirely out of convolutions, whose cost grows only in proportion to the pixel count. The trade does not work, and seeing why pins down exactly what attention is buying.

Drop attention, build the network out of convolutions only, and the cost problem disappears — but so does global coherence. Here is why.

Three terms first:

• A convolution applies a small window of weights, called a kernel, at every position of the image. A 3 × 3 kernel lets each output pixel see only its immediate neighbours.
• Stride is how far the window jumps between applications. Stride 1 means it skips nothing, so the output grid is the same size as the input. Stride greater than 1 shrinks the grid — which is downsampling.
• The receptive field of an output position is the region of the input it can possibly depend on. It grows one layer at a time.

The receptive-field recurrence from Convolution weight sharing and what it buys is r_l = r_{l-1} + (k_l - 1)·j_{l-1}, where k_l is the kernel size at layer l and j_{l-1} is the accumulated stride below it. At stride 1 every j is 1, so a 3 × 3 kernel adds k - 1 = 2 pixels of reach per layer. Stack them and the reach grows linearly:

to connect opposite corners of a 1024 image with stride-1 3x3 convs:
    distance to cover  =  1024 - 1  =  1023 pixels
    reach per layer    =     3 - 1  =     2 pixels
    layers needed      =  1023 / 2  =  511.5, so 512 layers

Five hundred layers, just to let one corner of the image know what the other corner is doing.

And a generative model needs exactly that. One nose, two eyes, a single light source, one horizon: those are long-range dependencies — relationships between parts of the image that are far apart.

So you need one of two things. Either stride greater than 1, which is downsampling and therefore the very thing we are choosing where to put; or attention, which is quadratic. There is no third option. That is why the framing question in Framing is the whole design.

Assumptions in this section.

• State out loud: a DiT-XL at d = 1152 and L = 28, 30 sampling steps with guidance on, an H100 delivering 300 TFLOP/s end to end, $2.50 per GPU-hour, and 60% fleet utilization. Every figure here is a product of those five, and none of them changes the conclusion, because the conclusion is four orders of magnitude wide.
• Ask: nothing. This section is arithmetic on stated inputs, and there is no product question that changes it.
• Load-bearing: that the denoiser needs genuine long-range dependencies at all. The convolution-only design is ruled out on that claim alone, and the claim is what makes compression unavoidable rather than merely economical. A generator that only had to produce locally plausible texture, with no global structure to keep consistent, could be built from convolutions, and none of this chapter would apply.

3. Latent diffusion, derived

Here is the derivation to be able to reproduce on a whiteboard: the compression saving, exact; what the compression costs in quality; and the one implementation detail that silently ruins the model if you skip it.

The idea in one sentence: train an autoencoder once, then run the entire diffusion process on its compressed output instead of on pixels, and decode to pixels a single time at the very end.

In symbols, an encoder E maps an image of height H, width W and 3 colour channels down to a grid of height H/f, width W/f and c channels. A decoder D maps that grid back to pixels. Three names to hold onto:

• The compressed grid is the latent. One position in it is a latent cell.
• f is the downsampling factor per side. At f = 8, a 1024 × 1024 image becomes a 128 × 128 grid.
• c is the channel count — how many numbers each latent cell holds.

Both networks are trained once and then frozen: their weights never change again, so every later diffusion model reuses them unchanged. That is what lets the autoencoder’s training cost amortize away.

One thing trips people up straight away. The compression is 8× per side, but the number of stored values falls by far less than 64×, because the latent trades spatial resolution for channels. Work it out at the c = 16 configuration this chapter ships (The ceiling the autoencoder sets, and the CONFIG line in Cost per image end to end):

values      1024 · 1024 · 3   =  3,145,728  pixel values
             128 ·  128 · 16  =    262,144  latent values
            3,145,728 / 262,144                       =   12x fewer values

positions   1024 · 1024       =  1,048,576  pixel positions
             128 ·  128       =     16,384  latent cells
            1,048,576 / 16,384                        =   64x fewer positions

Attention charges you for positions, not for values. That is why c can be raised almost for free and f cannot — a point The ceiling the autoencoder sets turns into the highest-leverage change in the chapter.

The diagram below shows the full path. The grouped box labelled “Trained once, frozen” is the autoencoder; everything to the right of it runs on every request. Watch the count fall twice: 1,048,576 pixels, then 16,384 latent cells, then 4,096 tokens. Watch also that the loop arrow runs around the DiT only — the decoder runs once, at the end, not once per step.

flowchart LR
    subgraph OFF["Trained once, frozen"]
        IMG(["1024 x 1024 x 3<br/>3,145,728 values"]) --> ENC["Encoder<br/>f = 8"]
        ENC --> LAT["128 x 128 x 16<br/>16,384 cells<br/>262,144 values"]
    end
    LAT --> PATCH["Patchify p = 2<br/>64 x 64 = 4,096 tokens"]
    PATCH --> DIT["DiT<br/>28 blocks, d = 1152<br/>5.82 TFLOP per forward"]
    DIT --> LOOP{"30 steps"}
    LOOP -->|"loop"| DIT
    LOOP -->|"done"| DEC["Decoder<br/>2.2 TFLOP"]
    DEC --> OUT(["1024 x 1024 x 3"])

The 64×, and the 4,096×

The derivation runs in four steps — what the autoencoder saves, why attention saves the square of that, what patching adds on top, and where the end-to-end number lands — and steps 1 and 2 are the two lines to memorize.

STEP 1 — the autoencoder removes a factor of f per SIDE.

    pixel positions     n_pix = 1024 · 1024              = 1,048,576
    latent cells        n_lat = (1024/8) · (1024/8)      =    16,384
    ratio                     = f^2 = 8^2                =        64x

STEP 2 — attention is quadratic in token count, so the saving squares.

    attention FLOPs ~ n^2
    ratio           = 64^2                               =     4,096x

    check, exactly:
    pixel   4 · (1,048,576)^2 · 1152 · 28  =  141,863.4 TFLOP
    latent  4 · (16,384)^2    · 1152 · 28  =       34.6 TFLOP
    141,863.4 / 34.6                       =        4,096   exactly

An 8× spatial downsample is a 64× reduction in tokens, and because attention is quadratic in tokens, a 4,096× reduction in attention cost. That single line is the reason latent diffusion exists, and it is the answer to “why not just run diffusion on pixels”.

Say the chain out loud once: 8× per side, so 8^2 = 64× on tokens, so 64^2 = 4,096× on attention. The exact check in step 2 confirms it lands on 4,096 with no rounding.

Patchification stacks a second, independent factor on top. At patch size p = 2, each 2 × 2 block of latent cells becomes a single token, so the token count falls by another factor of four.

Steps 3 and 4 apply that, and then account for the fact that the two cost terms shrink at different rates.

STEP 3 — patch size p = 2 folds 2x2 latent cells into one token.

    tokens  = 16,384 / 4                                 =     4,096
    total token reduction  = 64 · 4                      =       256x
    total attention reduction = 256^2                    =    65,536x

STEP 4 — the token-linear term falls by 256x, not 65,536x, so the
         end-to-end saving lands between the two.

    pixel  per forward   935.100 lin + 141,863.400 att  =  142,798.5 TFLOP
    latent per forward     3.653 lin +       2.165 att  =        5.817 TFLOP

    end-to-end ratio     142,798.5 / 5.817              =     24,546x

    why not 65,536x: the linear term fell by only 256x (935.1 / 3.653),
    and once attention is small the linear term is what is left to pay.

Notice what happened to the balance between the two terms.

In pixel space the split is 935 : 141,863 — attention-dominated, 152 to 1. In latent space it is 3.65 : 2.17 — dominated instead by the token-linear term. The compression did not just shrink the bill, it changed which term you are paying.

That matters beyond the money. When cost is proportional to parameters, scaling laws are clean — meaning the empirical relationships that predict how much better a model gets for a given increase in size and compute hold their shape. Adding capacity then has a knowable payoff instead of being a gamble, because doubling the parameters doubles the cost rather than doing something quadratic and unpredictable.

The code below reproduces the whole derivation from scratch and asserts every number against the hand arithmetic above.

def dit_flops(n_tokens: int, d: int = 1152, layers: int = 28,
              per_token_params: float = 445.9e6) -> dict:
    """FLOPs for one forward pass of a DiT-style backbone at n tokens.

    token_linear: 2 FLOPs per parameter per token (one multiply-add).
    attention:    4·n·d per token per layer for QK^T and the AV product.
    """
    linear = 2.0 * per_token_params * n_tokens
    attention = 4.0 * n_tokens ** 2 * d * layers
    return {"linear": linear, "attention": attention, "total": linear + attention}


def latent_tokens(side: int, f: int = 8, patch: int = 2) -> int:
    return (side // f // patch) ** 2


pixel = dit_flops(1024 * 1024)
latent = dit_flops(latent_tokens(1024))          # 4,096 tokens
no_patch = dit_flops((1024 // 8) ** 2)           # 16,384 latent cells

small = dit_flops(latent_tokens(256))            # a 256-pixel model: 256 tokens

print(f"pixel   {pixel['total'] / 1e12:12,.1f} TFLOP/forward")
print(f"latent  {latent['total'] / 1e12:12,.4f} TFLOP/forward  "
      f"(attention alone, unpatched: {no_patch['attention'] / 1e12:.1f})")
print(f"end-to-end saving {pixel['total'] / latent['total']:,.0f}x")
print(f"256px {small['total'] / 1e12:.4f}  1024px {latent['total'] / 1e12:.4f}  "
      f"tokens {latent_tokens(1024) // latent_tokens(256)}x  "
      f"compute {latent['total'] / small['total']:.2f}x")

assert round(no_patch["attention"] / 1e12, 1) == 34.6
assert round(pixel["attention"] / no_patch["attention"]) == 4096      # = f^4 = 8^4
assert round(pixel["total"] / latent["total"]) == 24546

# The s9 claim. Tokens rise 16x from 256px to 1024px; compute rises ~25x,
# because attention rises with the square of the token count.
assert latent_tokens(1024) // latent_tokens(256) == 16
assert round(small["total"] / 1e12, 4) == 0.2368
assert round(latent["total"] / 1e12, 4) == 5.8175
assert round(latent["total"] / small["total"], 2) == 24.57

What the autoencoder costs, and the one place a GAN loss belongs

A GAN — generative adversarial network — trains one network against a critic, a second network whose job is to tell generated output from real data. Chapter 07 rejected that training scheme outright.

Inside the autoencoder, that same loss is not merely acceptable but necessary — a good example of a loss being safe or dangerous depending entirely on what else sits in the objective beside it.

The autoencoder is trained with four terms:

• L1 — the sum of absolute pixel differences between the input and the round-tripped output. Straight pixel fidelity.
• LPIPS (learned perceptual image patch similarity) — compares two images not pixel by pixel but through the internal features of a pretrained classifier, here VGG, an older convolutional network still used as a standard feature extractor. LPIPS tracks human judgements of similarity far better than raw pixel distance does.
• PatchGAN — a small adversarial critic that judges local patches for realism rather than scoring the whole image.
• KL — the Kullback-Leibler divergence, a measure of how far one distribution sits from another, pulling the latent distribution toward a standard bell curve. It is weighted so lightly here (about one part in a million) that it does almost nothing beyond keeping the latent’s scale from drifting.

The lambda coefficients below are the weights on each term. x is the input image, so D(E(x)) is that image after a round trip through encoder and decoder.

L_AE  =  L1(x, D(E(x)))                       pixel fidelity
      +  lambda_p · LPIPS(x, D(E(x)))         perceptual, VGG feature distance
      +  lambda_a · L_patchGAN(D(E(x)))       texture realism
      +  lambda_kl · KL(q(z|x) || N(0, I))    tiny, ~1e-6

The adversarial term is doing a job that L1 structurally cannot. Here is the mechanism, in three steps.

First, what a reconstruction loss optimizes. A loss built on absolute or squared differences is minimized by a conditional statistic of everything still consistent with the input — the median for L1, the mean for L2. Not by any single plausible answer, but by an average over them.

Second, what that does to texture. When several fine textures are equally plausible for a patch — the exact arrangement of pores, threads, gravel — the average of those textures is smooth. So the loss does not merely tolerate blur, it actively prefers it. This is the same argument derived for the VAE (variational autoencoder, the compress-to-a-code-and-expand-back model) in Vae a bound and why the bound blurs, and it transfers here unchanged.

Third, why LPIPS is not enough. It moves the comparison into feature space, which helps, but it is still a regression loss with the same averaging behaviour one level up.

Only an adversarial critic supplies pressure to commit to one plausible texture instead of averaging over all of them — because an averaged texture is exactly the artifact a critic learns to spot.

The adversarial loss is dangerous when it is the only signal and safe when it is a texture prior on top of a reconstruction loss. In chapter 07 the GAN was rejected because mode collapse — the generator abandoning whole kinds of output, in the extreme case ignoring its random input entirely — carries no penalty in the objective. Here the reconstruction term makes ignoring the input impossible: the autoencoder is handed the image x and must return it, so drifting away from x costs L1 immediately. Same loss, opposite risk, entirely because of what else is in the objective.

The ceiling the autoencoder sets

Compression is lossy, and exactly how lossy is worth measuring, because that number is a hard ceiling on everything built on top.

The diffusion model can never produce detail the decoder is incapable of reconstructing. Whatever the autoencoder throws away is gone for every model you will ever build on top of it.

So measure it before training anything downstream. The procedure is one line: push the validation set through the encoder and straight back out through the decoder, with no diffusion involved at all, then compare the originals against the round-tripped copies.

The comparison uses reconstruction FID (rFID). FID — the Fréchet Inception Distance, defined in Fid and what it actually measures — compares two collections of images through the internal features of a pretrained classifier. Lower is better; zero means indistinguishable. Because both collections here contain the same images, differing only by a trip through the compression, rFID isolates exactly what the compression destroyed.

The table below prices four autoencoder configurations. Read it as a two-column trade: TFLOP/forward is what the diffusion model costs downstream, rFID is the quality ceiling you accept in exchange. The two bold rows have identical compute and very different ceilings.

Row 3 is the free lunch, and it is worth understanding why.

Raising the channel count c does not change the token count at all. Tokens come from positions, and c is depth per position. Compare rows 2 and 3: both are 128 × 128 latents, both 4,096 tokens, both 5.82 TFLOP per forward.

What c does change is the width of the projections at the very input and output of the transformer — the layers that map c numbers into d = 1152 and back. That is a rounding error against the 12·L·d^2 parameters in the body of the network.

So raising c from 4 to 16 quadruples the information stored in each latent cell at essentially zero extra diffusion cost, and cuts rFID from 0.74 to 0.28, a (0.74 - 0.28) / 0.74 = 62% reduction. That is why recent systems use 16-channel latents, and it is the single highest-leverage change available at this layer.

Latent scaling, the one detail that silently breaks everything

The diffusion noise schedule is written in absolute terms. The blending rule is:

x_t  =  sqrt(abar_t) · x_0  +  sqrt(1 - abar_t) · eps

It mixes the clean input x_0 with noise eps, in proportions set by the schedule value abar_t at step t. That rule assumes x_0 has roughly unit variance — variance being the standard measure of how spread out a set of numbers is, and its square root being the standard deviation.

Nothing forces a freshly trained encoder to obey that assumption. Its latents come out at whatever scale training happened to land on.

Suppose your encoder emits latents with a standard deviation of 5 instead of 1. The signal is then 5× larger than the schedule expects while the noise is exactly the size the schedule chose, so the SNR — signal-to-noise ratio, how much of the original is still visible relative to the noise drowning it — is 5^2 = 25× what was intended, at every single step.

Follow that through. The model never experiences a genuinely high-noise step, because even the noisiest step in its schedule still has the signal shouting through it. The high-noise steps are exactly where global layout gets decided. So the model never learns to produce a global layout at all.

The fix is one line: measure the latent standard deviation over the corpus once, and divide every latent by it. That is the origin of the famous 0.18215 constant in public checkpoints — it is one specific encoder’s measured scale, not a universal number.

The symptom if you skip it is distinctive: locally beautiful texture, globally incoherent composition. Nothing errors. The loss curve looks fine.

Assumptions in this section.

• State out loud: f = 8 and c = 16, patch size 2, and the four rFID values in the table. Those rFIDs are corpus-specific and would be re-measured on yours.
• Ask: what the product’s smallest meaningful detail is — small text, distant faces, fine print. That answer, not the rFID number on its own, decides whether f = 8 is acceptable.
• Load-bearing: that one autoencoder, trained once and frozen, serves every future generator. That is what amortizes its cost to nothing and what lets the diffusion model be retrained freely. If the autoencoder had to change with each generator, latent diffusion’s clean single-training-run advantage over the cascade in Cascaded super resolution the alternative largely disappears.
• Also load-bearing: that latent scaling is applied. Skip it and the model trains to a low loss and produces incoherent images, with no error message anywhere.

4. Cascaded super-resolution, the alternative

Latent diffusion has one serious competitor, and the argument against it is not the one most people reach for: the decisive objection is operational, not arithmetic.

The cascade approach is: generate a small image, then enlarge it with a chain of super-resolution (SR) models. Each SR model is itself a conditional diffusion model, taking the previous stage’s output as its condition and producing a larger version of it.

Two things to notice in the diagram below. First, the resolutions climb 64, 256, 1024 while the models get smaller, because the top stage only has to add local detail. Second, every stage runs entirely in pixel space, so there is no autoencoder anywhere and no compression ceiling — that is the cascade’s whole selling point.

LR in the diagram means low-resolution: the input each SR stage is handed. The “noise-conditioning aug” boxes are explained immediately after.

A note on the token counts, since they do not follow from the resolutions on their own: each stage patchifies more aggressively than the last. At p = 2 the 64-pixel base gives (64/2)^2 = 1,024 tokens; at p = 4 the 256-pixel stage gives (256/4)^2 = 4,096; at p = 8 the 1024-pixel stage gives (1024/8)^2 = 16,384. Without that escalation the top stage would be back in Why pixel space diffusion at 1024 is hopeless territory.

flowchart LR
    Z(["noise"]) --> B["Base model<br/>64 x 64 pixel space<br/>1,024 tokens · 1.05 TFLOP<br/>60 forwards"]
    B --> L1(["64 x 64"])
    L1 --> NCA1["Noise-conditioning aug<br/>corrupt the LR input,<br/>tell the model how much"]
    NCA1 --> S1["SR 64 -> 256<br/>4,096 tokens · 2.75 TFLOP<br/>30 forwards"]
    S1 --> L2(["256 x 256"])
    L2 --> NCA2["Noise-conditioning aug"]
    NCA2 --> S2["SR 256 -> 1024<br/>16,384 tokens · 12.68 TFLOP<br/>small model, 30 forwards"]
    S2 --> OUT(["1024 x 1024"])

Noise-conditioning augmentation, and the mechanism it fixes

Every cascade has one problem that is not optional, and the standard fix for it is an instance of a pattern worth recognizing everywhere.

Start with the mismatch. Each SR stage is trained on real low-resolution images — genuine photographs shrunk down. It is then deployed on generated ones.

Those are not the same thing. Generated low-resolution images carry the base model’s characteristic artifacts, and the SR stage has never seen those artifacts in training. That is a distribution shift: at run time the model is handed something statistically unlike anything it was shown. Worse, it compounds down the chain, because each stage’s own artifacts become the next stage’s unfamiliar input.

The fix is noise-conditioning augmentation, and it has two halves:

1. During training, deliberately corrupt the low-resolution input with a random amount of noise.
2. Tell the model how much noise you added, by passing the level in as an extra conditioning input.

The SR model then learns not one behaviour but a family of them, indexed by “how much should I trust this input”. At generation time you dial in the level that matches how artifact-laden the base model’s output actually is.

The general name for the problem is exposure bias — a model exposed only to clean inputs in training and dirty ones in deployment — and the general fix is the one used here: make the training distribution contain the corruption the model will actually see.

Cost, honestly

Price the cascade against latent diffusion honestly, and the price turns out not to be the reason to reject it.

Give each stage a model sized for its job. The top stage only needs to add local detail, so it is shallow and narrow.

Read the last column of the table below, and then the bottom two rows against each other: 525.8 TFLOP for the cascade against 351.2 for the single latent model. Note also that the largest single contribution — 380.4 TFLOP — is the top SR stage, even though it uses the smallest model in the chain. Its token count is fixed by the output resolution, and that is what you cannot design away.

1.5× the FLOPs (525.8 / 351.2 = 1.50), and that understates it. Size the SR stages generously rather than shaving them and the ratio goes past 4×, for the reason just given: the top stage runs at full pixel resolution and its token count is fixed by the output size no matter how you architect it.

Now compare the two designs on everything that is not FLOPs. The cascade wins exactly two rows — quality ceiling, and behaviour at 4K — and loses the rest. The row that decides the chapter is “retrain coupling”.

The decisive argument is not the 1.5×, it is the retrain coupling.

Latent diffusion has one training run, with one loss curve you can read. A cascade has three, and the second and third are conditioned on a distribution that changes whenever you touch the first. Improve the base model and you have not finished — you now have two SR stages tuned for artifacts that no longer exist. That is an operational cost you pay every quarter, forever, and it appears in no FLOP table.

Where the cascade wins

At 4K — 4096 pixels per side — the arithmetic turns around. Run the same latent construction: f = 8 gives a latent of 4096/8 = 512 per side, and patch size 2 gives (512/2)^2 = 65,536 tokens. That is 16× the token count at 1024, so 256× the attention cost, and attention is back to dominating.

The standard answer at that resolution is a hybrid: latent diffusion up to 1024, then either a refiner working in latent space or a lightweight pixel-space SR stage to reach 4K. That is a cascade whose base happens to be a latent model, and it is what production systems actually ship.

That design also needs tiled decoding, because a 4K decode does not fit in memory (Serving memory is the binding constraint).

Assumptions in this section.

• State out loud: three cascade stages sized d=1152/L=28, d=1024/L=16 and d=768/L=12, at 60, 30 and 30 forward passes. Size the SR stages more generously and the FLOP gap widens past 4×, which strengthens rather than changes the conclusion.
• Ask: what the maximum output resolution has to be within a year. The answer flips this decision — the cascade is wrong at 1024 and right beyond about 2K.
• Load-bearing: that the base model will be retrained regularly. The whole argument against the cascade is the coupling between its stages, and that coupling only costs you when something upstream changes. Freeze the base forever and the cascade’s three training runs become a one-time cost, and the decision comes back down to the 1.5× in FLOPs.

5. Architecture: U-Net vs DiT

Which network shape should the denoiser be? The short version: the U-Net was designed to solve the Why pixel space diffusion at 1024 is hopeless problem internally, and once an autoencoder is solving it externally, the U-Net is answering a question nobody is asking any more.

Read the two halves of the diagram below against each other, and notice that the top half changes shape at every step while the bottom half never does.

The U-Net (top) descends through convolution stages at falling resolution, turns attention on only once the grid is 32 × 32 or smaller, then climbs back up. The dotted arrows are skip connections: each downward stage’s output is carried straight across to the matching upward stage, so detail lost to downsampling can be restored on the way back.

The DiT (bottom) does none of that. 4,096 tokens go in, pass through 28 blocks of identical shape, and 4,096 tokens come out. Each block contains attention, an FFN (feed-forward network — the two fully connected layers every transformer block applies to each token independently), and adaLN-zero conditioning, explained below.

flowchart TD
    subgraph U["U-Net — compression inside the model"]
        U1["128x128 conv"] --> U2["64x64 conv"] --> U3["32x32 conv + attention"]
        U3 --> U4["16x16 conv + attention"] --> U5["32x32 up"] --> U6["64x64 up"] --> U7["128x128 up"]
        U3 -.->|"skip"| U5
        U2 -.->|"skip"| U6
        U1 -.->|"skip"| U7
    end
    subgraph D["DiT — compression already done by the autoencoder"]
        D1["4,096 tokens"] --> D2["Block 1<br/>attn + FFN + adaLN-zero"]
        D2 --> D3["Block 2 ... Block 28<br/>identical shapes"]
        D3 --> D4["4,096 tokens"]
    end

The U-Net’s design is a direct response to the Why pixel space diffusion at 1024 is hopeless problem: put attention only where the token count is small enough to afford it. Convolution stages carry the high-resolution work using a locality prior — the built-in assumption that nearby pixels matter most to each other, which is true of images and saves the model from having to learn it. Attention appears only at 32 × 32 and below. And the skip connections restore the spatial detail that downsampling threw away, so the fine structure survives the trip through the narrow middle (Pooling).

Once a trained autoencoder is doing the compression, the U-Net’s entire reason for existing has been outsourced. That is the one-sentence version of why the field moved.

Four supporting reasons follow, in the table below. Four terms in it need naming first:

• MFU — model FLOP utilization: the fraction of the hardware’s peak arithmetic rate the model actually achieves. It is mostly a matter of shapes, because GPUs run large uniform matrix multiplications far more efficiently than small ragged ones.
• Power law — the loss falls by a steady fractional amount for each doubling of compute. That regularity is what lets you predict the outcome of a training budget instead of running it to find out.
• Cross-attention — the standard way a U-Net takes a text condition: image features attend to the text, but never the reverse.
• MMDiT (multimodal diffusion transformer) — the alternative that becomes available once everything is one sequence. Text tokens and image tokens sit side by side and attend in both directions.

The table’s rows are ordered by how much each mattered in practice. The bottom row is the one honest advantage the U-Net keeps.

adaLN-zero, and why the zero matters

Build the term up from the bottom.

Layer normalization (LN) rescales each token’s vector to a standard size, then applies a learned scale and a learned shift. Those two are fixed constants once training finishes.

adaLN is adaptive layer normalization: the scale and shift are not constants. A small network produces them on the fly by reading the conditioning information — here the diffusion timestep and the class. That is how the condition gets into the network at all.

The DiT adds a third produced value, a gate, which multiplies the block’s entire output:

h  =  x  +  gate · Block( scale · LN(x) + shift )

The zero is the crucial part. The small network producing gate is initialized so that it outputs zero.

Follow what that means at training step 0. Every gate is zero, so gate · Block(...) is zero, so every block contributes nothing and h = x. The whole 28-block network is exactly the identity function, with the residual stream — the running sum each block adds into, described in Residual connections the gradient highway — passing straight through untouched.

So the network starts as a well-behaved identity function and switches blocks on one at a time, as each earns its gradient, instead of starting as 28 layers of random noise stacked on each other. That is worth several FID points against a naive initialization, and it is the difference between “trains” and “diverges in the first 2,000 steps”.

One accounting note, because two parameter counts float around this chapter. From Training, the adaLN networks are 6·d^2 parameters per block: 6 · 1152^2 · 28 = 223M. But they act on a single conditioning vector per image, not on every token, so they contribute nothing to the per-token FLOP count. That is why the model gets described as a ~675M-parameter model while every FLOP figure in this chapter uses 445.9M — the 12·L·d^2 that actually touches each token.

Assumptions in this section.

• State out loud: 35% matmul MFU for the U-Net and 55% for the DiT on H100. Those two are what the 55 / 35 = 1.57× is computed from. Different hardware moves both together, and the ratio is what matters.
• Ask: how much training data there actually is. The U-Net’s locality prior genuinely wins in the small-data regime, and the table says so.
• Load-bearing: that the data regime is large. Every advantage claimed for the DiT — that it learns spatial structure rather than assuming it, that its loss follows a clean power law — assumes enough data to pay for the missing prior. At small scale this section’s conclusion inverts, and the honest answer is a U-Net.

6. Sampling: DDPM, DDIM, and the step-count curve

How many denoising steps should the sampler run? The answer here comes from a fitted curve rather than a number copied from a paper. The two names in the heading are the two samplers being compared: DDPM (denoising diffusion probabilistic models), the original random-walk sampler, and DDIM (denoising diffusion implicit models), the deterministic one that replaced it. Beyond both sits distillation, the technique that removes most of the remaining cost — and, more importantly, quietly takes something away in exchange.

Why DDPM needs many steps

DDPM, the original formulation, needs on the order of a thousand steps — and the limit comes from mathematics, not from the network being weak.

DDPM generates by ancestral sampling: running the chain backwards one link at a time, in the reverse of the order the forward noising process created it. Each link is one Markov step — a step whose result depends only on the immediately preceding state and nothing earlier. The update rule is:

x_{t-1} = (1/sqrt(alpha_t)) · ( x_t - (beta_t / sqrt(1 - abar_t)) · eps_hat )  +  sigma_t · z

Reading it left to right: eps_hat is the network’s predicted noise, alpha_t and beta_t are the schedule’s per-step coefficients, sigma_t sets how much randomness the step re-injects, and z is a fresh random draw. So each update is partly a denoise (the bracketed part) and partly a re-randomization (the sigma_t · z on the end).

Here is the constraint that fixes the step count. The reverse of a Gaussian noising step is itself Gaussian only in the limit of infinitesimally small steps. Over a large step, the true reverse distribution is some more complicated shape, and a Gaussian approximates it badly — but a Gaussian is exactly what the update rule above assumes.

So the step size is bounded by the accuracy of a Gaussian approximation, not by the quality of the network. That is why the original formulation uses T = 1000, and why you cannot simply skip every twentieth step and keep the same update rule. A better-trained network does not buy you larger steps.

DDIM: drop the stochasticity, gain a solver

DDIM removes the randomness from the sampler, and that single change turns step-count reduction from impossible into a routine numerical-methods question.

The trick is to rewrite the update in terms of the predicted clean image rather than as a noisy jump. Two lines: first estimate what the finished image is, then re-noise that estimate down to the next noise level.

x_0_hat  =  ( x_t - sqrt(1 - abar_t) · eps_hat ) / sqrt(abar_t)

x_{t-1}  =  sqrt(abar_{t-1}) · x_0_hat  +  sqrt(1 - abar_{t-1}) · eps_hat
            \____ point at the estimate ____/   \___ re-noise to level t-1 ___/

The random draw z is gone. Nothing in either line samples anything, so the trajectory from noise to image is deterministic and a given starting seed maps to exactly one output.

That changes what the sampler is. The sequence of steps is now the numerical approximation — the discretization — of a smooth continuous path described by a probability-flow ODE: an ordinary differential equation whose solution carries pure noise continuously into an image. The steps are no longer links in a chain that must be followed one at a time; they are sample points on a curve.

Two consequences follow immediately:

1. You can take any evenly spaced subset of the timesteps. Going from 1000 steps to 50 to 20 is just solving the same equation on a coarser grid, not running a different model.
2. The remaining error is discretization error, so the accuracy of the numerical method applies. The naive Euler method — take the current slope and step along it — has error proportional to 1/N for N steps, written O(1/N). A second-order method such as Heun or DPM-Solver++ 2M, which corrects each step using slope information from more than one point, has error O(1/N^2) for the same number of network evaluations plus one reused derivative.

The step-count/quality curve, derived

With a deterministic sampler in hand, the step count stops being a copied convention and becomes a fitted curve — and the fit is what tells you where the returns stop.

Total error splits into a part that shrinks as you add steps and a part that does not. Write it down and the shape of the trade falls out:

FID(N)  =  FID_inf  +  C / N^k        k = solver order
           \______/    \_______/
           model error  discretization error

FID_inf is the quality the model would reach with infinitely many steps. It is the irreducible error, set by how good the trained network is, and no sampler change touches it. The second term is everything you lose by taking finite steps, and it is the only part you can buy down.

Fit that form to measured points and you get FID_inf = 7.3, with C = 60 for a first-order solver and C' = 240 at k = 2 for a second-order one.

Substitute once to see how the columns are produced. At N = 20, first order gives 7.3 + 60/20 = 7.3 + 3.0 = 10.3; second order gives 7.3 + 240/400 = 7.3 + 0.6 = 7.9. Every row below is that same substitution.

Two things to read off, and the second is the one people miss.

First, compare the bold row against the N = 100 row. A second-order solver at 20 steps gives 7.9, and a first-order solver needs 100 steps to give the same 7.9. That is a 5× compute saving from a change to the sampler alone, with no retraining and no quality loss.

Second, look at what happens past N ~ 50. Cost grows linearly in N while the benefit shrinks as 1/N^2. Going from 30 steps to 250 is 8.3× the cost for 0.3 FID (7.6 -> 7.30 on the second-order curve), and 0.3 FID is below the noise floor of the metric (Offline metrics) — meaning two runs of the same model can differ by that much.

The code below regenerates every row of the table. c_eff = 4 · c is how the second-order constant C' = 240 is produced from C = 60; the asserts check each printed value against the table.

def fid_vs_steps(n_steps: int, fid_inf: float = 7.3,
                 c: float = 60.0, order: int = 1) -> float:
    """Total error = irreducible model error + solver discretization error."""
    c_eff = c if order == 1 else 4.0 * c          # 2nd order: larger constant, /N^2
    return fid_inf + c_eff / n_steps ** order


def equivalent_first_order(n_second: int) -> float:
    """How many Euler steps match an N-step 2nd-order solver?"""
    target = fid_vs_steps(n_second, order=2)
    return 60.0 / (target - 7.3)


# Every row of the table above, at the precision the table prints it.
STEP_TABLE = [
    #  N,  1st order, dp,  2nd order, dp
    (  4,  22.3, 1,  22.3, 1),
    (  8,  14.8, 1,  11.1, 1),
    ( 16,  11.1, 1,   8.2, 1),
    ( 20,  10.3, 1,   7.9, 1),
    ( 30,   9.3, 1,   7.6, 1),
    ( 50,   8.5, 1,   7.40, 2),
    (100,   7.9, 1,   7.32, 2),
    (250,   7.54, 2,  7.30, 2),
]

for n, first, first_dp, second, second_dp in STEP_TABLE:
    f1, f2 = fid_vs_steps(n, order=1), fid_vs_steps(n, order=2)
    print(f"N={n:4d}  1st={f1:7.3f}  2nd={f2:7.3f}")
    assert round(f1, first_dp) == first, (n, f1)
    assert round(f2, second_dp) == second, (n, f2)

print(f"20 second-order steps == {equivalent_first_order(20):.0f} Euler steps")

assert round(fid_vs_steps(100, order=1), 1) == 7.9
assert round(fid_vs_steps(20, order=2), 1) == 7.9
assert round(equivalent_first_order(20)) == 100

Distillation, and what it costs

Distillation means training a second, cheaper model — the student — to reproduce in few steps what the trained model — the teacher — produces in many. It comes in three variants, and it carries a cost that is real and does not show up in the headline metric.

The three variants, in increasing order of aggression:

• Progressive distillation — train a student to take one step that lands exactly where two of the teacher’s steps land. Then repeat, with the student as the new teacher. Each round halves the step count: 50, 25, 12, 6, 3.
• Guidance distillation — train a student that takes the guidance scale w as an input and reproduces the guided prediction in a single forward pass. This removes the unconditional pass entirely: a clean 2× saving with no change in step count.
• Adversarial and consistency distillation — push all the way down to one to four steps by adding a critic network that judges the student’s output directly.

Now the cost, which has a mechanism worth being able to state.

The teacher’s ODE trajectory is a smooth curve. The student has to approximate that curve with N straight chords. As N falls, the map from starting noise to final image gets smoother and lower-entropy — entropy being the standard measure of how much variety a distribution contains. Fewer distinct starting points end up at distinct images, so the range of images the model can produce narrows.

That loss shows up in recall, not in precision. Both are defined in Precision and recall for generative models: precision is the fraction of generated images that look real, recall is the fraction of the real variety the model still covers. A few-step model is sharp — precision holds — and less diverse. It is the same axis the guidance scale moves along (Classifier free guidance): buy fidelity, pay in coverage.

In the table below, read the FID and Recall columns together rather than one at a time. FID barely moves down the first four rows while recall falls steadily, and that divergence is the entire argument.

Row 3 is the default. It is $0.00136 / $0.00046 = 3× cheaper than row 1 and better on FID (8.1 against 9.3), because upgrading the solver from first to second order more than pays for the small regression guidance distillation introduces.

Rows 5 and 6 are interactive-preview modes, not the delivered image. At 4 steps recall falls from 0.62 to 0.48 — 0.14 / 0.62 = 23% of the coverage gone. That is not a quality wobble. It means the product has quietly stopped showing certain kinds of output altogether, and nobody will report the absence of an image they never saw.

Assumptions in this section.

• State out loud: the fitted constants FID_inf = 7.3, C = 60 at first order and C' = 240 at second, plus the measured recall figures. All are fits to one model on one corpus and would be re-fitted for yours.
• Ask: whether the product has an interactive preview surface at all. That is the only place the few-step models belong, and without one, rows 5 and 6 are dead weight.
• Load-bearing: that FID’s noise floor is wide enough to make the 30-to-250-step difference meaningless, and that recall is measured at all. Drop recall from the dashboard and 4-step distillation reads as a free 9× saving: FID moves 4.5 points (7.9 -> 12.4), which looks survivable, while the model has stopped producing nearly a quarter of the distribution it used to cover.

7. Serving: memory is the binding constraint

Running this model in production means unlearning the intuitions carried over from serving language models, because two of them invert here: batching stops helping, and the memory ceiling is set by the decoder rather than the transformer.

One node in the diagram below does the deciding — the diamond. Above an output side of 1536 pixels the decode is split into tiles rather than done in one piece — the memory budget later in this section is what makes 1536 the threshold.

Two labels need naming. The safety cascade and watermarking step is the output-side screening chain from Serving architecture. A CDN is a content delivery network: the global cache of servers that puts the finished file near the user.

Note also that the batch is 2 and always 2 — it is the guidance pair from Why pixel space diffusion at 1024 is hopeless, packed together. The subsection after the diagram explains why it never grows.

flowchart TD
    REQ(["Request<br/>1024 x 1024"]) --> Q["Queue"]
    Q --> PACK["Pack the CFG pair<br/>into one batch of 2"]
    PACK --> GPU["DiT sampler<br/>20 steps, 2nd order<br/>4,096 tokens"]
    GPU --> LAT(["128 x 128 x 16 latent"])
    LAT --> TILE{"output side<br/>> 1536?"}
    TILE -->|"no"| DEC["VAE decode<br/>2.2 TFLOP<br/>~2.5 GB peak"]
    TILE -->|"yes"| TDEC["Tiled decode<br/>512-px tiles, 64-px overlap<br/>blend seams"]
    DEC --> SAFE["Safety cascade + watermark<br/>ch 07 section 9"]
    TDEC --> SAFE
    SAFE --> CDN(["CDN"])

A diffusion step is a prefill, not a decode

This is the contrast that reorders everything you learned from serving large language models, and it is the reason the serving design here looks nothing like an LLM’s.

The vocabulary first, since the whole argument rides on it.

A language model serves a request in two phases. Prefill processes the entire prompt at once, in parallel. Decode then emits one token at a time, each one depending on the last.

Decode is slow per token for a specific reason: it drags the whole model’s weights out of memory in order to do a tiny amount of arithmetic on a single token. It is memory-bandwidth-bound — the chip is waiting on memory, not on math. The standard cure is batching many users together, so that one trip through the weights serves all of them at once. Decode also relies on a KV cache: a store of the intermediate keys and values from previous tokens, kept so they need not be recomputed at every new token.

Now the measurement that tells you which regime you are in.

Arithmetic intensity is how many FLOPs an operation performs per byte it moves from memory. Compare it against the hardware’s ridge point, the ratio at which the chip’s peak arithmetic rate and its peak memory bandwidth are balanced. Above the ridge point the operation is limited by compute; below it, by memory.

Here the dominant operation is a (B·n, d) × (d, d) projection — the per-token linear layers — at batch size B = 1, n = 4096 tokens, and width d = 1152. The byte count has three parts: the input activations 4096 × 1152, the weight matrix 1152 × 1152, and the output activations 4096 × 1152, each at 2 bytes per number because weights and activations are 16-bit.

FLOPs  =  2 · 4096 · 1152 · 1152                          =  1.087e10
bytes  =  2 · (4096·1152 + 1152·1152 + 4096·1152)         =  2.153e7
intensity  =  1.087e10 / 2.153e7                          =  505 FLOP/byte

H100 ridge point  =  990e12 FLOP/s / 3.35e12 byte/s       =  296 FLOP/byte

505 is greater than 296, so the model is compute-bound even at batch size 1.

The reason is structural. A diffusion sampling step processes all 4,096 tokens in parallel, so it is a prefill — and then it repeats that prefill 20 times. There is no decode phase anywhere, because nothing is autoregressive: no token depends on a previously emitted token, and nothing carries over between steps except the latent image itself. For the same reason there is no KV cache to amortize.

Contrast that with language-model decode (The kv cache the most important mechanism in this chapter), which is memory-bandwidth-bound at batch 1 and where batching is the entire optimization.

The consequence is the opposite serving strategy. Batching a diffusion model buys perhaps 1.1 to 1.2× — from amortizing kernel-launch overhead, the fixed cost of dispatching each operation to the GPU, and from better tile efficiency — and then it goes flat, because there was never any memory-bandwidth waste to recover. Meanwhile it adds latency in proportion to the batch size, since everyone in the batch waits for the slowest member. So:

• Use batch 2, which is the guidance pair you were going to run anyway. It is free.
• Scale out with replicas — more independent copies of the service — not with batch size.
• The one place batching still helps is the decode, which is a small convolutional network where launch overhead is a larger share of the total.

The memory budget, and what actually binds

Account for every gigabyte, and the total decides both the tiling threshold in the diagram and how far the design can be pushed before it stops fitting on a GPU at all.

The budget below is for an 80 GB H100 serving at 1024. Three terms in it:

• bf16 is the 16-bit floating-point format the weights are stored in, so two bytes per number. That is why every line multiplies by 2.
• autograd is the bookkeeping a framework does to enable training. It is off at inference; leave it on and it would dominate this table.
• CUDA context is the fixed overhead the GPU driver and compiled kernels occupy before any of your data arrives.

Compare the last two blocks against each other; that comparison, not the absolute numbers, is what decides the design.

FIXED
  DiT weights          669M params, bf16                    =   1.34 GB
  VAE (enc + dec)      84M params, bf16                     =   0.17 GB
  CUDA context, allocator, kernels                          =   1.50 GB
                                                               --------
                                                                3.01 GB

PER SAMPLE — diffusion (no autograd, FlashAttention)
  residual stream      4096 · 1152 · 2 bytes                =   9.4 MB
  FFN intermediate     4096 · 4608 · 2 bytes                =  37.8 MB
  ~3 layers live at once, plus workspace                    =  ~0.3 GB

PER SAMPLE — VAE decode at 1024
  one full-res feature map  1024 · 1024 · 128 · 2 bytes     =   268 MB
  the decoder holds input + skip + output at full res,
  and a 256-channel stage at 512^2 of the same size
                                                            =  ~2.5 GB

Peak memory is the decoder, not the transformer — by roughly 8×. The diffusion model works on 4,096 tokens; the decoder works on 1,048,576 pixels with 128 channels. This inverts at every resolution step: at 2048 the decode peak is ~10 GB per sample, and at 4K it is ~40 GB and you are out of memory on a single image. That is why tiled decoding with overlap-and-blend exists, and why the tiling threshold is a resolution, not a batch size.

Tiled decoding means decoding the latent in overlapping square pieces and blending the overlaps together, so that no single piece ever needs the memory of a full-resolution image. It buys the memory back, and it costs two things.

Cost one: seams. A seam appears wherever the decoder’s receptive field — the region of input each output pixel depends on, from Why you cannot just delete the attention — reaches beyond the overlap into a neighbouring tile that was decoded separately. The two tiles then disagree slightly, which shows up as a faint discontinuity, most visibly in smooth gradients such as skies. The usual settlement is a 64-pixel overlap on 512-pixel tiles, blended with a cosine weighting so each tile fades into the next rather than ending abruptly.

Cost two: redundant compute. With 512-pixel tiles overlapping by 64, consecutive tiles start 512 - 64 = 448 pixels apart. So the decoder processes a 512-wide square for every 448 pixels of new output:

per side           512 / 448                    =  1.143 processed per delivered
in two dimensions  (512/448)^2                  =  1.31
                                                =  31% redundant, in the limit

at 2048, finite tile counts make it worse:
  tiles per side   ceil((2048 - 512)/448) + 1   =  5,  so 25 tiles
  pixels processed 25 · 512^2                   =  6,553,600
  pixels delivered 2048^2                       =  4,194,304
                   6,553,600 / 4,194,304        =  1.5625  =  56% redundant

The finite case is worse than the limit because the edge tiles hang off the side of the image and have nothing to amortize against.

Assumptions in this section.

• State out loud: an 80 GB H100, 4,096 tokens, a 128-channel decoder at full resolution, 512-pixel tiles with 64-pixel overlap, and a tiling threshold at 1536. Every memory figure follows from those.
• Ask: the maximum output resolution the product will ask for within the year. The decode peak scales with output pixels, and 4K needs ~40 GB per sample, so that answer decides whether tiling is an optimization or a requirement.
• Load-bearing: that a diffusion step is compute-bound at batch 1. The entire serving strategy — batch 2, scale with replicas — rests on that one measurement. If it were bandwidth-bound like language-model decode, large batches would be the correct answer and the fleet would be sized completely differently.

8. Cost per image, end to end

Everything so far converges on one number — $0.00091 per image — and on a ranking of every available lever against the one decision that actually mattered.

The block below runs top to bottom: FLOPs per image, then seconds, then dollars, then fleet size. Every line is arithmetic on the line above it, so you can rebuild the whole thing from the CONFIG line alone.

CONFIG   1024 x 1024, f=8 c=16 latent, DiT-XL, 20 steps 2nd-order, CFG

diffusion    20 steps × 2 forwards × 5.817 TFLOP           =  232.7 TFLOP
VAE decode   conv stack to 1024^2                          =    2.2 TFLOP
safety       NSFW + ArcFace + ANN over 3M vectors          =   ~0.02 TFLOP
                                                              -----------
                                                              234.9 TFLOP

H100 at 300 TFLOP/s effective    234.9 / 300               =  0.783 s GPU
$2.50/GPU-hour  =  2.50 / 3600  =  $0.00069444/s
  0.783 s × $0.00069444/s                                  =  $0.00054375 at 100%
                                                    rounded    $0.00054
at 60% fleet utilization         0.00054375 / 0.60         =  $0.00090625 per image
                                                    rounded    $0.00091

                                                              $0.91 / 1,000 images
at 2M images/day                 2e6 × $0.00090625         =  $1,813/day
steady-state load                2e6 × 0.783 s / 86,400 s  =  18.1 GPUs
provision for the diurnal peak   18.1 × 2.2                =  40 GPUs

The chain carries unrounded figures, which is why the last digit looks off. Divide the printed $0.00054 by 0.60 and you get $0.00090, not $0.00091. The extra digit is real: 0.783 × 2.50 / 3600 is exactly $0.00054375, and $0.00054375 / 0.60 is $0.00090625, which is what rounds to $0.00091 and what makes 2e6 × come to $1,813/day rather than $1,820. The chapter quotes the rounded $0.00091 everywhere else.

That last pair of lines is the one people run together. 18.1 GPUs is the average load; 40 is what you have to own. The daily peak is 2.2× the mean, and a GPU cannot be conjured for the busy hour, so provisioning follows the peak while the bill above follows the mean.

The safety line covers the output-side screening chain from Serving architecture: an NSFW classifier (not safe for work — sexual or graphic content), an ArcFace face-embedding match against known real people, and an ANN — approximate nearest neighbour — search over roughly 3 million stored vectors.

On the 300 TFLOP/s, because two different utilization numbers appear in this chapter and confusing them is a common error.

The 300 is the end-to-end effective rate — about 30% of the H100’s 990 TFLOP/s peak in bf16. It is the right number for costing because it absorbs everything the FLOP count ignores: attention kernels, normalizations, the adaLN conditioning path, the decoder, and the gaps between kernel launches.

The 55% quoted in Architecture u net vs dit is a different quantity. It is the DiT’s matmul MFU — the utilization of the matrix multiplications alone — and it appears there only as a ratio against the U-Net’s 35%.

Use the end-to-end rate for money and the matmul MFU for architecture comparisons, and never multiply the two together.

Latency

Now the wall clock, at batch 2 — the guidance pair, run together. WebP is the compressed image format the result is encoded into before upload. Notice that the sampling loop is 94% of the GPU path and the decode is about 1% of it, which is the reverse of the memory picture in Serving memory is the binding constraint.

per step   2 × 5.817 TFLOP / 300 TFLOP/s                   =    38.8 ms
20 steps                                                   =   776 ms
VAE decode  2.2 TFLOP                                      =     7 ms
safety cascade (3 nets + ANN)                              =    40 ms
                                                              --------
p50 GPU path                                                   823 ms
+ queue, transfer, encode to WebP, CDN put                     ~700 ms
p50 end to end                                              ~1.5 s
p95 (queue depth at peak)                                   ~3.1 s

The lever table

Every row below is the finished design with exactly one decision changed, priced against the baseline. The point of reading it is not the individual numbers but the ratio between the first row and all the others.

Read the first row against everything below it. The latent-space decision is worth $33.06 / $0.00091 = 36,500×. Every remaining lever in the table is under 10×, and the two cheapest of those are paid for in coverage rather than in dollars.

Once you are in latent space the cost problem is solved, and the remaining engineering is about not spending the savings on diversity.

Assumptions in this section.

• State out loud: 20 second-order steps with guidance on, an f=8, c=16 latent, an H100 at 300 TFLOP/s effective, $2.50 per GPU-hour, 60% fleet utilization, 2M images a day, and a 2.2× diurnal peak. Every dollar and GPU-count figure is a product of those seven.
• Ask: the real daily volume and the shape of its daily peak. Cost is set by the mean and provisioning by the peak, and quoting one number for both is how fleets end up half-idle or under water.
• Load-bearing: the 300 TFLOP/s effective rate and the 60% utilization. Between them they set every number in this section. Take the hardware’s 990 TFLOP/s sticker rate instead and you understate the bill by roughly 3×.
• Also load-bearing: that the levers below the first row are genuinely small. That is what licenses spending the next quarter on quality rather than on cost, and it is the single most useful conclusion in the chapter.

9. Metrics at high resolution

The cost is settled; measurement at this resolution is not, because the metric the whole field quotes stops carrying information above about 300 pixels.

FID stops working, and the reason is embarrassing. The Inception-v3 network that FID is computed through accepts 299 × 299 input, and nothing else. A 1024 × 1024 image is therefore shrunk by 1024 / 299 = 3.42× per side before the metric ever sees it. Count what survives:

pixels generated   1024 · 1024  =  1,048,576
pixels FID sees      299 ·  299  =     89,401
fraction discarded                =     91.5%

You spent about 25× the compute of a 256-pixel model to produce detail the metric literally cannot see.

Where that 25× comes from — this chapter’s own thesis, applied to itself. A 256-pixel model at f=8, p=2 has (256/8/2)^2 = 256 tokens, against 4,096 at 1024, so the token count rises 16×. The token-linear term rises with it, 16×. The attention term rises with the square, 256×. Blend the two at their actual sizes and total compute rises 24.6×: 0.2368 TFLOP per forward at 256 px against 5.8175 at 1024.

So a model with perfect global structure and mushy skin pores scores identically to one that has both, and the number everyone reports cannot tell them apart.

The table below is what to report instead. Two terms in it: VQA is visual question answering — you ask a VLM (a vision-language model, one that can look at an image and answer questions about it in text) a set of questions derived from the request, and score how many it gets right. AE is the autoencoder. Note the cost column: three of the five are free, and the expensive one — human raters counting named defects — is the only one that catches failures no feature extractor was trained to notice.

Assumptions in this section.

• State out loud: that FID is computed through Inception-v3 at 299 × 299, which is what makes the 91.5% figure exact, and roughly $900 per checkpoint for a human defect count.
• Ask: what the product’s users actually complain about. The defect taxonomy is only as good as the list of named classes in it, and that list is a product question, not a modelling one.
• Load-bearing: that high-frequency quality matters to the product at all. The entire case for patch-FID, and for spending about 25× the compute of a 256-pixel model, rests on it. If users view the output at thumbnail size, the honest answer is to generate at 512 and stop reading this chapter.

10. Failure modes

The four failures users report most often — text, hands, tile seams and duplicated subjects — each trace down to a mechanism, and one pattern recurs across all of them: each is a discrete global constraint over a tiny area, and the loss function is per-element and knows nothing about either.

Text rendering, traced

Illegible text is the most-reported failure of image generators, and it has three independent causes stacked on top of each other. Only the last is really the generator’s fault, which is why “train harder” does not fix it.

Cause 1 — the autoencoder destroys the letter shapes before diffusion is involved at all.

A glyph is the drawn shape of a character. At f=8, c=4, one latent cell has to represent an 8 × 8 block of pixels — that is 8 × 8 × 3 = 192 subpixel values compressed into 4 numbers, a 48× reduction, in one cell.

Now size a letter against that cell. A 12-point glyph in a 1024-pixel image has strokes about 2 pixels wide. The stroke is a quarter the width of the cell that has to encode it, in 4 numbers. There is nowhere for it to live.

The trace below measures exactly that, with no diffusion model anywhere in the loop.

TRACE — encode/decode a page of rendered text, no diffusion at all

  f=8  c=4    36 pt text   OCR character error rate   4%
  f=8  c=4    12 pt text   OCR character error rate  31%
  f=8  c=16   12 pt text   OCR character error rate   9%
  f=4  c=4    12 pt text   OCR character error rate   3%

  -> the generator has never been in the loop. The ceiling is the AE.

Two abbreviations in that trace. OCR is optical character recognition — software that reads text out of an image. CER is character error rate, the fraction of characters it gets wrong.

Read the four rows against each other. Every row is encode-then-decode with no generator involved, so the 31% on row 2 is damage the compression did on its own. Row 3 shows 16 channels fixing most of it (31% to 9%) at no change in token count; row 4 shows f=4 fixing nearly all of it at 8.5× the diffusion cost.

Cause 2 — the conditioning never contained the letters.

A subword text encoder splits text into common fragments rather than individual characters, so STARBUCKS arrives as two or three chunks with no character-level structure inside them.

On top of that, CLIP — contrastive language-image pretraining, the standard network that maps text and images into a shared space, and the usual source of a diffusion model’s text conditioning — was never trained with any character-level supervision.

So the model is not failing to render the letters. It was never told what the letters are.

Cause 3 — the objective does not care.

The training loss is squared error in latent space. Turning an E into an F moves a handful of latent numbers by a small amount. Nothing in the loss says “the sequence of glyphs must be exactly right”, so gradient pressure toward correct spelling is proportional to the area affected — and a word occupies a fraction of a percent of the image.

The fixes, in order of effect:

1. 16-channel latents, for cause 1. That alone takes character error rate from 31% to 9%, before anything else changes, and costs nothing in token count.
2. A character-aware text encoder such as ByT5 — a model that works directly on raw bytes, so individual letters survive — run alongside the semantic one, for cause 2.
3. Glyph-conditioned training, or a loss that weights text regions specially, for cause 3.

Hands and anatomy

Wrong hands are the second most-reported failure, and the argument has the same shape as text with different constants. Start by sizing the defect against the image. A hand in a 1024-pixel image occupies roughly 100 × 100 pixels:

hand area / image area  =  10,000 / 1,048,576  =  0.95%
in latent, at f=8:         12 x 12 cells out of 128 x 128  =  0.88%

A completely wrong hand costs under 1% of the reconstruction loss, while a slightly-off sky costs 40%. The loss is computed element by element, so it weights defects by area, and this defect is a discrete global constraint — “exactly five fingers” — over a tiny area.

Then add uncertainty on top. Hands appear in enormously varied poses with heavy self-occlusion — fingers hiding other fingers — so what belongs in a hand-shaped region, given everything else in the picture, is genuinely uncertain. Under uncertainty a regression model does what regression models do: it hedges.

Hedging over a continuous variable gives you a blur. Hedging over a discrete count gives you six fingers.

The practical fixes are all on the data side, because the loss cannot be talked out of averaging: oversample crops containing hands; weight the loss more heavily in hand regions; or condition on pose keypoints — explicit coordinates for the joints — so the count is supplied rather than inferred.

Aspect-ratio artifacts

The third failure appears the moment you serve a shape you did not train on — and the symptom is specifically duplicated subjects, not generic distortion.

Train at square 1024, serve 1536 × 640, and two things break at once. First look at what changes in the input:

trained:   128 x 128 latent cells  ->  64 x 64 patches  =  4,096 tokens
served:    192 x  80 latent cells  ->  96 x 40 patches  =  3,840 tokens

1. The position embeddings are out of distribution. A position embedding is the vector that tells the transformer where in the grid a token sits, since attention is otherwise blind to order. A learned 64 × 64 grid of them simply has no entry for column 95. Filling one in by interpolating between neighbours is a guess; extrapolating past the edge is worse.
2. Attention entropy shifts with token count. Attention weights come from a softmax, which turns a row of scores into probabilities summing to one. Running that softmax over 3,840 candidate keys instead of 4,096 changes how sharply the weights concentrate — the effective temperature of every attention distribution in the network (Attention derived as content based lookup).

The signature symptom is duplicated subjects at wide aspect ratios — two heads, two horizons. The mechanism is worth stating precisely.

The model learned local statistics consistent with “one subject fills a square frame”. At double the width, those same local statistics are equally well satisfied by two subjects side by side. Nothing prefers one. There is no global “exactly one” constraint anywhere in the objective, for the same reason there is no “exactly five fingers” constraint.

The fix has two parts:

• Aspect-ratio bucketing during training. Group the real images into a handful of aspect-ratio buckets, keep the total pixel area roughly constant across buckets so the token count barely moves between them, and train on all of them.
• RoPE (rotary position embedding), which encodes position as a rotation applied to the attention vectors rather than as a lookup in a fixed table. A position the model never saw is then handled by the same rule as every other position, instead of being a missing table entry.

Prompt adherence versus aesthetic quality

The fourth failure is not a modelling bug at all but an optimization working exactly as specified against the wrong target. It is the clearest case of reward hacking in the chapter.

Fine-tuning a generator on human preference data optimizes the expected reward E[reward] — the average reward over the images the model produces. That reward comes from one of two places: a reward model, a network trained to predict which of two images a human would prefer; or DPO (direct preference optimization), which skips the separate reward model and tunes the generator on preference pairs directly.

Either way, the preference data was collected as pairwise aesthetic comparisons. And aesthetic preference correlates with saturated colour, shallow depth of field, symmetry, and centred composition.

It does not correlate with “included all five requested objects”, because a rater comparing two images rarely re-reads the prompt. So that is not what the fine-tuning optimizes. The measurement below shows what happens.

after aesthetic fine-tuning, measured on the same 500 prompts:

  aesthetic score       6.21  ->  6.63     (+0.42)
  VQA prompt adherence  0.71  ->  0.68     (-3 pp)
  recall (coverage)     0.62  ->  0.52     (-0.10)

This is reward hacking with a pretty face. Reward hacking is the general phenomenon of a system maximizing the stated objective by a route that defeats the purpose of stating it. The model found the region of image space the reward model likes and moved there — exactly what you asked for, and not what you wanted.

The control is to report the two numbers as a pair and gate on the adherence metric, never to ship on the aesthetic number alone. It is the same failure shape as optimizing deflection without reopen rate in case study 06.

Assumptions in this section.

• State out loud: the measured character error rates, the 100 × 100-pixel hand, and the three aesthetic-fine-tuning numbers. All are measurements on one corpus and one checkpoint.
• Ask: whether the product needs legible text at all. If it does, c=16 latents and a character-aware encoder are requirements rather than improvements. If it does not, cause 1 can simply be accepted.
• Load-bearing: the shared diagnosis under all four failures — that the loss is per-element, and every one of these defects is a discrete global constraint over a small area. That claim predicts rather than merely describes: it says up front that any new defect of the same shape (exactly one horizon, exactly two hands, a correct clock face) will also resist being trained away, and that the fixes must come from the data or the conditioning rather than from more capacity.

The rest

The remaining six, each with its mechanism, the measurement that reveals it, and the control. Read the Detection column hardest — it is the difference between a failure you catch on a dashboard and one a user reports.

Three terms first. Std is standard deviation. An FFT (fast Fourier transform) decomposes an image into the repeating patterns it is made of, which makes a faint regular grid show up as a sharp spike at its own frequency. OOM is out of memory.

11. Alternatives rejected

The rejected alternatives matter for the reasons attached to them, because every reason names the assumption that would have to change to reverse it. The first row is 36,500× and the rest are single digits — that spread is the shape of the whole chapter.

12. Interviewer pushback

These are the twelve questions this design is actually asked, what each is testing, and an answer that survives the follow-up. Nearly every one attacks an assumption rather than a fact, so each answer names the mechanism instead of restating the conclusion. Read the italic line under each question first — it tells you what the answer has to demonstrate, which is usually not what the question literally asks for.

“Why can’t you just run the chapter 07 model at 1024?” Testing: whether you can do the arithmetic rather than assert the conclusion. Because attention is quadratic in token count and pixels are tokens. At 1024 that is 1,048,576 tokens, so the attention term is 4·n^2·d·L = 141,863 TFLOP per forward against 935 for the parameter term — attention is 99.3% of the cost. Sixty forwards for a CFG-guided 30-step sample is 8.57e18 FLOPs, which at 300 TFLOP/s effective is 7.93 hours on an H100 and $33.06 an image. And FlashAttention does not save you: it removes the 2.2 TB score matrix, not the FLOPs.

“Derive the latent-diffusion saving for me.” Testing: the centerpiece. Get this wrong and nothing else lands. An f=8 autoencoder downsamples 8× per side, so token count falls by 8^2 = 64. Attention scales as n^2, so attention cost falls by 64^2 = 4,096 — and that is exact, not approximate: 141,863 TFLOP becomes 34.6. Patchifying at p=2 folds four latent cells into one token for another 4×, so 256× on tokens and 65,536× on attention. The token-linear term only falls by 256×, so the end-to-end saving lands between the two: 142,798 TFLOP down to 5.82, which is 24,546×. The structural point is that compression moves you back below the n = 6d crossover, so the model’s cost is proportional to its parameters again.

“What does the autoencoder cost you?” Testing: whether you volunteer the downside. A hard quality ceiling. The diffusion model can never beat what the decoder can reconstruct, so I measure rFID before training anything: 0.74 at f=8, c=4. The failure is not uniform — it is concentrated in high-frequency structure, so small text goes from 0% to 31% character error rate through the autoencoder alone, with no diffusion involved. The fix that costs nothing is 16-channel latents: same token count, only the input and output projections change, and rFID drops to 0.28 and text CER to 9%. Going to f=4 also works and costs 8.5× the diffusion compute, which is the wrong trade.

“Why not a cascade? Google shipped one.” Testing: whether you can argue both sides. It is a legitimate design and at 4K it is the right one. On FLOPs it is 526 against 351, so 1.5× — and if you size the SR stages properly rather than shaving them, more like 4×, because the top stage runs at full pixel resolution no matter what you do to its depth. But the FLOPs are not the argument. The argument is that stages 2 and 3 are trained on ground-truth low-res and deployed on generated low-res, so every base-model change shifts the distribution they were tuned for. That is three coupled training runs re-tuned every quarter versus one. What I would actually build at 4K is a hybrid: latent diffusion to 1024, then one refiner stage — a cascade whose base is a latent model.

“U-Net or transformer, and why?” Testing: whether “everyone uses DiT now” has a reason behind it. Transformer, and the reason is that the U-Net’s job was already done. A U-Net exists to put attention only where the resolution is low enough to afford it — but the autoencoder already did the downsampling, so the multi-resolution machinery is solving a problem that no longer exists. What is left are four advantages: (L, d) scales predictably where per-stage channel multipliers do not; 55% versus 35% MFU on H100, so 1.57× wall clock before any quality claim; the locality prior stops paying at scale, same as CNN to ViT; and conditioning is cleaner because condition tokens can join the sequence instead of being cross-attended in at chosen depths. U-Net is still better in the small-data regime, and I would say so.

“How many sampling steps, and how did you choose?” Testing: whether the number is measured or copied. Twenty, with a second-order solver. Total error is model error plus discretization error, and discretization error is O(1/N^k) where k is the solver order — so I fit FID(N) = 7.3 + C/N^k on measured points. First order gives 7.9 at 100 steps; second order gives 7.9 at 20. Same quality, 5× the compute, purely from the sampler, no retraining. Beyond 50 steps you are paying linearly for a 1/N^2 benefit: 30 to 250 steps is 8× cost for 0.3 FID, which is inside the metric’s noise floor.

“Distill to 4 steps and cut your bill 9×. Why haven’t you?” Testing: whether you know what distillation costs. Because the cost shows up in recall, not FID. FID goes 7.9 to 12.4, which looks survivable on a dashboard, but recall goes 0.62 to 0.48 — the model has quietly stopped producing 14 points’ worth of the output distribution. The mechanism is that the student approximates a curved ODE trajectory with four chords, so the noise-to-image map gets smoother and lower-entropy. I do use it, for interactive previews where the user is scrubbing and diversity across a session matters less than latency. The delivered image runs the full sampler. What I take for free is guidance distillation: that removes the unconditional forward for a clean 2× with recall down 0.02.

“What is your serving batch size?” Testing: whether you transferred LLM intuition without checking. Two — which is the CFG pair, so effectively one. A diffusion step processes all 4,096 tokens at once, so it is a prefill, not a decode. Arithmetic intensity of the dominant matmul is 505 FLOP/byte against an H100 ridge point of 296, which means compute-bound at batch 1. Batching buys maybe 1.15× from launch amortization and costs latency linearly. This is the exact opposite of LLM decode, where batching is the whole optimization because decode is bandwidth-bound and there is a KV cache to amortize. Diffusion has no KV cache — nothing carries between steps except the latent.

“What actually runs you out of memory?” Testing: whether you have profiled or guessed. The VAE decoder, by about 8×. The transformer works on 4,096 tokens — roughly 0.3 GB per sample of live activations at inference. The decoder works on 1,048,576 pixels at 128 channels, so a single full-resolution feature map is 268 MB and it holds several: about 2.5 GB per sample at 1024. That scales with output pixels, so 10 GB at 2048 and 40 GB at 4K, and you are out of memory on one image before you are out of roadmap. Hence tiled decoding above 1536, 512-px tiles with 64-px overlap and a cosine blend — stride 448, so (512/448)^2 = 1.31 and I pay 31% redundant compute, and the seams stay under the perceptual threshold as long as the overlap exceeds the decoder’s receptive field.

“Why can’t it spell?” Testing: whether you can decompose a symptom into mechanisms. Three causes and only one is the generator. The autoencoder destroys glyphs first: at f=8, c=4 one latent cell covers an 8 × 8 block, 192 subpixel values into 4 numbers, and a 12 pt stroke is 2 px wide. Encode-decode alone, with no diffusion at all, takes character error rate from 0% to 31%. Second, the text encoder is subword, so the model was never told the letters — CLIP has no character supervision. Third, the loss is L2 over latents, so an E becoming an F is a tiny error over a tiny area; there is no term demanding an exact glyph sequence. So: 16-channel latents first, because that alone takes CER to 9%; a character-aware encoder second; glyph-region loss weighting third.

“You ask for 1536 × 640 and get two people. Explain.” Testing: whether you reach for a mechanism or say “it’s out of distribution”. Two things break together. The position embedding was learned on a 64 × 64 patch grid and there is no entry for column 95, so the model is extrapolating. And token count moves from 4,096 to 3,840, which changes the softmax normalization and therefore the effective attention temperature. But the duplication itself has a sharper cause: the model learned local statistics consistent with a subject filling a square frame, and at double the width those same statistics are equally well satisfied by two subjects. Nothing in the objective says “exactly one” — the same reason nothing says “exactly five fingers”. The fix is aspect-ratio bucketing at training time with roughly constant pixel area per bucket, plus RoPE so position extrapolation is principled instead of a lookup miss.

“Your aesthetic score went up 0.4 after preference fine-tuning. Ship it?” Testing: whether you recognize reward hacking. Not on that number alone. The reward model was trained on pairwise aesthetic comparisons, and raters comparing two images rarely re-read the prompt — so the reward correlates with saturation, bokeh (the soft out-of-focus background a wide aperture produces), and centred composition, not with including all the requested objects. Measured on the same 500 prompts, adherence went from 0.71 to 0.68 and recall from 0.62 to 0.52. The model found the region the reward model likes and moved there, which is what I asked for and not what I wanted. Gate on adherence and coverage; treat aesthetics as the thing being traded away.

“Your FID is great at 1024. Convince me it means anything.” Testing: whether you inherited chapter 07’s skepticism. It means less than at 512, because Inception-v3 takes 299 × 299 input — a 1024 image is downsampled 3.42× per side, so 91.5% of the pixels I spent about 25× the compute of a 256-pixel model to produce are thrown away before the metric is computed. A model with perfect layout and mushy pores scores the same as one with both. I would report patch-FID over random 299-crops at native resolution so high-frequency quality is in scope, keep precision and recall to separate fidelity from coverage, and keep a human defect-taxonomy count for the failures no feature extractor was trained to notice — text, hands, seams, duplicated subjects.

The assumption ledger

Every assumption the design has leaned on, gathered in one place, so that you can state its foundations in twenty seconds and say what replaces the design when each one fails.

Each assumption goes in one of three bins, the same three used in ch 01:

• State it — you are free to pick, and being wrong costs you a re-derivation, not the design.
• Ask it — the answer changes the architecture, so it is worth an interviewer’s time to raise.
• Load-bearing — if it is wrong the design is not suboptimal, it is invalid.

The table is sorted by bin, load-bearing first. The last column is the one to read: it is what you would build instead.

The sentence that makes this visible to an interviewer: “This design rests on three things. One, that the denoiser genuinely needs long-range dependencies — that is what rules out a pure convolutional model and makes compression unavoidable rather than merely economical. Two, that one autoencoder trained once serves every future generator, which is what amortizes its cost to nothing and lets me retrain the diffusion model freely. Three, that a diffusion step is compute-bound at batch size 1 — I measured 505 FLOP per byte against a ridge point of 296 — because that single number inverts every serving instinct carried over from language models and tells me to scale with replicas instead of batches.”

Next: 09 — Text-to-Image.