How to design high-resolution image synthesis

Generate 1024 x 1024 at scale by compressing 8x per side in a frozen autoencoder first; every other choice just spends the 24,546x that decision buys.

The core lever

• Diffusion model: network trained to remove a little noise; run over N steps from pure noise down to an image.
• Denoiser is a DiT (diffusion transformer), not a U-Net. Attention cost is O(n squared) in tokens: pixels are tokens, so pixel diffusion is hopeless.
• Attention and token-linear terms are equal at n = 6d tokens (6,912 at d=1152); pixel space sits 152x past it, latent space below it.

1024 pixels = 1,048,576 tokens  →  142,798 TFLOP/forward  →  ~8 h, $33/image
 f=8 autoencoder → 16,384 cells → 4,096 tokens (p=2)  →  5.82 TFLOP  →  $0.00091

Latent diffusion

• Encoder E maps H x W x 3 to (H/f) x (W/f) x c; decoder D maps back. Grid = latent, trained once and frozen.
• 8x per side → 64x fewer tokens → 64² = 4,096x less attention. Patchify p=2 adds 4x fewer tokens, 16x less attention.
• Attention charges for positions, not values: raising channels c is nearly free, raising f is not. Use c=16.
• Latent scaling (mandatory): divide every latent by its corpus std once, or SNR is off and global layout never trains. Origin of the 0.18215 constant (dataset-specific). Symptom: locally beautiful, globally incoherent, loss looks fine.
• Autoencoder loss = L1 + LPIPS + PatchGAN + KL. Adversarial term is safe here (reconstruction anchors it) though rejected for standalone generation.
• rFID (reconstruction FID): measure the AE ceiling before training the generator.

Cascade vs latent

• Cascade: small base + super-res stages (64→256→1024), pure pixel space, no AE ceiling. 525.8 TFLOP/image (1.5x latent), wins only at 4K+.
• Noise-conditioning augmentation: corrupt the LR input in training and pass the level in, fixing the train-clean/serve-dirty exposure bias between stages.
• Decisive objection is operational, not FLOP: retrain coupling — changing the base re-tunes all SR stages; latent needs 1 training run + a frozen AE.

Architecture and sampling

• U-Net compresses inside the model (attention only at low res, skip connections); once an AE compresses externally its job is done → DiT.
• DiT: 55% matmul MFU vs U-Net 35% = 1.57x wall clock; one (L, d) knob follows a clean power law. U-Net still wins on small data.
• adaLN-zero: gate initialized to zero → at step 0 the whole net is identity via the residual stream; blocks switch on as they earn gradient. Difference between “trains” and “diverges.”
• Params: 445.9M (12·L·d²) touches every token and drives FLOP; adaLN adds ~223M acting only on the condition vector (~675M total).

DDPM (random walk, T=1000, Gaussian reverse only valid for tiny steps) → DDIM (deterministic, solves a probability-flow ODE, subsample the grid).

FID(N) = FID_inf + C / N^k     k = solver order

• FID_inf = 7.3 (irreducible). Second-order solver (Heun, DPM-Solver++ 2M) has O(1/N²) error.
• 20 second-order steps ≈ 100 Euler steps — 5x saving, no retrain. Past ~50 steps benefit is below the metric’s noise floor.
• Distillation: student reproduces teacher in few steps; costs recall not precision (narrower output). 4-step drops recall 0.62→0.48. Full sampler for delivered images, few-step for previews only.

Serving

• A diffusion step processes all 4,096 tokens in parallel → it is a prefill, x20. No autoregression, no KV cache.
• Compute-bound even at batch 1: 505 FLOP/byte vs H100 ridge point 296. Batching buys only ~1.15x and adds latency → scale with replicas, run batch 2 (the free CFG pair).
• Peak memory is the decoder, not the transformer (~8x): AE decode ~2.5 GB at 1024, ~10 GB at 2048, ~40 GB at 4K.
• Tiled decode above 1536: 512-px tiles, 64-px cosine-blended overlap, ~31% redundant compute. Seams where receptive field exceeds overlap.

Cost and metrics

• ~235 TFLOP/image ÷ 300 TFLOP/s effective → $0.00091/image. Provision for peak (40 GPUs), bill follows mean (18.1). Never multiply the 300 TFLOP/s effective rate by the 55% matmul MFU.
• FID breaks above ~300px: Inception-v3 takes 299 x 299, so 91.5% of a 1024 image is discarded. Report patch-FID at native resolution, plus recall, prompt adherence (VQA), and AE rFID.

Failure modes

• Garbled text: AE destroys glyphs (2-px strokes in an 8-px cell) + subword/CLIP encoders lack characters + area-weighted loss. Fix: c=16 (31%→9% OCR error, free), ByT5 char-aware encoder, glyph-weighted loss.
• Wrong hands: a hand is ~0.88% of the latent grid, so a wrong one costs <1% of loss; discrete “five fingers” + occlusion → regression hedges into six fingers. Fix on the data side (oversample, weight, pose keypoints).
• Duplicated subjects at unseen aspect ratios: position embeddings go out of distribution + attention entropy shifts + no “exactly one” constraint. Fix: aspect-ratio bucketing + RoPE.
• Reward hacking: aesthetic fine-tuning lifts aesthetic score but drops adherence (−3pp) and recall (−0.10). Gate on adherence, never ship the aesthetic number alone.

Numbers to remember

• Latent-space decision = 36,500x; every other lever < 10x.
• 8x per side = 64x tokens = 4,096x attention = 24,546x cheaper end to end.
• c=4 → c=16: free in tokens, cuts rFID 62% and text error 31%→9%.
• Above ~2K: hybrid — latent to 1024, then a refiner (a cascade whose base is latent).