How to design text-to-video generation

Turn a sentence into a 5 s clip: 120 frames (5 s · 24 fps) at 1,024 · 576 that read as one continuous shot. Video is not images plus one axis; it is a sequence-length problem, and the token arithmetic decides the product, the metering, and the queue before any modelling taste gets a vote.

Core vocabulary

• Diffusion model: start from noise, remove a little per sampling step; network predicts the noise eps. Latent diffusion runs the loop on a VAE-compressed stand-in (8x smaller per side); latent video diffusion also compresses in time.
• DiT (diffusion transformer): latent cut into 2·2 patches, each a token. d = token width, plus a block count. Borrowed image model: 2.6B params, d=2048, 40 blocks. Production video: 5B, d=3072, 40 blocks.
• Attention: self (tokens read each other) + cross (tokens read the text prompt, the only place the prompt enters).
• CFG: run each step with and without the prompt, amplify the difference. Doubles work: passes = steps · 2.
• Temporal consistency: a property of the joint distribution, not a filter. Cannot be bolted on.

The two formulas (per pass)

• Parameter term 2 · params · tokens — linear in tokens.
• Attention term 4 · tokens² · d · layers — quadratic in tokens.
• Tokens per frame at 1,024·576: VAE → 128·72 latent, 2·2 patch → 64·36 = 2,304.
• Price: H100 300 TFLOP/s @ $2.50/GPU-h (A100 150 TFLOP/s @ $2.00). Video model runs 50 steps → 100 passes.

Four designs, and the two moves

Consistency needs the joint distribution over 120 frames, not the marginal over one. A flipbook (per-frame) only ever learned the marginal, so shared seeds (diffusion is chaotic in the initial condition) and optical-flow warping (fails at occlusion boundaries) cannot fix it.

• Full 3D blow-up: 120x tokens = 120x parameter work but 120² = 14,400x attention. Attention goes 13% of one image → 95% of a clip. $3.43 is not a product.
• Move 1 — factorize attention (114x on the term): spatial (each token sees its own frame, 120 groups of 2,304) + temporal (same position across frames, 2,304 groups of 120). Both linear in what full 3D squared. Info travels diagonally, one hop/layer; the 40 blocks are load-bearing.
• Move 2 — spatiotemporal compression: causal 3D VAE encodes frame 0 alone then groups of 4 → 30 latent frames (8·8 space, 4x time). Parameter 4x, spatial 4x, temporal 16x (quadratic). Only lever that hits a quadratic term. Causal = a still image is a valid 1-frame video → joint image+video training (~100x more image data).
• The two savings do NOT multiply (114·4 is nonsense): factorization is a ratio of the attention term, compression a ratio of tokens. Full 3D → compressed is 19.3x on the whole clip, and the model got bigger.

Nyquist / aliasing (the compression cost)

• 24 fps / 4x = 6 Hz samples; Nyquist limit = 3 Hz. Motion above ~3 Hz aliases (wagon-wheel, strobing, smeared limbs). Not a training bug; a sampling-rate fact. Fix: high-motion tier at 2x compression (~2-4x cost).

Conditioning, chaining, data

• First-frame image (I2V) is worth more than any architecture change: appearance settled, all capacity goes to motion, gives eval a reference. Mature shape = text-to-image front end ($0.002 loop) → animate ($0.178). Other signals: last-frame, motion strength / fps (micro-conditioning), camera 6-DoF, control latents.
• Chaining drifts ~linearly (identity cos falls ~0.06-0.07/boundary). Keyframe-first-then-interpolate anchors every segment twice → bounded drift; correct above ~10 s.
• Data: video-text pairs barely exist (title/description/ASR each describe something other than the pixels). Synthetic recaptioning is the only caption source, and must describe change, not configuration. Pipeline: 100M videos → shot-split → keep 5-10 s (22%) → 5 filters multiply (0.0900) → 105M usable clips (~1 clip per source video). Prep ≈ 12% of a training run; NVDEC decode costs 3x captioning — the line nobody budgets.

Metrics: every automated one is blind or gameable

• FVD (I3D-based FID for video): identity swap moves it ~3% while humans collapse; needs resampling; ~15% variance across sample count.
• Per-frame CLIPScore: frames match prompt, never each other.
• Temporal consistency is maximized (1.000) by a static clip; warp error is minimized (0.000) by one; the ratio is 0/0. Gate on a motion floor and report the pair, returning None (not inf or 0) below it.
• Human eval is the only instrument that sees flicker/morphing/reversed physics: $405/model pair (776 comparisons · 3 raters · 25 s) against a $342k/day fleet. Rate on separate axes (adherence, motion, consistency, aesthetic).

Serving: async job system, not synchronous

• 257 GPU-s/clip → return a ticket, render elsewhere, notify. Everything cheap runs before the queue; only the render pool is expensive.
• Delivered cost: render is 85% ($0.1784 of $0.210 at 100% util); /0.60 util → $0.349/clip, ~$0.07/s. A clip costs 113 images; $9.99 buys 29 clips or 3,200 images → why video meters credits.
• Fleet: 1M clips/day → ~5,700 GPUs @ 60% util = $342k/day = $125M/yr. A capacity commitment, not autoscaling: cannot burst → must queue → must tier → draft tier is architectural.
• Draft tier: same 5B model, 512·288, 20 steps → 576 tokens/frame, 10.8x cheaper ($0.0165, ~4.6 s wall-clock). Users discard ~4 drafts/keeper; draft-then-render is 3.4x cheaper.
• Latency ~85 s: render 49 s (8 GPUs @ 65% scaling) + decode 28 s + transcode 8 s. Shard the memory-bound decode across 4 GPUs → 85 s to 64 s. Safety scan every frame (72 ms, 0.08%); sampling every 8th misses 3-frame violations.

Failure modes → mechanism → fix

Load-bearing assumptions (rest can be 2x wrong): frame count (compute, squared in attention), temporal compression ratio (compute and 3 Hz quality ceiling), utilization (pricing). Inflation (frozen image spatial layers) ships object permanence as a known defect.