How to design text-to-video generation

Generating a five-second clip the obvious way costs about two thousand times more than a single image: 1,483 PFLOP against 0.77 PFLOP. Done well, it comes down to about 100 times the compute and 113 times the delivered cost. Every one of those numbers is derived below.

Video is not images plus one axis. The compute arithmetic decides the product shape before any modelling choice is made: generation runs asynchronously instead of interactively, it is metered instead of unlimited, it is tiered instead of single-quality, and it is judged by people because every automated metric is blind to the failures that matter.

In this lesson, we’ll design a system that turns a sentence of text into a five-second video clip, and work through the arithmetic that decides what the product can be. By the end you’ll be able to derive why full attention over a clip is unaffordable, name the two moves that drop it into the same cost class as a flipbook, and defend the metering, tiering, and job queue that the compute arithmetic forces on the product. The text-to-image chapter built the same machinery for still images and the headshots chapter built the serving and cost discipline. Neither is required reading here; each idea they supply is restated before it is used, with a link to the longer version.

Problem framing

Fix exactly what goes in, what comes out, and the terms the rest of the chapter uses.

What goes in and what comes out

In: a sentence. The user types a text prompt, say “a woman in a navy blazer speaking to camera, static shot”, and may optionally attach one still image to use as the clip’s first frame.

Out: a video file. Five seconds long, at 24 frames per second (fps, the number of still images shown each second), each frame 1,024 pixels wide by 576 tall, with motion that reads as one continuous shot instead of a slideshow. Five seconds at 24 fps is 120 frames, and that one number drives everything below.

Three constraints bound the design: the clip must look like a single continuous shot; the cost of one clip has to fit inside what a consumer subscription can charge; and the wait has to be short enough that people come back.

The difficulty is that those 120 frames must each be individually plausible and mutually consistent, that the behaviour has to be trained from a corpus whose captions were never written by anyone, and that no automated metric can see the failure that actually loses users.

The machinery this chapter assumes

A short vocabulary carries the whole chapter. Each term is defined here once and used freely afterwards.

Diffusion model. Generates a picture by starting from pure random noise and removing a little at a time. Training teaches one network a single skill: shown a noisy picture and told how noisy it is, predict the noise that was added. There is no “draw a cat” instruction anywhere in the objective. Generation runs that one network repeatedly; each pass is a sampling step, and after a few dozen steps a picture is left. The quantity the network outputs is written eps (epsilon, the conventional symbol for a noise term): its guess at the noise currently sitting in the input.

Latents and the VAE. Running the denoising loop on full-resolution pixels is wasteful, so production systems run it on a latent, a compressed numeric stand-in for the picture, typically eight times smaller on each side. The compressor and decompressor are a variational autoencoder (VAE): an encoder mapping pixels down to the latent and a decoder mapping back, trained together so the round trip loses as little as possible. Running diffusion on the latent is latent diffusion; doing it on a latent compressed in time as well as space, several frames squeezed into one latent frame, is latent video diffusion, the architecture this chapter builds toward.

The denoiser is a transformer. The diffusion transformer (DiT) is the same architecture that powers language models, applied to image or video latents instead of words. It chops the latent into a grid of small squares called patches and treats each patch as one token, exactly as a word is a token for a language model. Everything expensive in this chapter is counted in tokens. Tokens exchange information through attention: self-attention lets tokens read each other, and cross-attention lets each token read the encoded text prompt, the only mechanism by which the prompt influences the picture at all. Two numbers describe such a network: d, the width of the vector per token, and the number of stacked blocks. We use d = 2048 with 40 blocks for the borrowed image model and d = 3072 with 40 blocks for the production video model.

Classifier-free guidance (CFG). The knob that makes the output follow the prompt. Each sampling step runs twice, once with the prompt and once with an empty prompt, and the difference is amplified and added back, pushing the sample toward what the prompt implies. CFG doubles the work of every step, so every pass count below is the step count times two: 28 steps is 56 passes, 50 steps is 100 passes.

Temporal consistency. The same thing in the world looks like the same thing from one frame to the next: the same face, the same shirt colour, the same car. It is not a smoothing filter or a post-processing step. As shown below, it is a property of which distribution you modelled, which is why it cannot be bolted on afterwards.

Units and prices. A FLOP is one floating-point operation (a multiply or an add); a TFLOP is a trillion and a PFLOP is a thousand TFLOP. “GPU-s” means GPU-seconds, one second of one chip. We price compute at two standing rates: an NVIDIA H100 sustains about 300 TFLOP/s and rents for $2.50 per GPU-hour; the older A100 sustains 150 TFLOP/s at $2.00. Every dollar figure is one of those rates applied to the FLOP count printed next to it.

Three shifts in perspective

The arithmetic forecloses most of the design space before any preference is stated, and the whole design reduces to two moves. Factorizing the attention means letting each token look at the other tokens in its own frame, and separately at the same spot in other frames, instead of at all tokens everywhere. Compressing the latent in time means having the VAE represent four consecutive frames with one latent frame. Both are derived in full below.

Three shifts in perspective separate someone who has done this from someone who has read about it. In each row the naive view is not wrong so much as pointed at the wrong quantity.

Everything rests on a few assumptions collected at the end: five seconds at 24 fps (120 frames) at 1,024 · 576, a consumer subscription around $9.99/month, and an H100 at 300 TFLOP/s and $2.50 per GPU-hour. Change the clip length or frame rate and every cost below moves; change the accelerator and only the dollars move.

Why video is not “images plus one axis”

The obvious architecture is unaffordable. Four steps show why, and show what the affordable one looks like. Each step is one design and one cost, and the four together take a five-second clip from $3.43 to $0.178 while making the model larger.

The flipbook baseline

Start with the cheapest thing that could work: run a still-image generator 120 times, once per frame. It is a flipbook because that is what it produces, a stack of individually fine pictures with no relationship to each other.

Take the image backbone from the text-to-image chapter, a diffusion transformer with 2.6 billion parameters, token width d = 2048, and 40 stacked blocks, and run it once per frame at 1024 · 576.

Three pieces of setup recur in every cost block in this chapter:

• Tokens per frame. At 1024 · 576 the VAE’s eightfold spatial compression produces a 128 · 72 latent. Cutting that into 2 · 2 patches leaves a 64 · 36 grid, so 64 · 36 = 2,304 tokens per frame.
• The parameter term, 2 · params · tokens. The work of pushing every token through every weight. Each weight is used once per token, at a cost of two operations (a multiply and an add), hence the 2.
• The attention term, 4 · tokens² · d · layers. The work of letting every token look at every other. Per layer, attention scores every token against every other and then mixes values by those scores: two passes of tokens² · d multiply-adds, which is 4 · tokens² · d operations, times the layer count.

latent     128 · 72,  patch 2   ->  64 · 36  =  2,304 tokens per frame

per pass   parameter  2 · 2.6e9 · 2,304                  =  11.98 TFLOP
           attention  4 · 2,304^2 · 2,048 · 40 layers    =   1.74 TFLOP
                                                            -----
                                                            13.72 TFLOP
28 steps · 2 (CFG)  =  56 passes                         =    768 TFLOP per frame
120 frames                                               = 92,200 TFLOP  =  92.2 PFLOP
at 300 TFLOP/s, $2.50/GPU-h                              =    307 GPU-s  =  $0.21

The split in the per-pass total is what to watch: attention is 1.74 of 13.72 TFLOP, about 13%, and that share is what changes in the next step.

This is exactly 120x an image, and the output is unusable. Each frame draws its own independent starting noise, so the model draws 120 independent samples from p(frame | prompt), the distribution over single frames it was trained to represent. Textures reshuffle, faces change, backgrounds shift. Nothing in the procedure couples the frames, because nothing in it could.

Two repairs suggest themselves. Neither works:

• Sharing a seed across frames does not fix it. A seed is the number that picks which random noise you start from, so sharing one makes all 120 frames start from the same noise. But the starting noise is not the sample. Diffusion is chaotic in the initial condition: a tiny difference at the start is amplified into a large one at the end, and two starting latents differing by 1% become visibly different images.

• Warping frame t into frame t+1 does not fix it either. Optical flow is an estimate, per pixel, of where that pixel moved to in the next frame; warping drags the pixels along those estimates. It cannot invent content entering the frame from outside, and it fails completely at occlusion boundaries, the edges where one object passes in front of another and pixels genuinely disappear and reappear, which is exactly where the eye looks.

The general statement is the foundation for everything that follows. The joint distribution over all 120 frames is the probability of a whole clip. The marginal distribution is the probability of a single frame on its own, with the other 119 forgotten. A per-frame model only ever learned the marginal, and a marginal contains no information about what any other frame looked like.

Temporal consistency is not a polish step. It is a property of the joint distribution, so you get it only by modelling that distribution and cannot add it afterward.

The blow-up

If consistency requires the joint distribution, model the joint distribution and see what it costs. The honest version, every token in the clip attending to every other, produces a number no consumer product can carry.

Nothing about the model changes: same 2.6B parameters, same d = 2048, same 40 blocks, same two formulas. The only change is the token count, from 2,304 to 276,480 (120 frames flattened into one sequence).

tokens     120 · 2,304                       =  276,480

parameter term    2 · 2.6e9 · 276,480        =  1,438 TFLOP    (120x one frame)
attention term    4 · 276,480^2 · 2,048 · 40 = 25,050 TFLOP    (14,400x one frame)
                                                ------
per pass                                       26,488 TFLOP
56 passes                                    = 1,483 PFLOP
                                             = 4,944 GPU-s  =  82 minutes  =  $3.43

The parameter term grows linearly in frames while the attention term grows quadratically. Every token still passes through every weight once, so 120x the tokens is 120x the parameter work. But every token now compares itself against every other, and the number of pairs among n things grows as n², so 120x the tokens is 120² = 14,400x the attention work.

That inversion is the whole point. Attention is 13% of the cost for one image and 95% for a 120-frame clip (25,050 of 26,488). Video architectures look different from image architectures for exactly this reason: you are no longer optimizing a model, you are optimizing a sequence length. And $3.43 a clip is not a product; a $9.99 subscription would buy fewer than three clips a month. The next two moves are the ways out.

Factorized attention

The first move splits one expensive attention operation into two cheap ones, and costs almost nothing in quality. Instead of attending over the whole volume, attend twice:

• Spatial attention: each token sees the other tokens in its own frame. That is 120 independent problems, each over 2,304 tokens.
• Temporal attention: each token sees the token at the same spatial position in every other frame. That is 2,304 independent problems, each over 120 tokens.

spatial   120 groups of 2,304 tokens
          120 · 4 · 2,304^2 · 2,048 · 40   =  208.8 TFLOP

temporal  2,304 groups of 120 tokens
          2,304 · 4 · 120^2 · 2,048 · 40   =   10.9 TFLOP
                                              ------
                                              219.7 TFLOP   vs 25,050 full 3D
                                              -> 114x cheaper
per pass  1,438 (parameter) + 220            = 1,658 TFLOP
56 passes                                    =  92.8 PFLOP  =  309 GPU-s  =  $0.21

Spatial attention now counts pairs within 2,304 tokens 120 times over, which is 120 · 2,304² pairs instead of (120 · 2,304)². Temporal attention counts pairs within 120 tokens 2,304 times over. Both are linear in the thing the full version squared.

Factorized attention makes a temporally-joint model cost the same as the flipbook. Consistency is not something you pay for once you factorize, because the term you were afraid of was never the model, it was the T² (with T the number of frames) that full attention insists on computing.

What you give up is exact. In one layer, a token can only reach another token that shares a frame or shares a spatial position with it; information travels diagonally through the volume, one hop per layer. To get from position p in frame 0 to position q in frame 119 takes a temporal hop from (p, 0) to (p, 119) then a spatial hop to (q, 119): two hops, two layers. The stack has 40 blocks alternating spatial and temporal attention, twenty times the routing depth any pair actually needs, which is why the approximation holds in practice. A 12-block model factorized this way would genuinely lose long-range coherence, so the depth is load-bearing.

Spatiotemporal latent compression

The second move shrinks the sequence itself instead of the operation over it, and it is the only lever that touches a quadratic term. Compress the latent in time as well as space. The compressor is a causal 3D VAE: 3D because it compresses across height, width and time, and causal because it looks only backwards, encoding frame 0 by itself and then successive groups of four. The ratios are 8 · 8 spatially (unchanged from the image-synthesis chapter) and 4x temporally.

120 frames  ->  1 + (120 - 1)/4  ~=  30 latent frames
tokens      30 · 2,304            =  69,120      (4x fewer)

The first line is the causal encoding: one latent frame for frame 0 alone, plus one per group of four after it, which is 1 + 119/4 = 30.75. Every number below uses the round 30 (the same as 120/4); the 2.5% difference moves no conclusion. The frame count itself did not drop: the clip is still 120 output frames, now represented by 30 latent frames that the decoder expands back to 120 at the end.

Production video models are larger than the borrowed image backbone, so the compressed latent is priced against a realistic one, 5 billion parameters, d = 3072, 40 blocks, with 50 sampling steps instead of 28. Two things get worse (more parameters, more steps) and one gets much better (4x fewer tokens):

parameter   2 · 5e9 · 69,120                     =  691 TFLOP
spatial     30 · 4 · 2,304^2 · 3,072 · 40        =   78 TFLOP
temporal    2,304 · 4 · 30^2 · 3,072 · 40        =    1 TFLOP
                                                    ----
per pass                                            770 TFLOP
50 steps · 2 (CFG) = 100 passes                  = 77.0 PFLOP
at 300 TFLOP/s                                   =  257 GPU-s  =  $0.178

At fixed model size the saving decomposes cleanly:

• Parameter term: 4x. Linear in tokens, and there are 4x fewer.
• Spatial attention: 4x. Each group is still 2,304 tokens, but 30 groups instead of 120.
• Temporal attention: 16x. Each group went from 120 to 30 tokens, and attention is quadratic: (120/30)² = 16.

Temporal compression is the only lever that hits a quadratic term, which is why it is worth more than any parameter reduction.

But the block above is not at fixed model size: d is 3,072 instead of 2,048 and the parameter count is 5B instead of 2.6B, so the per-pass cost falls from 1,658 to 770 TFLOP, a factor of 2.15, not 4. The compression bought roughly 4x and the model spent about half of it. Keep those two statements apart; conflating them is where the arithmetic in this section usually goes wrong.

The escalation, in one view

All four designs side by side. The last column is the one that decides everything, since it is the only one that says whether the output is a video at all.

flowchart LR
    A["Per-frame<br/>$0.21<br/>NO consistency"] -->|"model the joint<br/>distribution"| B["Full 3D attention<br/>$3.43<br/>T^2 term is 95%"]
    B -->|"factorize:<br/>spatial + temporal<br/>114x on attention"| C["Factorized<br/>$0.21<br/>consistency now free"]
    C -->|"4x temporal VAE<br/>bigger 5B model"| D["Compressed latent<br/>5B model<br/>$0.178"]

    style A fill:#9d0208,color:#fff
    style B fill:#bc6c25,color:#fff
    style C fill:#1d3557,color:#fff
    style D fill:#2d6a4f,color:#fff

From full 3D attention to the compressed latent is 19.3x on the clip, 1,483 PFLOP to 77.0, and it is entirely sequence-length engineering. The model got bigger along the way and the clip got cheaper.

One trap: the two savings do not multiply. Factorization is 114x on the attention term; temporal compression is 4x on the token count. Those are ratios of different things, so 114 · 4 = 456 describes nothing. Pick one quantity and chase it through both moves:

• Chase the attention term (holding d = 2048 so model size does not contaminate it): spatial over 30 groups is 52.2 TFLOP, temporal over 30-token groups is 0.7, so 52.9 against 25,050. That is 474x.
• Chase the whole clip, model growth and all: 1,483 PFLOP to 77.0. That is 19.3x.

The gap between 474 and 19.3 is everything that is not attention: factorization never touches the parameter term, the 5B model makes it larger, and the sampler runs 100 passes instead of 56.

What temporal compression costs you

Compression in time is lossy in time, and the loss has a name. There is a speed of motion above which the system physically cannot represent what happened, computable from the compression ratio alone.

The Nyquist limit is a signal-processing result: if you record at f samples per second, the fastest repeating motion you can faithfully capture is f/2 cycles per second, because you need at least two samples per cycle to know a cycle happened. Faster motion does not vanish; it comes back disguised as some slower motion that was never there, which is called aliasing.

output frame rate                              24 fps
temporal compression                            4x
independent temporal samples in the latent      6 Hz     (24 / 4)     [Hz = per second]
Nyquist limit of the latent                     3 Hz     (6 / 2)
a wheel spinning at 3 revolutions/second        aliases

Motion faster than about 3 Hz cannot be represented in the latent and comes back as aliasing: the wagon-wheel effect (a spinning wheel appearing to turn slowly backwards), strobing on fast pans, smeared limbs on runners. This is not a model defect that more training fixes; it is a sampling-rate consequence of the compression ratio, and it would remain even with a perfect model. Its fix names itself: a high-motion tier at 2x temporal compression instead of 4x, which doubles the token count and costs roughly 2-4x more.

The choice of a causal VAE looks like a detail and is not. Encoding frame 0 by itself and then groups of four means a single still image is already a valid one-frame video, with no special casing. That lets you train one model on images and video jointly, which matters enormously: there is roughly a hundred times more usable image data than video data (see the data section), and a video-only model throws all of it away.

Architectures

One block of the production denoiser first, then the alternatives people reach for and what each gets wrong. The input is a noisy video latent, 30 latent frames of 2,304 tokens each, and four stages act on it in order:

1. Spatial self-attention. Tokens in the same frame read each other. 30 groups of 2,304.
2. Cross-attention to the text. The same mechanism the text-to-image chapter uses, and the only place in the network where the prompt enters.
3. Temporal self-attention. The token at a given position reads that position in all other frames. 2,304 groups of 30.
4. Feed-forward. A small two-layer network applied to each token independently. Most parameters live here, which is why the parameter term is 691 of the 770 TFLOP per pass.

That sequence repeats for 40 blocks; the last emits the eps prediction, the noise to subtract at this step. The loop-back arrow below is the 40-block repeat, not a recurrence in time.

flowchart TD
    IN["Noisy video latent<br/>30 latent frames · 2,304 tokens each"] --> SP["Spatial self-attention<br/>within each frame<br/>30 groups of 2,304"]
    SP --> XA["Cross-attention to text<br/>same as text-to-image"]
    XA --> TP["Temporal self-attention<br/>across frames at each position<br/>2,304 groups of 30"]
    TP --> FF["Feed-forward"]
    FF -->|"· 40 blocks"| IN
    FF --> OUT["eps prediction"]

    style SP fill:#1d3557,color:#fff
    style TP fill:#bc6c25,color:#fff
    style XA fill:#40916c,color:#fff
    style OUT fill:#2d6a4f,color:#fff

Six approaches, priced against each other. Temporal receptive field is how many frames away an output position can draw information from. Relative cost is normalized so the production answer (the $0.178 clip) is 1.0; every row near 1.0 is in that cost class, and only full 3D attention is not.

A 3D convolution slides a small fixed-size box (the kernel) over height, width and time; because the box is small, each output sees only three to five frames either side (“kernel-limited”). Inflation takes a trained image model, copies its 2D layers into a 3D shape that accepts a stack of frames, and inserts new temporal layers between them.

Inflation is the tempting shortcut and the one to argue against carefully. Initializing from a trained image model and freezing the spatial layers gets visual quality for free and trains in a fraction of the compute. The problem is structural: the spatial layers learned the marginal distribution of single frames, and a marginal has no notion of before or after. Long-range coherence, object permanence (a thing that goes behind a pole is the same thing when it comes out), and physical plausibility all have to be carried by the thin temporal stack you bolted on. Object permanence is the failure inflation cannot fix; unfreezing everything for a joint training run is what buys it. Most 3D convolutions in a production model live in the VAE instead of the denoiser, where their locality is a virtue: a decoder needs local smoothness, not long-range reasoning about four seconds ago.

Conditioning and motion control

Conditioning means every input other than the noise that steers generation. Text conditioning is the cross-attention above, unchanged: the prompt is encoded once and every token reads from it at every block. The video-specific signals are where the quality comes from, roughly in order of how much each is worth.

Four terms: I2V is image-to-video, where the user supplies a picture and the system animates it. Micro-conditioning feeds the model a single number alongside the prompt during training (motion strength, frame rate, an aesthetic score); at generation time you dial that number for a predictable change without retraining. 6-DoF is the six numbers that fully describe where a camera is and which way it points, three for position and three for rotation. Control latents are per-frame encoded versions of a driving signal (a depth map, a stick-figure pose, an edge drawing) added into early blocks so the output follows that motion exactly.

First-frame conditioning is worth more than any architecture change. Text-to-video asks the model to invent what things look like and how they move at once, from a prompt that under-specifies both. Given the first frame, appearance is settled, the identity anchor is exact, and all model capacity goes to motion. It also turns the hardest evaluation problem, “is this the same person throughout”, into one with a reference to compare against.

That is why the mature product shape is image-to-video with a text-to-image front end: generate the first frame with the cheap, fast, well-controlled image model, let the user approve it, then animate. The user gets a $0.002 iteration loop on appearance instead of a $0.178 one.

Chaining, and why drift is linear

Five seconds is not a film. Chaining generates five seconds, then the next five conditioned on the last frame of the first, and so on. It accumulates error. Two quantities tracked at each boundary: identity cosine (cosine similarity between face embeddings, where 1.00 is identical), and colour shift dE (the standard perceptual colour-difference unit, where about 2 is the threshold a person can notice).

segment 1 -> 2   identity cos vs original 0.94, color shift dE 2.1
segment 2 -> 3                              0.88,                4.4
segment 3 -> 4                              0.81,                6.9
segment 4 -> 5                              0.74,                9.8

Each segment’s last frame is a generated frame, slightly off-distribution, and it anchors the next segment. Errors compound roughly linearly: identity falls about 0.06 to 0.07 per boundary instead of collapsing all at once.

Keyframe-first-then-interpolate bounds the drift. A keyframe is a frame at a fixed time committed to in advance; frame interpolation generates the frames between two known frames, told what the interval starts and ends with. Two passes:

1. Generate keyframes at 0 s, 5 s, 10 s and 15 s in a single pass, so they are mutually consistent.
2. Interpolate each interval conditioned on both of its ends.

Every segment is now anchored twice, so error cannot accumulate past one interval: the far end of each interval is a frame the model was given, not one it drifted into. It costs one extra pass over the keyframes and is the correct architecture for anything over about 10 seconds.

Data

Video training data is a fundamentally different problem from image training data. The pipeline below turns 100 million raw videos into 105 million usable clips, and pricing it ends on a line item almost nobody budgets for.

Video-text pairs barely exist

A generative model needs pairs: a piece of media and a description of it. For video, the description side essentially does not exist, and the consequence is not “captions are noisy” but “you have to manufacture all of them.”

Images come with alt-text, the short written description HTML lets a page author attach to an image for screen readers. The quality is poor but it is present, in billions of instances, and the image chapter found a filtered 10% slice worth keeping. Video has no equivalent, only near-misses that each describe something other than the pictures:

Synthetic recaptioning, running a model over the media and having it write the description you wish the media had come with, is the fix. In images it was a 12%-of-budget improvement lever. In video it is not a lever at all; it is the only source of captions that exist, because there are no originals to keep 10% of.

And the captions have to carry more. An image caption describes a configuration (what is in the picture and where); a video caption must describe a change (what moves, in which direction, and how the camera behaves). The captioner is a vision-language model (VLM), which takes images and text together and produces text. A VLM shown 8 frames sampled from a clip is not good at describing motion, because eight stills are a poor description of five seconds. That failure is not neutral: captions describing only static content actively teach the model that the prompt does not constrain motion, the same “the loss rewards ignoring an uninformative caption” argument from the text-to-image chapter, sharper here because there is no clean caption anywhere in the corpus to dilute it.

The pipeline, with yields

Three terms first. Shot-boundary detection finds the cuts and splits a four-minute upload into its individual continuous takes; it runs first because a training clip containing a cut teaches the model that scenes teleport. Near-duplicate dedup removes clips visually almost the same as one already kept (on the open web, mostly the same stock footage across thousands of uploads). UI density refers to on-screen interface elements (menus, cursors, game overlays) a model should not learn to draw.

source                       100M videos, mean 4 min  =  2.4e10 seconds
shot-boundary detection      mean shot 4.5 s          ->  5.33e9 shots
keep shots of 5-10 s                          22%     ->  1.17e9 clips

filters (multiplicative)
  motion band: reject static screen-recordings AND excessive shake   46%
  resolution >= 720p and aesthetic threshold                         35%
  low burned-in text / subtitle / UI density                         72%
  near-duplicate and stock-footage dedup                             80%
  safety removal                                                     97%
                        combined  0.0900       ->  105M usable clips

100M videos at 240 s each is 2.4e10 seconds; divided by a 4.5 s mean shot, 5.33e9 shots; keeping the 22% that run 5 to 10 s leaves 1.17e9 clips. The five filter rates then multiply (a clip has to pass all of them): 0.46 · 0.35 · 0.72 · 0.80 · 0.97 = 0.0900, and 1.17e9 · 0.0900 = 105M.

flowchart TD
    A["100M source videos<br/>2.4e10 seconds"] --> B["Shot-boundary split<br/>5.33e9 shots"]
    B --> C["Keep 5-10 s shots (22%)<br/>1.17e9 candidate clips"]
    C --> D["Five filters combined (9%)<br/>motion, resolution, text density,<br/>dedup, safety"]
    D --> E["105M usable clips"]

Roughly 2% of shots survive, and one source video yields about one usable clip. That is the answer to “just scrape more video”: the corpus is gated not on collection but on the fact that most footage is static, shaky, subtitled, duplicated, or under-resolution, and none of those improve by scraping harder.

What the pipeline costs, and the surprise

Data preparation for video has a cost structure the image case does not. NVDEC is the dedicated video-decoding circuit on an NVIDIA GPU, which turns compressed video back into frames far faster than the general-purpose cores could.

decoding the corpus
  2.4e10 s of video at ~200x realtime on GPU NVDEC
  2.4e10 / 200                        =  1.2e8 GPU-s  =  33,300 GPU-h

captioning 105M clips with a 7B VLM
  8 frames · 256 image tokens + prompt =  2,300 in, 150 out
  (2,300 + 150) · 2 · 7e9              =  34.3 TFLOP per clip
  · 3 (decode memory-bound) / 300e12   =   0.34 GPU-s per clip
  105e6 · 0.34                         =  10,000 GPU-h
                                          ------
                                          43,300 GPU-h  =  ~$108,000

The captioning line is the standard way to price any transformer inference. Count the tokens (8 frames at 256 image tokens each is 2,048, plus prompt, so ~2,300 in and 150 out). Price a pass at two operations per parameter per token: (2,300 + 150) · 2 · 7e9 = 34.3 TFLOP. Then apply a · 3 memory-bound penalty: generating the 150 output tokens one at a time is memory-bound, the chip mostly waiting for weights to arrive instead of doing arithmetic, so you get about a third of advertised throughput. (This “decode” is a different hardware path from decoding the video corpus above; same word.) Both halves at $2.50 per GPU-hour give 43,300 · 2.50 ≈ $108,000.

Decoding the corpus costs three times what captioning it costs. That is the line item nobody budgets, and it is unique to video: in an image pipeline decode is free and the captioning model dominates. Against a training run of roughly 370,000 GPU-hours (~$925,000), data preparation is about 12% of the budget, the same shape as the image chapter’s recaptioning ratio arrived at by a completely different route.

Metrics

Every automated video-quality number is either blind to the failures that matter or maximized by a model that does nothing. The one instrument that works is human evaluation, whose cost objection does not survive being priced.

Why every automated metric is weak here

FVD (Fréchet Video Distance) is the video version of FID (Fréchet Inception Distance). FID runs a batch of real images and a batch of generated images through a fixed pretrained network, summarizes each batch by the mean and covariance of the resulting feature vectors, and reports the distance between the two summaries; lower means the generated batch looks statistically more like the real one. FVD does the same using I3D (the Inflated 3D ConvNet, a network trained to classify what action is happening in a clip) as the feature extractor. It inherits all of FID’s problems (two axes, never one number) and adds three:

1. The feature extractor was trained to classify actions. A clip where the subject’s shirt changes colour at frame 60 is still, to I3D, “a person walking.” A deliberate identity swap mid-clip moves FVD by about 3% while human preference collapses.
2. It needs fixed-length, fixed-rate windows. Your clip has to be resampled to what I3D expects, so you measure a resampled proxy, not your output.
3. Its variance is brutal. The same model scored on 512 versus 2,048 samples can differ by about 15%, larger than most improvements you will ship.

Per-frame CLIPScore uses CLIP (Contrastive Language-Image Pretraining, a pair of encoders trained so a picture and its caption land close together), scoring the cosine similarity between a frame’s vector and the prompt’s. Applied per frame it tells you the frames match the prompt; it says nothing about whether they match each other, because it never compares two frames.

Temporal consistency has a degenerate maximum: its best possible score is achieved by a worthless output.

temporal consistency  =  mean over t of  cos( CLIP_img(f_t), CLIP_img(f_{t+1}) )

a completely static video:   every consecutive pair identical
                             ->  consistency = 1.000, the maximum

The metric meant to measure temporal quality is maximized by the model failing to move. The obvious counterweight has the mirror problem: warp error drags frame t’s pixels along the estimated flow into frame t+1 and measures the residual, but on a static video the flow is zero, nothing moves, the warped frame is frame t+1, and the error is 0.000, the best score again. Either metric alone is gamed by a still image, and a model that under-moves is a real and common failure.

The ratio people reach for next, warp error per unit of motion, does not rescue this: its denominator is motion, which is gameable, and on a static clip the ratio is 0/0. So the scorecard must gate on motion before dividing by it, and report undefined, not a number, because inf and 0.0 are both lies about a clip that did not move. The code below does exactly that. On the static clip consistency is a perfect 1.000 (the degenerate maximum), under_motion is True, and warp_error_per_motion is None:

MOTION_FLOOR = 0.25          # px/frame. Below this the clip is not moving.

def temporal_scorecard(frames, embed, flow):
    """Consistency is only meaningful jointly with motion. Never report one.

    A static clip scores consistency 1.000 and motion 0.000 -- the pair
    identifies it instantly; either number alone hides it.
    """
    if len(frames) < 2:
        raise ValueError("a scorecard over fewer than two frames measures nothing")

    embs = [embed(f) for f in frames]
    n = len(frames) - 1
    consistency = sum(sum(x * y for x, y in zip(a, b))
                      for a, b in zip(embs, embs[1:])) / n

    motion = warp_err = 0.0
    for a, b in zip(frames, frames[1:]):
        magnitude, warped = flow(a, b)          # flow gives magnitude + a warped into b
        motion += sum(abs(m) for m in magnitude) / len(magnitude)
        warp_err += sum(abs(x - y) for x, y in zip(b, warped)) / len(b)
    motion, warp_err = motion / n, warp_err / n

    moving = motion >= MOTION_FLOOR
    return {
        "consistency": consistency,        # gameable alone: static -> 1.000
        "motion": motion,                  # gameable alone: chaos -> huge
        "warp_error": warp_err,            # gameable alone: static -> 0.000
        "under_motion": not moving,        # the finding a ratio would hide
        # 0/0 without real motion: inf would claim a residual this clip
        # does not have; None says "no motion", a different finding.
        "warp_error_per_motion": warp_err / motion if moving else None,
    }

Report the pair. The ratio is a summary of the pair, and it only exists above the motion floor. This is the same reason the image chapter refused to report a single quality number: two axes with different best-case directions cannot be collapsed into one without silently deciding which one you gave up. Returning None also forces the aggregation step to be honest, so the report reads “12% of clips were under-motion; the remaining 88% averaged 0.31” instead of a single misleading mean.

Human evaluation, and why it dominates here

Every failure that actually loses users, flicker, morphing identity, an object that changes across an occlusion, a physically reversed trajectory, is invisible to every metric above. Human evaluation is the instrument, and the cost objection does not survive being priced. Three terms set the study size: alpha is the false-positive rate you accept (0.05 means a 5% chance of declaring a winner when the models are identical); power is the chance of detecting a real difference when one exists (0.80 means four times in five); and “detect 55/45” means distinguishing a model that wins 55% of head-to-head comparisons from a coin flip.

per pairwise judgment    watch 2 clips (5 s each) + decide  ~=  25 s
                         (an image pair is ~4 s -- video is ~6x)

detect 55/45 at alpha 0.05, power 0.80          ->  776 comparisons
   · 3 raters for agreement                      =  2,328 judgments
   · 25 s                                        =  16.2 rater-hours
   at $25/hour                                   =  $405 per model pair

The power calculation is the same one used for image models; only the seconds-per-judgment changed, because a rater watches ten seconds of video instead of glancing at two pictures.

$405 against a serving fleet that costs $342,000 a day (derived in the serving section). Human evaluation is one tenth of one percent of a day of serving, and it is the only instrument that sees the failures. Structure the rating on separate axes, because they have different best directions and cannot be traded off: prompt adherence, motion quality, temporal consistency, and aesthetic. A single “which is better” question collapses all four and you can no longer tell which one regressed.

Online metrics

Offline numbers gate a release; online numbers tell you whether the release was any good. The bottom three exist only because generation is asynchronous.

The regenerate-rate split is a free diagnostic most teams do not log. Regenerating after editing the prompt says the model did not do what the words said, a prompt-adherence problem; regenerating without touching the prompt says this sample was bad, a sampling or quality problem. Same button, two different bug reports, separated by one boolean in the log.

Serving

At 257 GPU-seconds per clip this is not a synchronous product (one where the user waits with the connection open); pretending otherwise is the most common design error here. The architecture is a job system: the request returns a ticket, the work happens elsewhere, and the user is notified. Everything cheap happens before the queue, and the only expensive box is the render pool.

flowchart TD
    R([Prompt + optional<br/>first frame]) --> MOD{Text + image<br/>moderation}
    MOD -->|block| REJ([Refuse])
    MOD -->|pass| CR{Credit check<br/>+ tier}
    CR -->|none| PAY([Upsell])
    CR -->|ok| DRAFT["Draft tier<br/>512 · 288 · 20 steps<br/>24 GPU-s · $0.016"]
    DRAFT --> PRE([Preview to user])
    PRE -->|discard| R
    PRE -->|approve| Q[[Priority job queue<br/>free · paid · pro]]
    Q --> REN["Render pool<br/>5B DiT · 50 steps · CFG<br/>8 GPUs sequence-parallel<br/>257 GPU-s"]
    REN --> DEC["Causal 3D VAE decode<br/>30 latent -> 120 frames"]
    DEC --> SCAN{"Safety scan<br/>EVERY frame<br/>72 ms total"}
    SCAN -->|flag| HOLD[Hold for review]
    SCAN -->|clear| WM[Per-frame watermark<br/>+ C2PA manifest]
    WM --> TR[Transcode ladder<br/>1080p / 720p / 480p]
    TR --> CDN([CDN + notify])

    style DRAFT fill:#40916c,color:#fff
    style REN fill:#1d3557,color:#fff
    style SCAN fill:#2d6a4f,color:#fff
    style REJ fill:#9d0208,color:#fff

• Text + image moderation. Both inputs checked against policy before any compute is spent; the image half exists because a user can smuggle in through a picture what the prompt filter would catch.
• Credit check + tier. Looks up whether this user’s plan and remaining credits allow a render, and their service class. No credits means an upsell, not a queue position.
• Draft tier. A cheap low-resolution preview to approve or discard: ~24 GPU-seconds and $0.016 against the render’s 257, derived below.
• Priority job queue. One queue whose ordering respects three service classes (free, paid, pro), so a paying user does not wait behind a thousand free ones.
• Render pool. The real 5B model at 50 steps with guidance, split sequence-parallel across 8 GPUs: the token sequence is divided among the chips, which exchange the pieces each needs.
• Causal 3D VAE decode. Expands 30 latent frames back into 120 pixel frames.
• Safety scan. Classifies every frame; anything flagged goes to a human instead of being silently deleted or published.
• Watermark + C2PA manifest. C2PA (Coalition for Content Provenance and Authenticity) is an industry standard that attaches signed metadata recording that this file was machine-generated and by what.
• Transcode ladder. Re-encodes at 1080p, 720p and 480p so players on different connections each get a streamable version.
• CDN + notify. The files reach edge servers close to viewers, and the user is told the job is done.

Dollars per second of video

Everything the product can be follows from one number, so derive it end to end. Utilization is the fraction of the time reserved GPUs are actually busy; nobody runs at 100%, so the honest delivered cost divides by a realistic figure.

render        257 GPU-s                                  $0.1784
3D VAE decode ~15% of render                             $0.0268
safety scan   120 frames · 0.6 ms                        $0.0001
watermark + transcode  (CPU)                             $0.0040
egress        5 s at 8 Mbps = 5 MB, $0.09/GB             $0.0005
                                                         -------
per clip at 100% utilization                             $0.210
at 60% fleet utilization                                 $0.349

per second of generated video                            $0.0699
per second at 100% utilization                           $0.0419

The render is $0.1784 of a $0.210 clip, so 85% of the cost is the denoiser and everything else is rounding. Dividing by 60% utilization is not a modelling choice but the difference between what you compute and what you pay for, and it is what moves the delivered clip from $0.210 to $0.349.

Two comparisons, both priced on the same basis (one 1024 · 576 image by the text-to-image chapter’s method, one clip, both all-in):

one 1024 · 576 image, 56 passes · 13.72 TFLOP  =    768 TFLOP
  at 300 TFLOP/s, all-in                       =  $0.00185 at 100% util
one 5-second clip                              =  $0.210   at 100% util
                                                  -> a clip costs 113 images

$9.99/month buys, at 60% utilization:
   images   9.99 / 0.00308  =  3,200 images    (0.00185 / 0.60)
   clips    9.99 / 0.349    =     29 clips

Both sides are delivered costs, not compute costs, which is the only way the ratio means anything. The same subscription price buys 3,200 images or 29 clips. That ratio, not any modelling decision, is why video products meter credits and image products do not.

Fleet sizing

Unit cost becomes capacity by multiplying by demand, and at this scale capacity stops being an engineering decision and becomes a purchasing one.

1M clips/day  =  11.6 clips/s              (1e6 / 86,400 s in a day)
each          =  257 render + ~38 decode  =  295 GPU-s

at 100% utilization   11.6 · 295           =  3,420 GPUs
at 60% utilization                         =  5,700 GPUs
at $60/GPU-day                             =  $342,000/day  =  $125M/year

$60 per GPU-day is the standing $2.50 per GPU-hour times 24. Nearly six thousand GPUs is a capacity commitment, not an autoscaling group: hardware at that quantity is bought or reserved months ahead, not summoned when traffic arrives. Everything about the product follows from that one fact: you cannot burst, so you must queue; you cannot queue indefinitely without losing people, so you must tier; and you cannot tier without something cheap in the fast tier, so the draft tier is architectural, not a nicety.

Why the draft tier is not optional

The draft is the same model at lower resolution and fewer steps. At 512 · 288 the VAE gives a 64 · 36 latent, and 2 · 2 patches make a 32 · 18 grid, so 576 tokens per latent frame, a quarter of the render’s 2,304, on the same 30 latent frames.

draft: 512 · 288, 20 steps, CFG
  tokens        30 · 576                    =  17,280   (4x fewer)
  parameter     2 · 5e9 · 17,280            =  173 TFLOP
  spatial       30 · 4 · 576^2 · 3,072 · 40 =    4.9 TFLOP
  temporal      576 · 4 · 30^2 · 3,072 · 40 =    0.25 TFLOP
  per pass                                  =  178 TFLOP
  40 passes                                 =  7.1 PFLOP  =  23.7 GPU-s  =  $0.0165

  770 / 178 = 4.3x per pass · 100 / 40 = 2.5x on passes
                                            -> 10.8x cheaper than the render

That 10.8x is the honest like-for-like number: render compute against draft compute. The behaviour that justifies the tier is measurable: users discard about four drafts before approving one, so a completed clip costs five drafts plus one render.

draft-then-render   5 · $0.0165 + $0.178  =  $0.261
render every time   5 · $0.178            =  $0.892
                                             -> 3.4x cheaper

And the latency story is better than the cost story: a draft is 23.7 GPU-seconds, about 4.6 seconds of wall-clock time split across 8 GPUs, so the iteration loop feels interactive even though the product as a whole is asynchronous.

Latency

Wall-clock time is not the same shape as cost: the cheapest stage of the pipeline is a third of the wait.

render     257 GPU-s, sequence-parallel over 8 GPUs at 65% scaling
           257 / (8 · 0.65)                     =  49 s
decode     3D VAE, poorly parallel              =  ~28 s
transcode + upload                              =  ~8 s
                                                   ----
                                                   ~85 s, plus queue

The 65% scaling on the render line is the efficiency loss from splitting one job across 8 GPUs, which have to exchange data, so 8 chips do about 5.2 chips’ worth of useful work.

The decode is a third of the wall clock and gets no attention in most designs, and it is the cheapest thing to fix. Decoding is memory-bandwidth-bound (limited by how fast it moves data, not by arithmetic) and splits cleanly across groups of frames, each decoded independently. So shard it, handing each of 4 GPUs a quarter of the frames, for close to a 4x speedup: ~28 seconds becomes ~7, and the whole pipeline drops from 85 seconds to 64.

One safety consequence follows directly: scan every frame. Putting 120 frames through a CLIP-class classifier at 0.6 ms each is 72 ms against an 85-second pipeline, 0.08% of it. Sampling every 8th frame saves nothing measurable and can miss a three-frame violation, which is more than enough to be screenshotted and shared.

Failure modes

Four failures lose users, each traceable from a symptom to the specific component that produced it; the rest get a detector and a guard apiece. None is fixed by “train longer”; every one has a named mechanism and a matching intervention.

Flicker

Flicker is a sudden one-frame change in something that should have stayed constant. The trace below records the average colour of a jacket across nine frames as a hex colour code (#RRGGBB, each pair the red, green and blue intensity from 00 to FF). To compare by brightness, use luma, the weighted brightness summary 0.2126·R + 0.7152·G + 0.0722·B.

PROMPT   "a woman in a navy blazer speaking to camera, static shot"

blazer mean colour, frames 36-44
  36 #24365E   37 #24375F   38 #233560   39 #253863
  40 #2E4576   <- luma 53 -> 68, a 27% jump in one frame
  41 #24365D   42 #24365E   43 #233460   44 #24365F

frame 40 is the first frame of a new 4-frame VAE group

Frame 40 is 4 · 10, so it opens the eleventh decoder group. Temporal attention operates on 30 latent frames while the decoder generates 120 output frames in groups of 4. High-frequency detail the latent does not constrain (fabric texture, fine highlights, hair) is synthesized by the decoder per group, and group boundaries are exactly where it can discontinuously change its mind. Three fixes, in increasing effort:

1. Overlap the decoder’s temporal windows and blend where they meet, so a group boundary is never a hard seam. Pure serving-side change.
2. Add a temporal-consistency term to the VAE’s training loss, penalizing the decoder for changing its mind. Requires retraining the VAE.
3. Lower the guidance scale, because CFG amplifies exactly this high-frequency component. Free, but it costs prompt adherence.

Object permanence

The failure users describe as “it turned into a different car”, and the one that most cleanly separates architectures.

PROMPT   "a red car drives past a lamppost, camera fixed"

frames 0-40    red sedan moving left to right, four visible wheels
frames 41-55   fully occluded by the lamppost
frames 56-119  a red HATCHBACK, different wheels, different roofline

Temporal attention gives frame 56 access to frame 40, which is not the same as memory. Attention is a soft read over a distributed representation, not a slot storing “this specific car”, and nothing in the denoising objective rewards maintaining object identity across an occlusion; it rewards predicting plausible noise, and a hatchback is plausible. This is the failure inflation-based architectures cannot fix, because fixing it needs the spatial layers themselves to have learned that frames belong to sequences. Longer joint training and larger temporal receptive fields move it; bolting a temporal stack onto a frozen image model does not.

Physics and the arrow of time

The failure that looks like a bug in reality itself, caused by the data, not the architecture.

PROMPT   "a glass falls off a table and shatters on the floor"

3 of 8 samples: the glass falls, contacts, shatters -- and over the next
8 frames the shards travel BACK UP and reassemble

The denoising objective is symmetric in time: “is the noise prediction right for this frame” has the same answer whether the clip runs forwards or backwards, and nothing penalizes a trajectory that is locally plausible but globally reversed. The corpus makes it worse, because reversed and looped clips are a popular editing effect, so real footage of shards flying upward exists in quantity and teaches the model that reversal is normal. Both fixes are data-side and cheap: detect and drop reversed and looped clips at ingest, and add an explicit time-direction conditioning signal so the model has a handle it can be right or wrong about.

Identity morphing

A face slowly becoming a different face, with a signature you can read straight off the measurement. The numbers are ArcFace cosine similarity (ArcFace maps a face to a vector so two pictures of the same person point in nearly the same direction; the number is the cosine between frame 0’s vector and frame t’s). The headshots chapter established roughly 0.65 as the threshold below which two images stop reading as the same person.

PROMPT   "a woman in a red coat walks toward camera, 5 seconds"

ArcFace cosine, frame 0 vs frame t
  t=12   0.91
  t=36   0.78
  t=72   0.61
  t=119  0.44      <- below the same-person anchor (~0.65)

Monotone decay is the signature of re-generation, not tracking: a tracking failure would jump; this slides, a little further every frame. There is no identity state anywhere in the system, so the face has to survive as a pattern across 30 latent frames while CFG pulls each latent frame toward the text prior at every step, and the prompt says “a woman,” not “this woman.” First-frame conditioning is the strongest mitigation, converting an unanchored trajectory into one with a fixed endpoint; reference-image conditioning and a per-frame identity loss during fine-tuning help further.

The rest

Nine further failures, each with the signal that detects it and the guard against it. OCR below is optical character recognition, used to check whether a sign spells the same word in every frame.

Alternatives considered and rejected

Each option here appeals for a real reason, and each is ruled out by a specific number. GAN is a generative adversarial network, a generator and a critic trained against each other, producing an image in one forward pass instead of dozens of denoising steps; mode collapse is its characteristic failure, where the generator finds a few outputs that fool the critic and stops varying.

Conclusion

Video generation is a sequence-length problem wearing a modelling problem’s clothes. The load-bearing facts:

• Consistency is a property of the joint distribution, not a post-processing step. A per-frame model learns only the marginal and can never be made consistent by sharing seeds or warping frames.
• Attention scales with the square of frames · area. Full 3D attention over 120 frames is 14,400x an image’s attention term and $3.43 a clip. Factorizing into spatial and temporal attention (114x) plus 4x temporal latent compression (the only lever on a quadratic term) lands at $0.178 with a larger model.
• Temporal compression is lossy in time. 4x compression on 24 fps sets a 3 Hz Nyquist ceiling; faster motion aliases, which is a sampling-rate fact no training fixes and the reason a high-motion tier exists.
• First-frame conditioning is worth more than any architecture change, because it hands the whole capacity budget to motion and gives evaluation a reference to compare against.
• The economics decide the product. A clip costs about 113 images; a $9.99 subscription buys 29 clips or 3,200 images. That ratio forces metering, and a ~5,700-GPU capacity commitment forces the async job queue, tiering, and the draft tier.
• Every automated video metric is blind or gameable. Human evaluation, at $405 a model pair against a $342,000-a-day fleet, is the only instrument that sees the failures.

The three assumptions that reach furthest are the frame count (decides the compute), the temporal compression ratio (decides compute and the quality ceiling), and the utilization figure (decides the pricing model). The rest of the assumptions below can each be wrong by a factor of two without changing a single architectural conclusion.

One line to remember: video generation is a sequence-length problem in a modelling problem’s clothing, and the token arithmetic decides the product, the metering, and the queue before any modelling taste gets a vote.

Further reading

• Ho, Jain, Abbeel, Denoising Diffusion Probabilistic Models (2020), the diffusion training objective.
• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (2022), latent diffusion and the VAE.
• Peebles, Xie, Scalable Diffusion Models with Transformers (2022), the diffusion transformer (DiT).
• Ho, Salimans, Classifier-Free Diffusion Guidance (2022), CFG.
• Ho et al., Video Diffusion Models (2022), extending diffusion to the temporal axis.
• Blattmann et al., Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023), latent video diffusion, data curation, and the causal image-and-video training setup.
• Unterthiner et al., Towards Accurate Generative Models of Video: A New Metric & Challenges (2018), the FVD metric and its limitations.

Next: the ML system design track.