05 — Image Captioning

“Build a system that generates captions for the images our users upload.”

Image captioning takes a photograph and writes a sentence describing it. Three things carry the design.

1. “Caption” names an output format, not a goal. The two most common consumers — a screen reader announcing the image to a blind user, and a search index — want opposite outputs. So the framing question decides the model, the data and the metric.
2. How the system is actually built. A frozen image encoder, a frozen language model, and a small trained bridge between them that is 0.26% of the parameters.
3. Why the dominant failure is a confident description of an object that is not in the photograph — where that failure comes from mechanically, and which controls catch it.

By the end you will be able to size the fleet, compute the point at which self-hosting beats calling an API, and say which assumptions the whole design would collapse without.

The input is one image, optionally with a prompt, and the output is a string of text. Everything else in this chapter is a consequence of the fact that nothing in that specification says what the text is for.

Terms used throughout

You do not need to memorize this list. Skim it, and come back when a symbol stops making sense.

The prompt sounds like a modelling question. It is a framing question, and the framing decides the model, the data, and the metric. Candidates who start with “vision encoder plus text decoder” have already skipped the part that carries the interview.

1. What is the caption FOR?

One question has to come before any design, because two reasonable answers to it produce two systems with opposite error asymmetries.

Ask this before anything else. “Caption” names an output format, not an objective. Two of the most common consumers want opposite things.

The table below puts them side by side. Read the “Fatal error” row first — it is the one that propagates through the rest of the chapter, because it decides which mistake you tune the system to avoid.

Two more consumers show up often enough to name: moderation triage (recall of policy-relevant content at fixed precision, output is structured attributes not prose) and product catalog (attribute accuracy — material, color, sleeve length — where a wrong attribute is a return, not a bad sentence).

One row of that table needs unpacking. An A/B test splits live traffic between the current system and the new one and compares a metric; it can decide the search question because a search failure produces an observable event — the user reformulates the query — and it cannot decide the alt-text question because there is no event on the page that means “the description was true.”

The divergence is not cosmetic; it changes the loss you wish you could write. Alt text wants a loss that punishes false statements far more than omissions. Search wants the reverse. Cross-entropy — the standard training loss, which scores the model on the probability it assigned to each word the reference caption actually used — punishes neither asymmetrically. It punishes deviation from a particular string. Every failure in this chapter is a consequence of that gap between the training objective (match the reference tokens) and the product objective (be true, or be findable).

The diagram below traces the same four consumers to four different systems. One question at the top, four branches, and each branch carries its own data source, metric and launch gate:

• Alt text. A short, conservative decode configuration that can abstain when unsure. Trained on human-verified data. Measured on human correctness plus CHAIR — the hallucination metric derived in Offline metrics and why they all disagree with humans, which counts descriptions of objects that are not in the image. Gated offline only, because it does not A/B.
• Search index. The opposite branch, marked long · entity-dense, recall over precision. Recaptioned web-scale data, measured on search success rate, gated by an online A/B.
• Moderation. Structured attributes rather than prose. Its branch reads data: policy-labeled, metric: recall at 0.95 precision — meaning: of the policy-violating images, what fraction do you catch, holding false alarms at 5% of what you flag.
• Product catalog. Attribute extraction rather than prose. Trained on catalog attributes, measured per attribute with F1, the harmonic mean of precision and recall.

flowchart TD
    Q{"What is the caption FOR?"}
    Q -->|Alt text| A["Short · conservative<br/>abstain when unsure"]
    Q -->|Search index| B["Long · entity-dense<br/>recall over precision"]
    Q -->|Moderation| C["Structured attributes<br/>recall at fixed precision"]
    Q -->|Catalog| D["Attribute extraction<br/>not prose"]

    A --> A2["Data: human-verified<br/>Metric: human correctness + CHAIR<br/>Gate: offline only"]
    B --> B2["Data: recaptioned web scale<br/>Metric: search success rate<br/>Gate: online A/B"]
    C --> C2["Data: policy-labeled<br/>Metric: recall at 0.95 precision"]
    D --> D2["Data: catalog attributes<br/>Metric: per-attribute F1"]

    style Q fill:#1d3557,color:#fff
    style A2 fill:#2d6a4f,color:#fff
    style B2 fill:#bc6c25,color:#fff

One more thing to volunteer, because it changes the input specification: alt text is context-dependent and captioning is not. The same product photo needs “a navy wool overcoat, three-quarter length” on a shopping page and “the coat mentioned in paragraph 3” on a news article. If accessibility is the objective, the surrounding page text is an input, not a nicety — and that decision has to be made before you pick a model, because it doubles the prompt specification.

Assumptions in this stage. Every section of this chapter ends with a block like this one. It sorts the section’s assumptions into three bins: things you state (you are free to pick, and being wrong costs a re-derivation), things you ask (the answer changes the architecture, so it is worth an interviewer’s time), and things that are load-bearing (if one is wrong the design is not suboptimal — it is invalid). The full list is collected in the assumption ledger at the end.

• State out loud: that the four consumers named here are the ones this system serves, and that alt text and search are the two that matter most. Adding a fifth costs you a decode configuration and a metric, not an architecture.
• Ask, never assume: what the caption is for. This is the question the section exists to make you ask, and it belongs in the first two minutes, because the answer decides the length target, the error asymmetry, the data source, the metric and the launch process. A candidate who picks silently has optimized the wrong error asymmetry for half the traffic.
• Also ask: whether the surrounding page context is available at caption time, because that changes the input specification rather than the arithmetic.
• Load-bearing: that one model with two decode configurations can serve both consumers. If the two objectives turned out to need different weights rather than different decode settings — different training data, different fine-tunes — then this is two systems with two training pipelines and two eval harnesses, and every cost figure in Serving scale and cost roughly doubles.

2. The ML objective, and the two ways it is structurally wrong

Write down the loss the model is trained on and two structural defects fall out of it — the same two that show up later as the chapter’s dominant failure modes.

The model learns a conditional language model over caption tokens — meaning it predicts each next word of the caption given the image and the words so far:

P(caption | image) = prod_t P(w_t | w_<t, image)

loss = - sum_t log P(w_t* | w_<t*, image)        teacher forcing, cross-entropy

Read it symbol by symbol. w_t is the caption’s word at position t, and w_<t is every word before it. prod_t means multiply the per-word probabilities together across the whole caption, which is what “probability of the whole caption” decomposes into. The star in w_t* marks the reference caption — the human-written one in the training data — as opposed to whatever the model would have said.

The loss is the negative log of those reference-word probabilities, summed. It is zero when the model gave the reference word probability 1.0 at every position, and it grows as the model spreads probability elsewhere.

Teacher forcing is the choice hiding in w_<t*: at every position the model is shown the true previous words rather than its own. So position 12 is scored against the reference even if the model’s own words 1-11 went somewhere else entirely. Every position is graded independently against one string.

Two problems live in that expression, and both surface later as failure modes.

Problem 1: one-to-many. An image has thousands of valid captions. Cross-entropy against one sampled reference penalizes every valid alternative. Minimizing expected cross-entropy over a diverse reference distribution drives the model toward the mode of that distribution — the single most likely caption — and the mode of “captions humans write for this image” is the generic one. That is precisely why captioners produce “a man riding a horse on a beach” rather than “a ranch hand in a red jacket on a chestnut quarter horse.” The blandness is not a capability gap. It is the optimum of the loss you wrote.

Problem 2: no truth term. The loss measures token mismatch. A hallucinated refrigerator in a kitchen image and a stylistic variation cost the same. Nothing in the objective distinguishes wrong from different.

So the recipe has to install the missing terms later — either by preference optimization on hallucination-labeled pairs (Training) or by a verification pass at serving time (Object hallucination the dominant one). Say this in the framing, because it makes the rest of the design look inevitable rather than bolted on.

Assumptions in this stage.

• State out loud: that the caption is generated one token at a time by a language model conditioned on the image, rather than assembled from a detector’s output by a template — the template design is genuinely better on groundedness and is rejected in Alternatives considered and rejected for fluency reasons.
• Ask: how many reference captions per image the evaluation set will have, because the one-to-many problem is a statement about how thinly five references cover a space of thousands.
• Load-bearing: the claim that cross-entropy has no truth term, so nothing in training distinguishes a wrong statement from a differently-worded right one. Everything downstream — the preference-tuning stage, the detector gate at serving, the choice of CHAIR over CIDEr as the ship gate — exists only because of it. If the loss did contain a groundedness term, the honest design is to train harder and skip the entire guard layer.

3. Architecture

The model is three pieces — an image encoder, a language model, and a small bridge between them — and the interesting derivation is why the bridge can be tiny while both ends stay frozen.

3.1 The encoder: CNN or ViT

First, the image encoder — and the deciding column in the comparison below is not accuracy.

Two families compete. A CNN is a convolutional neural network: it slides small learned filters across the image, so it assumes from the start that nearby pixels belong together and that an object means the same thing wherever it appears. A ViT is a vision transformer: it cuts the image into a grid of fixed-size square patches, treats each patch as a token, and runs the same attention machinery a language model uses over that sequence — with no built-in assumption about locality at all.

The decisive column is “output shape.” A ViT cuts the image into patches and runs a transformer over them, which means its output is already the exact object a text decoder consumes: a sequence of vectors. That is not an aesthetic preference; it is what makes the prefix-projection join in The join cross attention vs prefix projection possible at all.

Token arithmetic for a ViT-L/14 at 336px — “L” for large, “/14” meaning 14-pixel patches, run on a 336-pixel-square image:

patches = (336 / 14)^2 = 24^2 = 576        plus 1 CLS token

The CLS token is one extra learned position, standing for “classification”, that the model uses to accumulate a summary of the whole image; it is why the sequence length is 577 rather than 576 in the compute below.

Now price a forward pass, derived rather than looked up. You will do this arithmetic three more times in Serving scale and cost, so it is worth following once slowly.

The three numbers you need about this encoder: it has L = 24 layers, a width of d = 1024 numbers per position (the length of the vector each patch is represented by), and T = 577 tokens in the sequence. Each layer holds an attention block plus a FFN — a feed-forward network, the two-layer perceptron every transformer block applies to each position independently, here four times as wide as d.

Two rules do all the work:

• Count the weights per layer. Attention has four square projection matrices — query, key, value, and output — each d x d, so 4d^2. The FFN goes d -> 4d -> d, which is two matrices of 4d^2 each, so 8d^2. Total 12d^2 per layer.
• A matrix multiplication costs two FLOPs per weight per token — one multiply, one add. So FLOPs = 2 · params · tokens.

Attention itself gets a separate line, because comparing every token to every other token is not a weight multiply: it grows with T^2 rather than with T. Per layer it is 4 · T^2 · d — two T x T score matrices (query-key, then score-value) at 2 FLOPs each.

Substituting:

params per layer (matmuls)  =  4d^2 (QKVO) + 8d^2 (FFN)  =  12d^2
                            =  12 x 1024^2               =  12.6 M
total                       =  24 x 12.6 M               ≈  302 M

matmul FLOPs   =  2 · params · tokens  =  2 x 302e6 x 577   ≈  349 GFLOP
attention term =  24 · 4 · T^2 · d     =  24 x 4 x 577^2 x 1024  ≈  33 GFLOP
                                                            -----------
                                                            ≈ 382 GFLOP / image

Notice that attention is only 9% of the total here (33 of 382). At 577 tokens the quadratic term has not yet caught up with the linear one — which is why the hi-res path in What visual tokens actually buy, running the encoder five times rather than on one long sequence, stays affordable.

One footnote worth carrying. Vision papers quote 191 “GFLOPs” for this configuration, because they are counting multiply-accumulates, treating one multiply plus one add as a single operation. Same quantity, halved. Knowing that is worth a point.

3.2 The join: cross-attention vs prefix projection

This is the architectural decision the interviewer is fishing for, and the point is to derive — rather than assert — why the simpler of the two options works.

There are two ways to let a language model see an image.

Cross-attention (Flamingo-style). Insert new layers into the decoder whose job is to attend from text positions out to the vision features. Text tokens attend to each other as usual; the new layers additionally look at the image. They are gated, meaning each starts with a learned multiplier at zero so the untouched language model is the starting point and the image influence is dialled up during training.

Prefix projection (LLaVA-style). A small MLP — a multilayer perceptron, which here is just two matrix multiplications with a nonlinearity between them — maps each patch embedding from the vision model’s width of 1,024 numbers to the language model’s width of 4,096, and the resulting vectors are spliced into the token sequence in the slots where token embeddings would otherwise sit.

Now derive why the second one works, because “it just works” is not an answer:

The LLM’s very first operation on a token id is a lookup in an embedding table that returns a vector in R^4096. Nothing downstream knows or cares where that vector came from. Any module that emits R^4096 vectors is a legal token source.

So the projector’s entire job is to land image features in the region of R^4096 that the frozen decoder already interprets as content words about a scene. That is why you can freeze the LLM. The knowledge about horses, beaches, and kitchens already exists in the frozen weights; training only has to learn the translation into a vocabulary the model already has. You are not teaching it the world.

Size the projector. It is two matrices: one that takes the vision width 1,024 up to the LLM width 4,096, and one 4,096-square layer after it. A matrix from m inputs to n outputs has m x n weights:

1024 -> 4096 -> 4096   =  1024·4096 + 4096·4096  =  4.2M + 16.8M  =  21 M params
against an 8B decoder:    21e6 / 8e9  =  0.26 %

You train 0.26% of the parameters and get most of the capability, which is why the alignment stage costs GPU-hours rather than GPU-months.

Freezing a component means holding its weights fixed and letting no gradient change them, so only the unfrozen part learns. Here both ends — the vision encoder and the language model — are frozen, and only the 21M-parameter bridge between them moves.

Three terms used in the table. The KV cache is the store of already-computed key and value vectors that lets a decoder avoid re-reading the whole prompt on every token (The kv cache the most important mechanism in this chapter). Prompt caching is the provider-side reuse of that store for a prefix it has seen before, so a repeated prefix is billed at a fraction of the normal rate. Structured output means forcing the model’s text to conform to a declared shape such as a JSON object.

The table compares the two joins on the axes that decide it. The decisive rows are “Serving stack” and “Trainable params” — cross-attention wins on context cost and loses on everything operational.

flowchart LR
    IMG(["Image<br/>336 x 336"]) --> P["Patchify 14x14<br/>576 patches"]
    P --> V["ViT-L encoder<br/>302M params · frozen<br/>382 GFLOP"]
    V --> R["Token reducer<br/>pool or resampler<br/>576 -> 64"]
    R --> PROJ["Projector MLP<br/>1024 -> 4096 -> 4096<br/>21M params · TRAINED"]
    PROJ --> SEQ["Token sequence<br/>64 image vectors<br/>+ prompt tokens"]
    TXT(["Prompt"]) --> SEQ
    SEQ --> LLM["Decoder LLM · 8B<br/>FROZEN in stage 2"]
    LLM --> OUT(["Caption"])

    style PROJ fill:#2d6a4f,color:#fff
    style LLM fill:#1d3557,color:#fff
    style R fill:#bc6c25,color:#fff

Read the diagram left to right as one image’s path. A 336 x 336 image reaches the box marked patchify 14x14 576 patches. Those run through the ViT-L encoder 302M params · frozen 382 GFLOP. A token reducer — either a pool or a resampler, both defined in What visual tokens actually buy — compresses 576 vectors down to 64. A projector MLP of 21M parameters, mapping 1024 -> 4096 -> 4096, is the only thing trained, and it turns those 64 vectors into things the language model will accept as tokens. They are concatenated with the prompt’s own tokens into one sequence, which the 8B decoder LLM, frozen in stage 2, reads before emitting the caption.

Only the green box is trained in stage 2. That single fact is the design.

3.3 Contrastive pretraining, and why one shared space buys zero-shot

Where does the image encoder come from? The way it was trained is what makes the 21M-parameter bridge sufficient.

CLIP — contrastive language-image pretraining — is the standard way to train an image encoder and a text encoder so that they land in the same space.

The procedure, in four steps:

1. Take a batch of N image-text pairs.
2. Encode both sides, giving N image vectors i_1..i_N and N text vectors t_1..t_N.
3. Compute the similarity of every image against every caption, producing an N x N grid.
4. Treat that grid as an N-way multiple-choice question, run in both directions: given this image, which of the N captions is its own — and given this caption, which of the N images.

The loss is that multiple-choice question written down:

L_i2t = - (1/N) sum_k  log[ exp(cos(i_k, t_k)/tau) / sum_j exp(cos(i_k, t_j)/tau) ]
L     = (L_i2t + L_t2i) / 2

Read the fraction inside the log as a softmax. The numerator is image k’s similarity to its own caption. The denominator sums over similarity to every caption in the batch. So the fraction is “the probability the model assigns to the right answer,” and the negative log of it is zero when the model is certain and correct.

The rest of the symbols: cos is cosine similarity, the cosine of the angle between two vectors, which is 1.0 when they point the same way. tau is a learned temperature that sharpens or flattens the resulting distribution. L_i2t is the image-to-text direction, L_t2i is the reverse, and the two are averaged.

The negatives are the other items in the batch, which makes batch size a hyperparameter of the loss and not merely of the optimizer. A positive is a true image-caption pair and a negative is a wrong pairing the model must reject. N = 32,768 gives 32,767 negatives per positive; N = 256 gives 255. The task is an (N-1)-way discrimination, so its difficulty — and therefore the gradient signal — scales with N. That is why contrastive training uses absurd batch sizes, and why gradient accumulation — summing gradients over several small batches before applying one update, the usual trick for simulating a big batch on small hardware — is not a substitute here: accumulation buys more steps of an easy problem, not one step of a hard one.

Now the payoff. Zero-shot means classifying into categories the model was never trained on. A CLIP-style embedding is a lossy compression, and the training objective decides what survives (Embeddings and why dense search misses err_4021 derives this at length). CLIP’s objective is cross-modal, so what survives is exactly what an image and its caption have in common. Classification then collapses into retrieval: embed the sentence "a photo of a {class}" for every candidate class, embed the image, and take whichever class has the highest cosine similarity — argmax meaning simply “the argument that maximizes”. No classifier head, no labels, and the set of classes can be changed at runtime by typing different sentences.

Two consequences for the captioner specifically:

1. A CLIP-pretrained encoder’s outputs are already text-shaped, so the projector’s job shrinks from “learn a modality translation” to “learn a near-linear remap.” This is why a frozen encoder + 21M-param MLP + frozen LLM converges on a few hundred thousand pairs instead of a few hundred million.
2. CLIPScore comes for free as a reference-free evaluation metric (Offline metrics and why they all disagree with humans) — where reference-free means it scores a caption against the image itself rather than against a human-written answer key — and it inherits every weakness of the space it measures in.

Assumptions in this stage.

• State out loud: a ViT-L/14 encoder at 336 pixels, an 8B decoder, a 21M-parameter projector, and a contrastive batch size in the tens of thousands. Different sizes move every FLOP figure in Serving scale and cost proportionally and change nothing structural.
• Ask: whether a suitable CLIP-style checkpoint can be downloaded or must be trained, because the answer is the difference between 100 GPU-hours and weeks on hundreds of GPUs — the first row of the training table.
• Load-bearing: that the encoder’s outputs are already text-shaped, because it was trained against text. That is what shrinks the projector’s job from learning a modality translation to learning a near-linear remap, and it is why a 21M-parameter bridge on a few hundred thousand pairs converges at all. With a vision encoder trained on image labels instead of captions, the bridge has to be far larger, the frozen decoder stops being viable, and the alignment stage becomes a real training project.
• Also load-bearing: that the decoder can stay frozen. If it cannot, the “train 0.26% of the parameters” claim collapses and this is an 8B fine-tune with the memory bill in Training.

4. Data and labels

Hundreds of millions of image-caption pairs have to come from somewhere, filtering throws most of them away, and one purchase buys more quality per dollar than any architecture change.

4.1 Web alt text is the only source at scale, and it is mostly garbage

One filter in the funnel below is both the largest cut and the most dangerous.

Three terms used in the funnel. A perceptual hash is a short fingerprint computed from an image’s visual content rather than its bytes, so that a re-encoded or lightly cropped copy produces a nearly identical fingerprint. NSFW means not safe for work, the standard label for sexual or graphic content. PII is personally identifiable information — names, faces, addresses, license plates.

The funnel below starts from raw crawled pairs and ends at what is trainable. Each line reads: the filter, how many millions of pairs it removes, and how many survive it. Watch the fourth line.

raw crawled image + alt-text pairs                  ~5,000 M
drop empty / under 5 chars / filename-shaped        -2,200 M   ->  2,800 M
dedupe: perceptual hash on image, exact on text     -  900 M   ->  1,900 M
drop non-target-language and gibberish              -  400 M   ->  1,500 M
CLIP similarity filter, cos(img, txt) ≥ 0.28        -1,100 M   ->    400 M
NSFW / face / PII / watermark filters               -   40 M   ->    360 M
                                                                 -------
                                                    survival rate  7.2 %

The CLIP filter is the sharpest cut in the funnel and by far the most dangerous. Sharpest: it throws away 1,100 M of the 1,500 M pairs reaching it, or 73%, where the first filter only cut 44% of what it saw (2,200 of 5,000). The earlier filters remove more pairs in absolute terms, but they remove obvious junk — empty strings, IMG_4021.jpg, exact duplicates — and nobody argues about them.

Most dangerous: you are filtering with a model that was trained on the data you are filtering. Round two keeps what the current model already understands and discards what it does not, which silently narrows the distribution toward the model’s existing competence. Nothing in the pipeline reports this, because the discarded pairs leave no trace.

Two mitigations worth naming: filter with a model trained on a different, smaller, cleaner source, and always retain an unfiltered random sample so you can measure what you threw away.

4.2 Synthetic recaptioning, and what it costs

The highest-leverage purchase in the chapter has a price worth deriving — and a reason you mix its output with the garbage rather than replacing it.

Surviving alt text is still short and non-descriptive: blue dress for a photograph of a woman in a blue dress on a beach at sunset. Recaptioning runs a strong vision-language model — the teacher — over your images and has it write dense descriptions, which then become the training targets for your own smaller model, the student. This is distillation: training a small model to imitate a large one.

Price it, because the number is the surprise. Read the middle line as: total images, divided by images per second, gives seconds; divided by 3,600 gives hours.

100 M images, 60 output tokens each, 70B teacher on rented H100s
throughput  ≈ 8 img/s/GPU  (prefill-dominated, batch 32)
GPU-hours    = 100e6 / 8 / 3600  =  3,472 GPU-hours
at $2.50/GPU-hr =  $8,700   compute
plus egress, storage, orchestration, reruns   ->  call it $25-40 k

Spelled out: 100e6 / 8 = 12.5 M GPU-seconds, and 12.5e6 / 3600 = 3,472 GPU-hours, and 3,472 x $2.50 = $8,680. The compute is the small part. The $25-40k all-in figure is dominated by everything around the compute.

Recaptioning 100M images costs roughly one engineer-month and moves quality more than any architecture change you could make in that month. That comparison is the point.

The quality/quantity tradeoff is not monotone, and the reason is that the two sources are not nested. Original alt text is the only place rare proper nouns live: a model trained purely on synthetic captions writes “a red sports car” where the alt text said “2019 Alfa Romeo Giulia Quadrifoglio.” Synthetic is the only place full-sentence grounded description lives. Mixing is a union of two non-overlapping capabilities, not a compromise between two grades of the same thing.

One hard ceiling to state out loud: a student distilled from teacher captions cannot beat the teacher on the axis being distilled — only on cost. If the teacher hallucinates appliances in kitchens, your student now hallucinates appliances in kitchens, fluently and confidently. Measure the teacher’s CHAIR_s (Chair_i and chair_s are two different numbers) before spending the $30k.

4.3 The eval set is where the human labels go

Where does the human labelling budget go? The evaluation set, not the training set.

You do not need human labels to train. You absolutely need them to decide. Buy 2,000 images, stratified by domain — meaning deliberately sampled so each category is represented rather than left to chance — across people, documents, products, scenes, charts and screenshots. Each gets 5 independent reference captions and a per-object presence annotation, a list of which objects are actually in the image, so that CHAIR (Offline metrics and why they all disagree with humans) is computable. Budget around $15k and treat it as infrastructure, not a project.

Assumptions in this stage.

• State out loud: a 5,000M-pair raw crawl surviving at 7.2%, a CLIP similarity cut at 0.28, an 80/20 mix of synthetic to original captions, 100M images recaptioned by a 70B teacher at 8 images per second per GPU, and a 2,000-image evaluation set at roughly $15k. Every one is a number you would measure and revise.
• Ask: whether you have the rights to train on the crawled images, and whether the teacher model’s licence permits using its output to train a competitor. Both are legal answers that delete a section of this design rather than adjusting it.
• Load-bearing: that the two caption sources are not nested — that original alt text is the only place rare proper nouns live and synthetic is the only place grounded descriptive structure lives. The 80/20 mix is a union of two disjoint capabilities on that assumption; if synthetic captions were simply better on every axis, the right move is 100% synthetic and the mixing argument is wasted complexity.
• Also load-bearing: that the teacher’s own hallucination rate is low enough to distil from. The student inherits it exactly, so a teacher with a bad CHAIR_s is a $30k purchase of a hallucination habit.

5. Training

Training runs in four stages, and the interesting part of each is the reason it is trainable at the scale it is.

Four abbreviations appear in the table and are unpacked immediately below it: InfoNCE (information noise-contrastive estimation) is the name of the contrastive loss written out in Contrastive pretraining and why one shared space buys zero shot, CE is cross-entropy, LoRA is low-rank adaptation, and DPO is direct preference optimization.

Read the “Trainable” and “Scale” columns together. Stage 1 is the only one that costs a cluster, and it is also the only one you can skip by downloading someone else’s checkpoint. Everything after it is small because of what is not in the Trainable column.

LoRA freezes the original weights and trains a small pair of thin add-on matrices alongside them, so a fraction of a percent of the parameters carries the update and one base model can host many task-specific adapters. DPO trains on pairs where something has declared A better than B, rather than on a single correct answer — which is exactly the shape you need when the problem is not “what is the right caption” but “which of these two captions is more truthful.”

Why stage 2 freezes the decoder. If you unfreeze an 8B decoder while the projector is still emitting noise, the decoder’s gradient is dominated by adapting to that noise — it will degrade its own language ability to accommodate a bad image representation, and you will not get it back. Frozen, the only path to lower loss runs through the projector. The gradient is forced into the 21M parameters that actually need to learn something.

Why stage 3 uses LoRA. The argument is memory, not quality. Price a full fine-tune of the 8B decoder and compare.

Training a parameter means holding four things about it in GPU memory: the weight, its gradient, and the optimizer’s two running statistics. bf16 is a 16-bit floating-point format, so a weight or a gradient costs 2 bytes each. Adam, the standard optimizer, keeps two running statistics per parameter — a running mean m and a running squared magnitude v — usually in 32-bit, so 4 bytes each, 8 bytes for the pair.

Multiply each by 8 billion parameters:

weights        8e9 x 2 bytes   =  16 GB
gradients      8e9 x 2 bytes   =  16 GB
Adam m, v      8e9 x 8 bytes   =  64 GB       (fp32 moments)
                                  ------
                                   96 GB   before a single activation

96 GB does not fit on an 80 GB H100. Note also that the optimizer state alone is four times the weights, which is why “the model is 16 GB, my card has 80” is a misleading way to think about training.

Now LoRA. Rank 16 on the attention projections trains ~20M parameters instead of 8e9, so the Adam moments are 20e6 x 8 = 160 MB — a rounding error. The frozen base weights still cost their 16 GB, but nothing else scales with 8 billion, and the whole job fits on one 80 GB card with room for activations and a real batch.

That is not a quality argument. It is the difference between a one-node job and a multi-node job, and therefore between iterating in an afternoon and iterating in a week.

Why stage 4 exists. The ml objective and the two ways it is structurally wrong: cross-entropy has no truth term. Running DPO on pairs of (accurate caption, hallucinated caption) for the same image installs the missing asymmetry directly into the model’s behaviour. Building the pairs is fully automatic, which is the whole reason it is affordable. An open-vocabulary detector is an object detector you can query with any noun in plain English — rather than one restricted to a fixed list of classes it was trained on — and it returns a confidence score and a box:

for each image:
    sample 4 captions at T = 1.0
    run an open-vocabulary detector over the image
    label each caption: clean if every named object is detected above 0.35
    emit (clean, hallucinated) pairs where both exist

The sampling temperature T = 1.0 means drawing each word from the model’s own probability distribution untouched, which produces four genuinely different captions rather than four near-copies. No annotators. The detector is the label function.

Assumptions in this stage.

• State out loud: stage sizes of 1M alignment pairs, 500k task-formatted examples and 30-60k preference pairs, LoRA at rank 16 on the attention projections, and a detector confidence threshold of 0.35. Each is a starting point you would tune.
• Ask: whether stage 1 can be skipped by downloading a contrastive checkpoint, since that is the difference between the whole pipeline fitting on one node and needing a cluster.
• Load-bearing: that the open-vocabulary detector is accurate enough to be a label function. Stage 4’s entire economics rest on labels being free; a detector that misses real objects manufactures “hallucinated” labels for true statements and trains the model toward saying less, which is the correctness-versus-specificity dial being turned the wrong way by a broken instrument. The same detector is also the serving-time gate in Object hallucination the dominant one, so its accuracy is load-bearing twice.
• Also load-bearing: that LoRA at rank 16 recovers most of a full fine-tune’s quality here. If it does not, stage 3 is a 96 GB multi-node job and the iteration speed the whole recipe depends on disappears.

6. Offline metrics, and why they all disagree with humans

Every automatic caption metric ranks models differently from humans. Two mechanisms account for the whole disagreement, and a real pair of models below shows the standard metric shipping the worse one.

Eight metrics appear in the table below. Read these definitions first; the table then says what each one breaks on.

The first five are reference-based — they compare your caption against human-written answer keys, usually five per image:

• BLEU (bilingual evaluation understudy) counts shared runs of consecutive words, an n-gram being a run of n of them.
• METEOR does the same with a looser matcher that accepts word stems and synonyms.
• ROUGE-L scores the longest sequence of words appearing in both, in order but not necessarily adjacent.
• CIDEr weights n-grams by TF-IDF — term frequency times inverse document frequency, which raises the weight of phrases that are common in this image’s references and rare across the corpus, so generic phrases count for little.
• SPICE parses both the candidate and the references into a scene graph — a set of (object, attribute) and (object, relation, object) tuples — and scores the overlap of those tuples with F1.

The last three need no answer key. They look at the image itself:

• CLIPScore compares the caption to the image directly in CLIP’s shared space.
• CHAIR — caption hallucination assessment with image relevance — counts objects the caption named that a detector cannot find. It comes in two versions, CHAIR_i and CHAIR_s, and the difference is what the next subsection is entirely about.

In the table, “human correlation” means how well the metric’s ranking of models matches a human panel’s ranking. Read the last column first: it is where each metric fails.

Two mechanisms explain the whole column of disagreement.

1. Reference-based metrics measure agreement with a sample, not correctness. The space of valid captions for a photograph is enormous; five references cover a vanishing fraction of it. N-gram overlap against five draws from a distribution of thousands is largely measuring stylistic conformity. A metric that rewards matching the reference rewards the generic caption — which is exactly the The ml objective and the two ways it is structurally wrong failure mode. Optimizing CIDEr actively pushes the model toward the thing you were trying to fix.

2. Reference-free metrics measure similarity, not entailment. CLIPScore asks “is this text plausibly about this image,” which is invariant to the differences that decide whether a caption is true.

Below, one image scored against three captions. Caption A is correct, B is its exact logical inverse, and C is a word salad with no relations at all. Look at the gap between A and B:

image: a dog chasing a man across a lawn

A  "a dog chasing a man across a lawn"      CLIPScore 0.79
B  "a man chasing a dog across a lawn"      CLIPScore 0.77
C  "a dog, a man, and a lawn"               CLIPScore 0.75

Two points separate a correct caption from its exact inverse. The mechanism: a bag-of-concepts representation — one that records which things are present and discards the relations between them — is sufficient to win the contrastive task on 400M web pairs, so nothing in CLIP’s objective ever required the text encoder to represent who did what to whom. The metric cannot see a fact and its reversal as different, because the space it lives in cannot.

CHAIR_i and CHAIR_s are two different numbers

A subscript rarely deserves this much attention, but these two versions can differ by a factor of two on identical data.

They share a name and a detector pass and they are not interchangeable. CHAIR_i divides by object mentions; CHAIR_s divides by captions. One caption that invents five objects moves CHAIR_i five times as much as a caption that invents one, and moves CHAIR_s identically.

Every headline CHAIR figure in this chapter is CHAIR_s — the 0.181 / 0.066 pair below, the four rows of What visual tokens actually buy, and the 0.048 -> 0.168 length sweep in Object hallucination the dominant one. The single worked example in Object hallucination the dominant one is CHAIR_i, because it scores one caption, and a per-caption rate computed over one caption is either 0 or 1.

Quote a CHAIR number without the subscript and nobody can reproduce it. The code below computes both from the same input, so you can see exactly where they diverge. Each caption is a pair (object mentions, hallucinated mentions) — so (7, 2) means the caption named seven annotated objects, two of which the detector could not find:

def chair(captions):
    """Both CHAIR numbers from one detector pass over the same annotations.

    captions: (object mentions, hallucinated mentions) per caption, counted
    over the annotated object vocabulary only — scene words like "kitchen"
    are not in it and are not counted either way.
    """
    return {
        "CHAIR_i": sum(h for _, h in captions) / sum(m for m, _ in captions),
        "CHAIR_s": sum(1 for _, h in captions if h) / len(captions),
    }


# the §9.1 kitchen caption: 3 tomatoes + board + knife found, fridge + stove not
KITCHEN = (7, 2)
assert round(chair([KITCHEN])["CHAIR_i"], 3) == 0.286
assert chair([KITCHEN])["CHAIR_s"] == 1.0            # 3.5x apart, one caption

# a five-caption toy set: same detector, same annotations, one factor of two
TOY = [KITCHEN, (6, 3), (4, 0), (4, 0), (4, 0)]
both = chair(TOY)
assert round(both["CHAIR_i"], 3) == 0.200            # 5 bad mentions of 25
assert round(both["CHAIR_s"], 3) == 0.400            # 2 bad captions of 5
assert both["CHAIR_s"] == 2 * both["CHAIR_i"]

Two hallucinating captions out of five is 0.400; five hallucinated mentions out of twenty-five is 0.200. Identical data. Which one you report decides whether a model that concentrates its hallucinations into a few very bad captions looks better or worse than one that spreads them thinly — and those are genuinely different products. Report both, or report CHAIR_s and say so.

The disagreement, on two real candidates

On two actual models, the disagreement looks like this — and gating on the easiest metric to compute ships the worse one.

Two candidate models scored on six things. Every column except CHAIR_s is higher-is-better. Compare the CIDEr column against the last three columns and note that they point in opposite directions:

model   CIDEr   SPICE   CLIPScore   CHAIR_s    human       human
                                    (lower=      correct   "useful as alt text"
                                     better)
  A      1.21   0.211     0.762      0.181       71 %          44 %
  B      1.08   0.226     0.781      0.066       89 %          78 %

Model A learned the reference style. Model B is more specific and hallucinates a third as often. CIDEr picks A. Everything you actually care about picks B. If CIDEr is your ship gate, you ship A — and this is not a hypothetical, it is the normal outcome of gating on the metric that is easiest to compute.

Human eval, on separated axes

The most common human-evaluation mistake is asking raters to “rate this caption 1-5.” One number blends four independent properties, two of which move in opposite directions. (Fluency, the fifth row below, is saturated on any modern model and is not one of the four.)

Correctness and specificity are a dial, not a bug. A captioner tuned to say less is more correct and less useful. So report the pair, or report correctness at a fixed specificity band — the same discipline as reporting precision at a fixed level of recall (ml/06).

Assumptions in this stage.

• State out loud: five reference captions per image, a detector threshold of 0.35, and the fixed CHAIR object vocabulary — CHAIR only counts objects that appear in the annotation vocabulary, so scene words like “kitchen” are invisible to it in both directions.
• Ask: which of the five human axes the product is actually being judged on, because correctness and specificity move against each other and you cannot maximize both.
• Load-bearing: that CHAIR, computed by a detector, tracks the hallucination rate a human would report. It is the ship gate, the production monitor and the DPO label function, so if the detector’s vocabulary or accuracy fails to cover your image distribution, all three are measuring something other than what they claim.
• Also load-bearing: that the 2,000-image evaluation set represents live traffic. A gate stratified over the wrong domains passes models that are broken on the domains your users actually upload, which is worse than no gate because it is believed.

7. Online metrics and A/B

Offline metrics gate the launch; live traffic is the other half of the story. The two objectives from What is the caption for differ not only in metric but in whether an online experiment can decide them at all.

Search indexing A/Bs cleanly. The primary metric is search success rate: the user issues a query, clicks a result, and does not reformulate the query within 30 seconds. The guardrail — a metric that must not move, whatever the headline does — is CTR, click-through rate, on results that previously had no caption text at all, so that you can detect having added noise rather than signal. Size the experiment. MDE is the minimum detectable effect, the smallest change you want to be able to see; alpha is the false-positive rate you accept; power is the chance of detecting a real effect that exists:

baseline success 62 %, MDE 1 point absolute, alpha 0.05, power 0.8
n per arm ≈ 16 · p(1-p) / delta^2  =  16 x 0.62 x 0.38 / 0.0001  ≈  37,700 queries
two arms                                                          ≈  75,400 queries

Three things about that line, because the constants are not obvious.

Where the 16 comes from. It is 2 · (z_alpha/2 + z_power)^2, the standard sample-size constant for comparing two proportions. At alpha 0.05 and power 0.8 those z-values are 1.96 and 0.84, so 2 x (1.96 + 0.84)^2 = 2 x 7.84 = 15.68, rounded to 16. Change alpha or power and only this constant moves.

Where p(1-p) comes from. It is the variance of a coin flip at rate p. A metric that sits near 50% is the noisiest and needs the most traffic; one near 1% or 99% needs far less. Here 0.62 x 0.38 = 0.2356.

Where delta^2 comes from. delta is the MDE as a fraction, so 1 percentage point is 0.01 and delta^2 = 0.0001. It is squared, which is the number that hurts: wanting to detect half the effect costs you four times the traffic.

Putting it together: 16 x 0.2356 / 0.0001 = 37,696 per arm, so about 75,400 across two arms.

Now convert that to wall-clock. At 1M queries/day the rate is 1e6 / 86,400 = 11.6 queries/s, so 75,400 / 11.6 ≈ 6,500 seconds ≈ 1.8 hours — about two hours of traffic, and only if the whole stream is in the experiment. Put 10% of traffic in it and you wait 18 hours.

One trap to name: captions change a shared index, so a user-level split leaks between arms. If both variants write into the same index, a user assigned to the control arm can still be served a document the treatment arm recaptioned, and the two arms stop being independent. You need two indexes with a query-level split, or an interleaving design, where both systems’ results are mixed into a single result list for every user and you measure which system’s items get clicked — the same interference argument as Why offline ranking metrics disagree with online ctr.

Alt text does not A/B. There is no online event that means “the description was correct.” A screen-reader user who hears a false description has no way to signal that, and the page produces no click, no scroll and no reformulation that means “wrong.”

So you work with proxies. The table below ranks four of them by how much decision weight each can carry — read the third column, because only one row is strong enough to gate a launch:

Edit distance is the number of single-character insertions, deletions and substitutions needed to turn one string into another, and it is the underrated signal here. A 12% edit rate with a median edit distance of 30 characters is a completely different system from a 12% edit rate with a median of 3 — the first means authors are rewriting the content, the second means they are fixing a comma. Report the distribution, not the rate.

Assumptions in this stage.

• State out loud: a 62% baseline search success rate, a 1-point absolute minimum detectable effect, 1M queries a day, and a report-this-description base rate of about 0.02%. Those set the experiment’s duration and nothing else.
• Ask: whether the alt-text surface allows authors to edit the generated description, because that single product decision is the difference between having a decision metric and having none.
• Load-bearing: that captions are written into a shared index rather than a per-user one. That is what makes a user-level split leak between arms and forces either two indexes with a query-level split or an interleaving design; a design that assumed user-level randomization works would produce a contaminated readout that looks perfectly valid.
• Also load-bearing: that alt text genuinely has no online success event. Everything in the launch process for accessibility follows from it — the offline gate, the human correctness panel, the edit-distance proxy. If a real online signal existed, the whole apparatus is replaced by an ordinary A/B.

8. Serving, scale, and cost

Now turn the design into a request path, price it per image, and derive the routing decision that is the dominant cost lever — along with the honest statement of what that routing gives up.

Read the diagram below as one upload’s path, top to bottom. Six stages, and only two of them are the model:

1. Dedupe. The image is hashed twice — a content hash on the exact bytes, and a perceptual hash on the visual content. A hit returns the cached caption immediately and costs nothing.
2. Safety pre-filter. NSFW classifiers, a hash match against known child sexual abuse material (CSAM), and a face count. Anything blocked returns no caption · flag. This runs before the captioner, not after it — a captioner has no notion of policy and will describe anything neutrally.
3. The router. A text-presence detector costing 3 ms. The 92% of images with no text are encoded at 336 pixels into 64 visual tokens. The 8% with text go to dynamic tiling — four native-resolution crops plus a thumbnail, 2,880 visual tokens. This single box is the dominant cost lever in the whole chapter.
4. The model. Both lanes converge on projector + LLM decode, batched.
5. The guards. Contrastive decoding subtracts the measured language prior. Then a post-filter checks noun-phrase grounding against a detector, checks attribute terms against a lexicon, and checks agreement across three samples. On an unsupported span the gate strips or hedges it, and what emerges is a caption + confidence.
6. The writes. The result goes both to an alt-text store and to a search index holding the text and its embedding.

The orange box and the green box are where the interesting decisions live: the router decides cost, the post-filter gate decides truth.

flowchart TD
    U(["Upload"]) --> H["Content hash + pHash<br/>dedupe cache"]
    H -->|hit| CACHED(["Cached caption"])
    H -->|miss| SAFE{"Pre-filter<br/>NSFW · CSAM hash · face count"}
    SAFE -->|block| REJ(["No caption · flag"])
    SAFE -->|pass| TXT{"Text-presence<br/>detector · 3 ms"}
    TXT -->|no text · 92%| LOW["Encode at 336<br/>64 visual tokens"]
    TXT -->|text · 8%| HI["Dynamic tiling<br/>4 crops + thumbnail<br/>2,880 visual tokens"]
    LOW --> DEC["Projector + LLM decode<br/>batched"]
    HI --> DEC
    DEC --> CD["Contrastive decode<br/>subtract language prior"]
    CD --> GATE{"Post-filter<br/>noun-phrase grounding<br/>attribute lexicon<br/>3-sample agreement"}
    GATE -->|unsupported span| STRIP["Strip or hedge the span"]
    GATE -->|clean| OUT(["Caption + confidence"])
    STRIP --> OUT
    OUT --> S1[("Alt-text store")]
    OUT --> S2[("Search index<br/>text + embedding")]

    style TXT fill:#bc6c25,color:#fff
    style GATE fill:#2d6a4f,color:#fff
    style SAFE fill:#1d3557,color:#fff

Latency budget, sync path, batch 8

The per-image latency decomposes into lines that each have a different fix, and the two available levers can then be priced against each other.

This is the flat 576-token configuration — the pre-routing baseline that the rest of this section exists to beat.

Two phases split the model’s work, and they are limited by different things:

• Prefill reads the whole input — image tokens plus prompt tokens — in one parallel pass. It is limited by arithmetic, so its cost follows one formula: 2 · params · tokens.
• Decode then emits the caption one token at a time. It is limited by how fast the model’s weights can be pulled out of memory, not by arithmetic — every single token requires reading all 8 billion weights. That is why halving the model halves decode, and why doubling the token count does not change the per-token decode time at all.

Every prefill line below counts the 40-token prompt alongside the image tokens, and batch 8 means eight images are processed together, so the token counts are multiplied by 8. To convert any FLOP figure into milliseconds, divide by the H100’s 300 TFLOP/s and multiply by 1,000 — for example the encoder line: 3.1 / 300 = 0.0103 s = 10 ms.

fetch + JPEG decode                                    40 ms
resize + normalize                                      5 ms
vision encoder   8 x 382 GFLOP = 3.1 TFLOP @ 300 TF/s  10 ms
projector                                              ~0 ms
LLM prefill      (576 + 40) x 8 = 4,928 tokens
                 2 x 8e9 x 4928 = 78.8 TFLOP @ 300 TF/s  263 ms
LLM decode       25 tokens x 4.8 ms (memory-bound)    120 ms
post-filter (detector + lexicon)                       15 ms
                                                      -------
                                                      453 ms  p50
                                                     ~900 ms  p95 with queueing

Prefill dominates, and prefill is 94% image tokens (576 of 616). That is the single most actionable number here — but be careful about why it points at tokens.

Prefill is 2 · params · tokens, which is symmetric in its two factors, so a 2x cut on either side buys the same prefill. Price both:

halve the decoder (8B -> 4B)     prefill 263 -> 131 ms   saves 131 ms
                                 decode  120 ->  60 ms   saves  60 ms   (weight-bandwidth bound)
                                                         ------------
                                                              191 ms

halve the visual tokens (576 -> 288)
                                 prefill 263 -> 140 ms   saves 123 ms
                                 decode unchanged

At equal ratio the decoder is the better lever, by 1.6x — 191 / 123 = 1.55 — because halving the decoder takes decode with it and halving the tokens does not.

So why does the chapter cut tokens instead? Not because tokens are worth more per unit. Because tokens have far more slack. You can cut visual tokens 9x (576 -> 64, prefill 263 -> 44 ms, saving 218 ms) for a 0.012 move in CHAIR_s — that is the 0.066 to 0.078 in the next table — and there is no 9x cut of the decoder that costs that little. An 8B model cut 9x is a 0.9B model, and it will not write a usable caption.

On the 64-token path this same budget lands at ~235 ms p50: 40 + 5 + 10 + 44 + 120 + 15.

What visual tokens actually buy

What do you lose by sending the language model fewer image tokens? The answer — fine detail, and almost nothing else — is what turns a token budget into a router.

Three terms in the table. Average pooling merges each 2x2 block of neighbouring patches into one vector by averaging them, cutting 576 to 144. A Perceiver resampler is a small learned module holding a fixed number of query vectors — here 64 — that attend over all 576 patch vectors and each produce one output, so the output length is fixed by design rather than by the input. OCR is optical character recognition, reading the text printed inside an image, scored here by exact match on the string.

Every cost row is the same formula — 2 x 8e9 x (V + 40) x 8 FLOP at 300 TFLOP/s, where V is the visual-token count in column one. No exceptions, so you can regenerate any row.

Read down the table and compare row 1 (the baseline) against row 3 (the cheap lane). Cost falls 6x. CHAIR_s barely moves. OCR falls off a cliff.

Note the floor. At 64 visual tokens the sequence is 64 + 40 = 104 tokens, so the prompt is 40 / 104 = 38% of prefill. Cutting visual tokens to 32 would only take the sequence to 72, a 1.4x saving rather than a 2x one. The last halving of tokens buys much less than the first.

Token count buys fine detail and almost nothing else. Across a 9x token reduction (576 -> 64), general captioning barely moves: CHAIR_s goes 0.066 -> 0.078. OCR collapses by 5x: 0.41 / 0.08 = 5.1. Counting drops from 0.58 to 0.42.

So the answer is not a single token budget — it is a route. Send the images that need detail down the expensive lane and everything else down the cheap one. Blended prefill with a 3 ms text detector and an 8% hi-res rate:

0.92 x 13.3 TFLOP  +  0.08 x 373.8 TFLOP  =  12.2 + 29.9  =  42.1 TFLOP
flat 576-token path:                                         78.8 TFLOP

78.8 / 42.1 = 1.87, so the router is 1.87x cheaper on prefill.

Now blend the quality columns on the same traffic split, because “and better on both axes” is the claim people make here and it is false. Each lane below is a tuple of (traffic share, prefill TFLOP, CHAIR_s, OCR, counting), and blended(col) is the traffic-weighted average of one column across the two lanes. The assertions are the argument — read them as the claims, not as test scaffolding:

LANES = [
    # share, prefill TFLOP, CHAIR_s, OCR exact-match, counting accuracy
    (0.92, 13.3, 0.078, 0.08, 0.42),          # 64-token lane
    (0.08, 373.8, 0.061, 0.79, 0.66),         # 2,880-token hi-res lane
]
FLAT = (78.8, 0.066, 0.41, 0.58)              # 576-token path, same columns


def blended(col: int) -> float:
    return sum(lane[0] * lane[col] for lane in LANES)


assert round(blended(1), 1) == 42.1
assert round(FLAT[0] / blended(1), 2) == 1.87        # prefill: this is the win

# CHAIR_s (lower is better) and counting (higher is better) both get WORSE
assert round(blended(2), 4) == 0.0766 and blended(2) > FLAT[1]
assert round(blended(4), 3) == 0.439 and blended(4) < FLAT[3]

# OCR is the one column where a traffic blend is the wrong statistic. It
# prices OCR on the 92% of images that contain no text:
assert round(blended(3), 3) == 0.137
# The router is conditioned on text presence, so the text-bearing images all
# land in the hi-res lane, and that lane's number is the one to compare:
assert round(LANES[1][3] / FLAT[2], 2) == 1.93

1.87x cheaper on prefill and ~1.9x better on OCR for the images that actually contain text — paid for with 0.011 of CHAIR_s and 0.14 of counting accuracy. The 92% lane is the 64-token lane and it has the 64-token lane’s quality; routing does not improve it, it only makes it cheap. And the 8% were selected for text, which does nothing for counting, so counting takes the full hit of the token cut. That is a good trade for an alt-text or search product and a bad one for a catalog product that counts things — in which case the fix is a second route, to a detector (Counting), not a second opinion about this one. The shape — a cheap classifier that routes to unequal lanes — is the same lever as the router in case study 06, and it is the answer interviewers are listening for; the honest version of it names what the cheap lane gave up.

Throughput and cost per 1M images

Per-lane compute converts into images per second and then into dollars, and the end of that chain is the build-versus-buy line — a volume, not an opinion.

Batch path, batch 64, one H100 at 300 TFLOP/s effective. Same prefill formula as above — the 40-token prompt is not free in a batch job either:

flat 576-token lane
  encoder   64 x 382 GFLOP     =   24.4 TFLOP  ->     81 ms
  prefill   64 x (576 + 40)    =   39,424 tok
            2 x 8e9 x 39424    =  630.8 TFLOP  ->  2,103 ms      <- 91% of the batch
  decode    25 steps x 4.8 ms                  ->    120 ms
                                                    -------
                                                    2,304 ms / 64  =  36.0 ms/image  =  27.8 img/s

64-token lane
  encoder (unchanged — the ViT still runs; the resampler cuts what reaches the LLM)   81 ms
  prefill   64 x (64 + 40)     =    6,656 tok  ->  106.5 TFLOP  ->  355 ms
  decode                                                        ->  120 ms
                                                    556 ms / 64  =   8.7 ms/image  =  115 img/s

2,880-token hi-res lane (4 crops + thumbnail = 5 encoder passes)
  encoder   64 x 5 x 382 GFLOP =  122.2 TFLOP  ->    408 ms
  prefill   64 x (2880 + 40)   =  186,880 tok  ->  2,990 TFLOP  -> 9,967 ms
  decode                                                        ->   120 ms
                                                 10,494 ms / 64  = 164.0 ms/image  =  6.1 img/s

The hi-res lane is 19x slower than the cheap lane (164.0 / 8.7 = 18.9 ms per image). That changes how you combine the two rates, and getting it wrong is a classic slip.

Do not average the rates. 0.92 x 115 + 0.08 x 6.1 = 106.3 img/s is wrong, because the fleet does not spend images — it spends seconds. You have to average the seconds per image, then invert. That is a harmonic blend:

1 / routed  =  0.92 / 115  +  0.08 / 6.1  =  0.0080 + 0.0131  =  0.0211
routed      =  47.4 images/s

The two terms on the right are seconds of GPU time spent per image of total traffic. The cheap lane contributes 0.0080 and the tiny hi-res lane contributes 0.0131 — more than the other one. 0.0131 / 0.0211 = 62%.

The 8% pays 62% of the time. That is the number to volunteer: a route to an expensive lane is only cheap if the lane’s cost ratio is smaller than the traffic ratio, and here it is not — 19x cost against 11.5:1 traffic. It still wins, but by less than the 92/8 split suggests.

Converting images per second into dollars per million is one line, and it is worth doing once by hand so the table is reproducible:

1M images at 47.4 img/s  =  1e6 / 47.4        =  21,100 GPU-seconds
                         =  21,100 / 3600     =  5.86 GPU-hours
                         =  5.86 x $2.50      =  $14.66 per 1M images

Every row below is that same division. Faster lane, fewer GPU-seconds, smaller bill — the relationship is exactly inverse, so a 19x slower lane costs 19x more.

Three things about that last row, because it is the one people quote without checking.

What $3/$15 per MTok means. Three dollars per million input tokens and fifteen per million output tokens. That is the mid-tier hosted price.

Why ~1.5k image tokens is not the 576 or 64 above. Those are the ViT’s own 14-pixel patch grid, internal to a model you host. An API bills you on its patch grid, which you do not control. That grid is ceil(width / 28) × ceil(height / 28) tokens, capped on the standard tier at 1,568 pixels on the long edge and 1,568 tokens. So a roughly 1,092 x 1,092 upload is 39 × 39 = 1,521 tokens — just under the cap, and the source of the ~1.5k. Send a smaller image and the row gets cheaper in proportion; send a bigger one and it stops at the cap.

Where $4,950 comes from. Per image: 1,500 × $3/1e6 = $0.0045 of input plus 30 × $15/1e6 = $0.00045 of output, which is $0.00495. Multiply by a million images: $4,950.

A factor of ~340 against the routed path (4,950 / 14.66 = 338). Which does not mean “never call the API.” It means the build/buy line is a volume, so compute it.

The shape of the calculation: self-hosting has a large fixed cost and a tiny per-image cost; the API has zero fixed cost and a large per-image cost. Divide the fixed cost by the per-image saving and you get the volume at which they cross.

API cost per image                     $0.00495
self-hosted marginal cost per image    $0.0000147
delta                                  $0.00494

fixed cost of self-hosting:
  1 H100 at $2.50/hr, 24x365           $21,900 / yr
  20% of one engineer, fully loaded    $50,000 / yr
                                       --------
                                       $71,900 / yr

breakeven = 71,900 / 0.00494  =  14.6 M images/yr  ≈  40 k images/day

Two lines to check. The self-hosted marginal cost is the routed $14.66 per 1M restated per image: 14.66 / 1e6 = $0.0000147. And the H100 fixed cost assumes you rent it around the clock whether or not traffic arrives: $2.50 x 24 x 365 = $21,900. The final division gives 14.6 M images a year, and 14.6e6 / 365 ≈ 40,000 a day.

Below ~40k images/day, call the API and spend the engineer on something else. Being able to produce that line — rather than an opinion about self-hosting — is the difference between an answer and a preference.

Caching, and the one trap in it

Caching is the cheapest saving available, and it has exactly one way of failing silently.

Content-hash dedupe keys the cache on a hash of the exact bytes. It catches re-uploads and CDN variants — the same file served from a content delivery network under different URLs — and typically eliminates 15-30% of traffic for the cost of a hash. An exact-byte match is an exact-byte match, so there is nothing to get wrong.

Perceptual hashing (pHash) catches what exact hashing misses: re-encodes and crops. That is strictly more hits — and it is where the trap lives.

A pHash collision means captioning image A with image B’s description. Two visually similar but different images collide, the cache returns the wrong caption, and nothing anywhere raises an error. The output is a well-formed caption about a different photograph.

So a pHash hit must be confirmed by a cheap embedding-distance check before the cached caption is reused. Exact hash: reuse freely. Perceptual hash: verify, then reuse.

Assumptions in this stage.

• State out loud: batch 8 on the synchronous path and 64 on the batch path, an H100 at 300 TFLOP/s effective and $2.50/GPU-hour, a 25-token caption, a 40-token prompt, a 3 ms text detector, and 15-30% cache hits. Every figure in this section is a product of them, and none changes the shape of the answer.
• Ask: the daily image volume, because it is the only input to the build-versus-buy crossover and the answer flips at about 40k images a day.
• Load-bearing: that 8% of images contain text and that a 3 ms classifier can identify them. The 1.87x prefill saving, the 47.4 images per second, and the $14.66 per million all follow from that split; at 40% text-bearing traffic the hi-res lane dominates the fleet and the router stops being a saving.
• Also load-bearing: that the text-presence classifier’s mistakes are cheap. It is a router, not a gate, so a false negative sends a receipt down the 64-token lane and produces the confidently wrong number in Ocr failure — which means the classifier’s recall, not its precision, is the number to watch.

9. Failure modes

This system fails in four ways, and every one of them has the same shape: a missing evidence channel is filled in by the model’s prior, and the output is indistinguishable from a grounded one.

9.1 Object hallucination — the dominant one

Why does a captioner describe a refrigerator that is not there? The explanation yields two testable predictions — both check out below — and four fixes ordered by cost.

A logit is the raw score the model assigns a candidate next token before those scores are turned into probabilities. Loosely, the logit for token w at step t decomposes into two sources of conditioning:

logit(w)  ≈  f_lang(w | w_<t)  +  f_vis(w | image, w_<t)

This is a sketch, not an identity — the two channels are not literally separable inside the network. But it names the competition correctly. f_lang is the pull from “what word usually follows these words,” and f_vis is the pull from “what is actually in this picture.” The model emits whichever wins.

Three facts make f_lang win:

1. f_lang was trained on trillions of text tokens; f_vis reaches the decoder through a 21M-parameter projector trained on 1-100M pairs. The language prior is enormously better estimated than the visual channel.
2. The image contributes a fixed number of key/value entries (576, or 64) while the text prefix grows by one every decode step. Softmax — the function that converts a row of scores into probabilities summing to 1 — normalizes attention, so the share of attention mass available to image tokens falls monotonically as the caption lengthens. There is a fixed number of image entries competing against an ever-growing number of text entries for a fixed total of 1.
3. Cross-entropy never penalized a plausible-but-absent object more than an implausible one (The ml objective and the two ways it is structurally wrong). There was no gradient distinguishing them.

Fact 2 makes a falsifiable prediction: hallucination rate should rise with caption length. Not “might” — must, if the mechanism is right. Measure it and it does. Group captions by how long they came out, and read the two right-hand columns:

generated length      CHAIR_s     hallucinated objects per caption
 15-25 tokens          0.048                0.07
 40-60 tokens          0.091                0.19
 90-130 tokens         0.168                0.51

Take the midpoints, 20 tokens and 110 tokens. That is a 5.5x increase in output length. It buys a 3.5x increase in the hallucination rate (0.168 / 0.048) and a 7.3x increase in hallucinated objects per caption (0.51 / 0.07).

Which means “describe this image in detail and be thorough” is a hallucination prompt. People add it to improve alt text and make it measurably worse.

Fact 3 predicts something else checkable: the hallucinated objects should be the co-occurrence-likely ones, not random ones. Also true. Here is one image, the prompt, what the model said, and what an external detector found when asked about each named object:

IMAGE: a kitchen counter, a wooden cutting board, a chef's knife,
       three tomatoes. No refrigerator visible. No stove visible.

PROMPT: "Describe this image in detail."

OUTPUT: "A modern kitchen with light wood cabinets. On the counter sits a
         wooden cutting board with three ripe tomatoes and a chef's knife.
         A stainless steel refrigerator stands against the far wall, and a
         gas stove is visible to the left."

open-vocabulary detector pass, threshold 0.35:
   cutting board    FOUND       0.91
   knife            FOUND       0.88
   tomato x3        FOUND       0.93 / 0.90 / 0.87
   refrigerator     NOT FOUND   max 0.04
   stove            NOT FOUND   max 0.06

CHAIR_i = 2 hallucinated mentions / 7 object mentions  =  0.286
          three tomatoes are three mentions; "kitchen", "counter" and
          "cabinets" are scenes and surfaces, not annotated objects
CHAIR_s = 1 caption with a hallucination / 1 caption    =  1.000

That is one caption, so CHAIR_i is the informative number here and CHAIR_s is degenerate — the reverse of every table in this chapter, which averages over a whole evaluation set (Chair_i and chair_s are two different numbers).

Why refrigerator and stove specifically: in the caption corpus, P("refrigerator" | "kitchen") is high. The model is not guessing randomly — it is sampling from the conditional distribution of kitchen captions, and that distribution contains appliances. The model is doing exactly what it was trained to do. The objective is what is wrong.

Four fixes follow from that mechanism, ordered from free to expensive:

a) Shorten the target and permit abstention. Free, and by the table above worth ~3x. Do this before anything else.

b) Visual contrastive decoding. Run a second forward pass on a blanked image and subtract. a is a strength knob, typically around 0.5:

logit_final(w)  =  (1 + a) · logit(w | image)  -  a · logit(w | blank)

The second term is the language prior — measured, not assumed. Subtracting it removes exactly the probability mass that had no visual support. Nominally 2x prefill, but the blank-image prefix is constant across every request, so its KV cache is computed once and reused forever (The kv cache the most important mechanism in this chapter). Real cost: one extra decode stream, ~120 ms.

c) Detector-gated verification. Chunk noun phrases out of the caption — a noun phrase being a noun plus the words that modify it, such as “a wooden cutting board” — run an open-vocabulary detector on each, and strip or hedge the unsupported ones. About 40 ms, and it hands you production CHAIR for free.

The gate has to check phrases, not tokens. Splitting on whitespace and querying a detector with every word asks it about with, sits and the, which score zero and are duly reported as hallucinations — on the caption above that flags 33 of 37 terms and hedges 70% of the caption, including “three ripe tomatoes.” A gate that hedges everything is not conservative, it is off: the reader learns to ignore the marker, and the two genuinely invented objects are hidden inside the noise.

The implementation below finds noun phrases without a POS tagger — a part-of-speech tagger, the model that labels each word as noun, verb or preposition — by using the closed-class function words as boundaries, since a noun phrase never contains one.

Four things to look at as you read it, in order:

1. Three word lists. FUNCTION_WORDS are the phrase boundaries. QUANTIFIERS stay in the displayed phrase but are dropped from the detector query, because no detector has an opinion about “three.” NOUN_STOP are heads like “background” and “scene” that no detector can draw a box around, so checking them would only manufacture fake hallucinations.
2. noun_phrases slices the caption into maximal runs of content words. That is the whole chunking rule.
3. grounded_spans queries the detector twice per phrase — the full phrase and then just its head noun — and keeps the phrase if either clears the threshold. Detectors are stronger on bare heads than on long modifier stacks.
4. The assertions at the bottom are the claim: on the kitchen caption, exactly two spans get hedged, and “three ripe tomatoes” survives.

import re

# Closed-class words. A noun phrase never contains one, so they are the
# boundaries a chunker can find without a POS tagger.
FUNCTION_WORDS = {
    "a", "an", "the", "this", "that", "these", "those", "some", "any", "each",
    "every", "several", "its", "his", "her", "their", "our", "your", "my",
    "of", "on", "in", "at", "by", "for", "with", "from", "to", "into", "onto",
    "over", "under", "above", "below", "near", "beside", "behind", "against",
    "across", "around", "through", "between", "beneath", "atop", "and", "or",
    "but", "while", "as", "there",
    "is", "are", "was", "were", "be", "been", "being", "has", "have", "had",
    "sits", "sit", "stands", "stand", "lies", "lie", "rests", "rest",
    "holds", "hold", "hangs", "hang", "sitting", "standing", "lying",
    "holding", "hanging", "wearing", "placed", "seen", "shows", "showing",
    "appears", "visible",
}

# Kept in the phrase text, dropped from the detector query: no open-vocabulary
# detector has an opinion about "three". Counting is §9.4's problem.
QUANTIFIERS = {"one", "two", "three", "four", "five", "six", "seven", "eight",
               "nine", "ten", "dozen", "pair", "couple"}

# Heads that name a scene, a region or a property rather than an object a
# detector can box. CHAIR is defined over an object vocabulary; checking these
# would only manufacture hallucinations.
NOUN_STOP = {"image", "photo", "picture", "view", "background", "foreground",
             "scene", "left", "right", "side", "wall", "color", "colour",
             "distance", "middle", "top", "bottom", "front", "back"}

TOKEN = re.compile(r"[A-Za-z]+(?:'[A-Za-z]+)?|\d+")


def singular(word: str) -> str:
    """Crude, deterministic de-pluralization — detector prompts are singular."""
    if word.endswith("ies") and len(word) > 4:
        return word[:-3] + "y"
    if word.endswith(("ses", "xes", "zes", "ches", "shes", "oes")):
        return word[:-2]
    if word.endswith("s") and not word.endswith(("ss", "us", "is")):
        return word[:-1]
    return word


def noun_phrases(caption: str):
    """Chunk a caption into (phrase, start, end) noun phrases.

    The rule is one line: a noun phrase is a maximal run of content words,
    bounded by a function word or by punctuation. No tagger, no model, no
    dependency — and the failure mode is bounded, because an unlisted verb
    only ever *merges* two phrases into one the detector then scores low.
    It fails toward hedging, never toward passing something unchecked. Swap
    in a real tagger's NP chunks when you have one; what matters here is that
    the unit sent to the detector is a phrase.
    """
    out, run, prev_end = [], [], None
    for m in TOKEN.finditer(caption):
        word = m.group(0).lower()
        punctuated = prev_end is not None and caption[prev_end:m.start()].strip()
        if word in FUNCTION_WORDS or punctuated:
            if run:
                out.append((caption[run[0].start():run[-1].end()],
                            run[0].start(), run[-1].end()))
            run = []
        if word not in FUNCTION_WORDS:
            run.append(m)
        prev_end = m.end()
    if run:
        out.append((caption[run[0].start():run[-1].end()],
                    run[0].start(), run[-1].end()))
    return out


def grounded_spans(caption: str, detect, threshold: float = 0.35):
    """Return (kept_caption, unsupported) after checking each noun phrase.

    `detect(term) -> float` is an open-vocabulary detector's max box score
    for that term on the image. Nothing here is the model's opinion of
    itself: every check is an external measurement.

    Each phrase is queried twice — whole, then by its head noun — and kept if
    either is supported, because detectors are stronger on heads than on long
    modifier stacks. An unsupported modifier over a supported head ("stainless
    steel" on a real refrigerator) is the attribute gate's job (§9.2).
    """
    kept, unsupported = caption, []
    for phrase, start, end in reversed(noun_phrases(caption)):
        words = [w.lower() for w in phrase.split()]
        head = singular(words[-1])
        if head in NOUN_STOP:
            continue
        query = " ".join(w for w in words if w not in QUANTIFIERS)
        if max(detect(query), detect(head)) >= threshold:
            continue
        unsupported.append(phrase)
        kept = kept[:start] + "[unverified]" + kept[end:]
    return kept, sorted(unsupported)


# --- the chapter's own caption, the chapter's own detector scores -----------

CAPTION = ("A modern kitchen with light wood cabinets. On the counter sits a "
           "wooden cutting board with three ripe tomatoes and a chef's knife. "
           "A stainless steel refrigerator stands against the far wall, and a "
           "gas stove is visible to the left.")

SCORES = {"wooden cutting board": 0.89, "cutting board": 0.91, "board": 0.72,
          "chef's knife": 0.86, "knife": 0.88,
          "ripe tomatoes": 0.90, "tomato": 0.93,
          "modern kitchen": 0.55, "kitchen": 0.62,
          "light wood cabinets": 0.64, "cabinet": 0.71, "counter": 0.86,
          "stainless steel refrigerator": 0.03, "refrigerator": 0.04,
          "gas stove": 0.05, "stove": 0.06}

phrases = [p for p, _, _ in noun_phrases(CAPTION)]
assert phrases == ["modern kitchen", "light wood cabinets", "counter",
                   "wooden cutting board", "three ripe tomatoes",
                   "chef's knife", "stainless steel refrigerator", "far wall",
                   "gas stove", "left"]

# no fragment reaches the detector: every token is a whole word of the caption
assert all(re.search(rf"\b{re.escape(w)}\b", CAPTION.lower())
           for p in phrases for w in p.lower().split())

kept, unsupported = grounded_spans(CAPTION, lambda t: SCORES.get(t, 0.0))
assert unsupported == ["gas stove", "stainless steel refrigerator"]
assert "three ripe tomatoes" in kept          # the grounded case stays grounded
assert kept.count("[unverified]") == 2        # two hedges, not twenty-nine


def contrastive_logits(with_image, blank_image, alpha: float = 0.5):
    """Subtract the measured language prior from the image-conditioned logits.

    `with_image` and `blank_image` are equal-length logit sequences for the
    same step. The blank-image pass has a constant prefix, so its KV cache
    is built once at startup and reused for every request.
    """
    return [(1.0 + alpha) * a - alpha * b
            for a, b in zip(with_image, blank_image)]

What comes back is the caption with exactly the two invented objects hedged:

A modern kitchen with light wood cabinets. On the counter sits a wooden
cutting board with three ripe tomatoes and a chef's knife. A [unverified]
stands against the far wall, and a [unverified] is visible to the left.

Two detector queries per phrase — ten phrases, of which two are dropped as scene words before any query — is where the ~40 ms goes. Replace noun_phrases with the per-token version and the last assertion fails with 30 hedges instead of 2, three, ripe and tomatoes among them.

d) DPO on auto-labeled pairs (Training). Highest quality, needs a training run, and the detector from (c) is already your label function.

9.2 Demographic assumption from context

The next failure is a fairness incident rather than a quality miss, and the two controls for it are both code rather than prompt text.

The demonstration below samples the same image five times and lists what changed between the samples. The image contains no evidence for occupation, seniority or gender — so anything the model asserts about those came from somewhere other than the pixels.

IMAGE: a person in blue scrubs holding a clipboard in a hospital corridor.

5 samples at T = 1.0:
  1. "A nurse reviews a patient chart in a hospital hallway."
  2. "A female nurse holding a clipboard in a hospital."
  3. "A young nurse checking notes."
  4. "A doctor reviewing a chart."          <- same image, different role
  5. "A nurse in scrubs walking down a corridor."

role agreement:   nurse 4/5, doctor 1/5
age asserted:     "young" in 1/5, unsupported by any pixel
gender asserted:  "female" in 1/5, unsupported by any pixel

Nothing in the pixels indicates occupation, seniority, or role. The model is completing from P(role word | scrubs, appearance) learned from web captions where those correlations are strong. The output is not a description; it is a demographic prior rendered in declarative grammar — and when it happens to be right, it is right because it encoded a stereotype, which is the same mechanism.

Two controls, and note that both are code:

• An attribute lexicon gate. Role, gender, age, and ethnicity terms are checked against visual support; unsupported ones are rewritten to the supported form (“a person in scrubs”). This must be enforced in the harness, not requested in the prompt — a prompt reduces the rate and does not eliminate it, and the residue is a fairness incident rather than a quality miss (ch 07).
• Sample disagreement as the detector. Any attribute term that does not appear in all 3 samples is unsupported. Costs 3x decode on a 25-token output — genuinely cheap — and needs no labels. Self-consistency across samples is a real uncertainty signal in a way that asking the model for a confidence score is not (case study 06).

9.3 OCR failure

Here is a failure where a bigger model provably cannot help, because the information was destroyed before the model ran.

The block below traces one receipt from camera resolution down to what the encoder actually sees. Follow the pixel counts: the point is that by the time the model runs, the digits no longer exist in the input.

IMAGE: restaurant receipt photographed at 3024 x 4032
resize to 336:  downsample factor 9.0x
   10-point text, ~32 px cap height  ->  3.6 px
   one 14 x 14 patch covers 126 x 126 original px  =  about 4 lines of text
   that patch becomes ONE 1024-d vector

model output:  "A receipt from a restaurant showing a total of $12.50."
ground truth:  total $47.83

The number is not misread — it is generated from the prior over receipt totals, because glyph identity was destroyed by the resize before the model ever ran. Note the shape: not “unreadable,” but a confident, plausible, wrong number. That is the general form of every failure in this section — a missing evidence channel is filled by the prior, and the output is syntactically indistinguishable from a grounded one.

The fix is dynamic high-resolution tiling — cutting the image into several native-resolution crops and encoding each separately, plus a downscaled thumbnail for global layout — routed by a cheap text-presence detector (Serving scale and cost). Five times the visual tokens, on 8% of traffic.

9.4 Counting

Counting is the failure whose honest fix is to stop asking the captioner.

Two mechanisms, and the second is the one people miss.

1. The encoder pools. Self-attention gives you set membership far more readily than cardinality; there is no architectural component that counts.
2. The caption corpus almost never enumerates. Humans write “several,” “a group of,” “a handful.” So even a perfect visual representation has no supervision mapping it to “seven.”

Measure it by asking for a count on images with a known number of objects, and score exact matches. Read across the row and watch where it falls off:

true count     1      2      3      4      5      6      7     8+
exact-match  1.00   0.95   0.81   0.52   0.31   0.19   0.12  0.07

Accuracy collapses past 4 — which is roughly the human subitizing limit, subitizing being the ability to see how many objects there are instantly without counting them one by one, which for people runs out at about four. That is not a coincidence: the training captions were written by humans who also stopped counting at four and switched to “several.” The honest fix is not to ask the captioner. Route counting queries to a detector and count boxes. “Use the right tool for the sub-task” is a stronger answer than “fine-tune harder,” and it is also correct.

Summary

One table collects all eight failures, each with its mechanism, the way you would detect it, and the control that stops it.

Assumptions in this stage.

• State out loud: the CHAIR-versus-length curve (0.048 at 20 tokens rising to 0.168 at 110), the 0.35 detector threshold, a contrastive-decoding strength of about 0.5, three samples for the agreement check, and a 25-token caption target. Each is a tuned operating point rather than a structural claim.
• Ask: whether the product may show a hedge marker such as [unverified] to users at all, because the entire detector gate degrades to “silently drop the span” if not, and dropping changes the meaning of a sentence in ways hedging does not.
• Load-bearing: that the blank-image prefix is constant across every request, so its KV cache is computed once at startup and reused forever. That is the only reason contrastive decoding costs one extra decode stream instead of doubling prefill; if the blank pass had to be recomputed per request, the technique roughly doubles the largest line in the latency budget and stops being affordable on the synchronous path.
• Also load-bearing: that self-consistency across samples is a real uncertainty signal while a self-reported confidence score is not. Both fairness controls in Demographic assumption from context rest on it; if a model could reliably report its own confidence, the 3x decode cost buys nothing and one extra output field replaces it.

10. Alternatives considered and rejected

Each design below was seriously considered, and a number killed it — rejecting on a number rather than a preference is what the round is scoring.

11. Interviewer pushback

These are the ten questions this design actually gets asked, what each one is testing, and an answer that survives the follow-up. Almost every one of them is an attack on an assumption rather than on a fact.

“Just build a captioning model. Why are you asking what it’s for?” Testing: whether framing is a habit or a slogan. Because alt text and search indexing want opposite things and I would build different systems for them. Alt text is one short sentence where a false statement is the fatal error and it cannot be A/B tested — so it gets a conservative decode config, an abstention path, and an offline human gate. Search wants 100 entity-dense words where the fatal error is omission, and it reads out in a couple of hours of live traffic. Same encoder, same decoder, different decode configs, different metrics, different launch criteria. If I picked one silently I would be optimizing the wrong error asymmetry for half the traffic.

“Why project image features into the LLM’s token space instead of cross-attention?” Testing: whether you can derive the design or only name it. Because the decoder’s first operation on a token id is an embedding lookup returning a vector in R^4096, and nothing downstream knows where that vector came from. So any module emitting R^4096 vectors is a legal token source, which means the LLM needs zero modification — the KV cache, prompt caching, batching, and structured output all keep working on an unmodified serving stack. And it is why the LLM can stay frozen: the knowledge about kitchens is already in the weights, so training only has to learn the translation, which a 21M-param MLP does. That is 0.26% of the parameters. Cross-attention wins when you have many images or video, where the context cost of 576 tokens per image becomes the binding constraint.

“Your CIDEr went up 12%. Ship it?” Testing: whether you take a good number at face value. No, not on that number. CIDEr is n-gram overlap against five references, and for an image with thousands of valid captions that mostly measures stylistic conformity to the reference set. It systematically prefers the generic caption — which is the exact failure I am trying to fix. I have seen the pairing where the CIDEr winner hallucinates at 0.181 CHAIR_s and 71% human correctness while the loser hallucinates at 0.066 and 89% — and I would say the subscript out loud, because CHAIR_s is the fraction of captions carrying a hallucination and CHAIR_i is the fraction of mentions, and on one eval set they can differ by 2x. I gate on CHAIR and human correctness, keep CIDEr as a regression tripwire, and check CLIPScore knowing it cannot distinguish “a dog chasing a man” from “a man chasing a dog” — two points apart.

“Why does it describe a refrigerator that isn’t there?” Testing: mechanism, not vocabulary. This is the question. Three things stack. The language prior was trained on trillions of tokens; the visual channel reaches the decoder through a 21M-param projector trained on a hundred million pairs, so it is far worse estimated. The image contributes a fixed 576 key/value entries while the text prefix grows every step, and softmax normalizes, so the attention share going to pixels falls monotonically as the caption lengthens. And cross-entropy never penalized a plausible-but-absent object more than an implausible one, so there was never a gradient distinguishing them. That predicts two things I can check: hallucination rises with length — CHAIR_s goes 0.048 to 0.168 from 20 to 110 tokens — and the hallucinated objects are the co-occurrence-likely ones. Both hold. So “describe this in detail” is literally a hallucination prompt.

“Fix it without retraining.” Testing: whether the mechanism generates the fix. Shorten the target and allow abstention — free, worth about 3x by that same length curve. Then visual contrastive decoding: run a second pass on a blank image and compute (1+a)·logit_image - a·logit_blank. That second term is the language prior, measured rather than assumed, and subtracting it removes exactly the mass with no visual support. It looks like 2x prefill but the blank prefix is constant across every request, so its KV cache is built once at startup. Then a noun-phrase gate against an open-vocabulary detector at ~40 ms, which also gives me production CHAIR for free. Retraining with DPO is better, and the detector is already the label function when I get to it.

“How many GPUs to caption 10 million images a day?” Testing: whether you can size, and whether you know where the cost sits. At 576 visual tokens I get about 27.8 images/s per H100 — encoder is 81 ms per batch of 64, LLM prefill is 2,103 ms, decode is 120 ms. Prefill is 91% of that and 94% of prefill is image tokens. So the lever is visual tokens: a 64-token resampler takes me to 115 images/s. Halving the decoder would actually save more prefill per unit ratio — it takes decode with it — but I can cut tokens 9x for 0.012 of CHAIR_s and I cannot cut the decoder 9x for anything. 10M/day is 116 images/s, so just over one GPU on the cheap lane and 2.5 on the routed path — call it three or four with headroom — that lane is 19x slower per image, so it eats 62% of the GPU time and the routed rate is 47 images/s, about $14.66 per million against $4,950 on a frontier API. But I would compute the crossover before recommending either — with a dedicated H100 at $2.50/hr and 20% of an engineer, breakeven is about 40k images/day.

“OCR is bad. Bigger model?” Testing: whether you diagnose before you spend. Bigger model does nothing, because the information is destroyed before the model runs. A receipt at 3024px resized to 336px is a 9x downsample; 10-point text drops from 32 pixels to 3.6, and one 14x14 patch covers four lines of text and becomes a single 1024-d vector. There is no glyph identity in the encoder output to decode. The fix is resolution — dynamic tiling into four native-res crops plus a thumbnail, which is 5x the visual tokens. I route it with a 3 ms text-presence detector so only ~8% of traffic pays. That is 1.9x cheaper on prefill than a flat 576-token path, 1.7x end to end, and roughly twice as good on OCR for the images that have text, because the 8% that needed pixels got more than the flat path ever gave them. I would not claim it is better across the board — blend the quality columns on the same 92/8 split and general captioning goes from 0.066 to 0.0766 CHAIR_s and counting from 0.58 to 0.439, because the 92% is riding the 64-token lane and the router selected for text rather than for counting. It is a cost-and-OCR win with a named price, not a free lunch.

“It said ‘a nurse’ and she was a surgeon.” Testing: whether you treat fairness as a control or as a prompt. Nothing in the pixels says occupation. The model is sampling from P(role | scrubs, appearance) learned off web captions, so the output is a demographic prior in declarative grammar — and when it is right it is right for the same reason. Two controls, both in code. Sample three times: any attribute term that does not appear in all three is unsupported, and on a 25-token output that costs almost nothing. Then a lexicon gate over role, gender, age, and ethnicity terms that rewrites unsupported ones to the visually supported form. I would not do this in the prompt, because a prompt reduces the rate and leaves a residue, and the residue here is a fairness incident rather than a quality miss.

“How do you launch alt text with no online metric?” Testing: whether you can ship without a dashboard telling you to. Accept that it does not A/B and move the decision offline: 2,000 stratified images with five references and per-object presence annotations, gated on human correctness and CHAIR. Then get a real online signal by making it editable — author edit rate with edit distance is direct human judgement at zero labeling cost, and the distance matters more than the rate, since 12% at 30 characters means rewriting and 12% at 3 characters means punctuation. Keep the report-this-description rate as a guardrail only; the base rate is around 0.02% so it will never decide anything. And run the detector on live captions so CHAIR is a continuous monitor, not a launch-day number.

“Where would you spend the next $50k?” Testing: whether your cost model produces a decision. Recaptioning. 100M images with a strong teacher is about 3,500 GPU-hours, roughly $8.7k of compute and $25-40k all-in — one engineer-month — and it moves quality more than any architecture change I could make in that month. I would mix it 80/20 with the original alt text rather than replacing it, because the two sources are not nested: synthetic gives grounded descriptive structure, original alt text is the only place rare proper nouns live, and a purely synthetic student writes “a red sports car” where the alt text said the model and trim. And I would measure the teacher’s own CHAIR first, because a distilled student cannot beat its teacher on the axis being distilled — only on cost.

The assumption ledger

Every assumption the chapter has leaned on, collected in one place — so the design’s foundations can be stated in twenty seconds, along with what replaces the design when each one fails. Each sorts into one of three bins: state it (you are free to pick, and being wrong costs a re-derivation), ask it (the answer changes the architecture, so it is worth an interviewer’s time), and load-bearing (if it is wrong the design is not suboptimal, it is invalid) — the same three bins used in ch 01.

The sentence that makes this visible to an interviewer: “This design rests on three things. One, what the caption is for — alt text and search indexing want opposite outputs and opposite error asymmetries, and I cannot design either until someone answers. Two, that the vision encoder was trained against text, which is the only reason a 21-million-parameter bridge between a frozen encoder and a frozen 8-billion-parameter language model works at all. Three, that an open-vocabulary detector is accurate enough to act as a label function — because it is simultaneously my free preference labels, my serving-time gate, and my production hallucination metric, and if it is wrong all three are wrong together.”

Cheat sheet

The one-line answers to the questions this design is most often asked. Everything here is derived above; this is the recall test.

Next: 06 — Retrieval-Augmented Generation.