How to design image captioning

In this lesson, we’ll design an image captioning system end to end: one image goes in, optionally with a prompt, and a string of text comes out. By the end you’ll be able to pick the model, data, and metric from what the caption is for, explain why a 21M-parameter bridge between a frozen encoder and a frozen decoder is enough, and name the controls that catch the dominant failure. Nothing in that input/output spec says what the text is for, and that gap is where the whole design comes from. A caption written for a blind user’s screen reader and a caption written for a search index want opposite sentences about the same photo.

Three things follow:

1. “Caption” names an output format, not a goal. The purpose decides the model, the data, and the metric, so it has to be settled first.
2. The system is a frozen image encoder, a frozen language model, and a small trained bridge between them that is 0.26% of the parameters.
3. The dominant failure is a confident description of an object that is not in the photograph. It comes from the training objective, and specific controls catch it.

Terms used throughout

What is the caption for?

Two of the most common consumers want opposite things, and the divergence produces two systems with opposite error asymmetries. The “Fatal error” row is the one that propagates through the rest of the design: it decides which mistake you tune the system to avoid.

An A/B test splits live traffic between the current system and a new one and compares a metric. It can decide the search question because a search failure produces an observable event: the user reformulates the query. It cannot decide the alt-text question because nothing on the page signals “the description was true.”

Two more consumers show up often enough to name: moderation triage (recall of policy-relevant content at fixed precision, output is structured attributes not prose) and product catalog (attribute accuracy (material, color, sleeve length) where a wrong attribute is a return, not a bad sentence).

The four consumers lead to four different systems, each with its own data source, metric, and launch gate:

flowchart TD
    Q{"What is the caption FOR?"}
    Q -->|Alt text| A["Short · conservative<br/>abstain when unsure"]
    Q -->|Search index| B["Long · entity-dense<br/>recall over precision"]
    Q -->|Moderation| C["Structured attributes<br/>recall at fixed precision"]
    Q -->|Catalog| D["Attribute extraction<br/>not prose"]

    A --> A2["Data: human-verified<br/>Metric: human correctness + CHAIR<br/>Gate: offline only"]
    B --> B2["Data: recaptioned web scale<br/>Metric: search success rate<br/>Gate: online A/B"]
    C --> C2["Data: policy-labeled<br/>Metric: recall at 0.95 precision"]
    D --> D2["Data: catalog attributes<br/>Metric: per-attribute F1"]

    style Q fill:#1d3557,color:#fff
    style A2 fill:#2d6a4f,color:#fff
    style B2 fill:#bc6c25,color:#fff

CHAIR is the hallucination metric defined later; F1 is the harmonic mean of precision and recall; “recall at 0.95 precision” means of the policy-violating images, what fraction you catch while holding false alarms to 5% of what you flag.

One more input decision: alt text is context-dependent and captioning is not. The same product photo needs “a navy wool overcoat, three-quarter length” on a shopping page and “the coat mentioned in paragraph 3” on a news article. If accessibility is the goal, the surrounding page text is an input, and that decision doubles the prompt specification, so it has to be made before a model is picked.

The ML objective, and the two ways it is structurally wrong

The model learns a conditional language model over caption tokens. It predicts each next word given the image and the words so far:

P(caption | image) = prod_t P(w_t | w_<t, image)

loss = - sum_t log P(w_t* | w_<t*, image)        teacher forcing, cross-entropy

w_t is the caption’s word at position t, w_<t is every word before it, and the star marks the reference caption, the human-written one in the training data. Teacher forcing is the choice hiding in w_<t*: at every position the model is shown the true previous words, not its own, so each position is graded independently against one string. The loss is zero only when the model gave the reference word probability 1.0 at every position.

Two problems live in that expression, and both surface later as failure modes:

• Problem 1: one-to-many. An image has thousands of valid captions. Cross-entropy against one sampled reference penalizes every valid alternative, and minimizing it over a diverse reference distribution drives the model toward the mode, the single most likely caption. The mode of “captions humans write for this image” is the generic one. That is why captioners produce “a man riding a horse on a beach” instead of “a ranch hand in a red jacket on a chestnut quarter horse.” The blandness is what minimizes the loss, not a sign that the model lacks the capability.

• Problem 2: no truth term. The loss measures token mismatch. A hallucinated refrigerator in a kitchen image and a stylistic variation cost the same. Nothing in the objective distinguishes wrong from different.

So the missing terms have to be installed later, either by preference optimization on hallucination-labeled pairs (see Training) or by a verification pass at serving time (see Failure modes). Everything downstream exists because cross-entropy has no truth term.

Architecture

The model is three pieces: an image encoder, a language model, and a small bridge between them. The point worth deriving is why the bridge can be tiny while both ends stay frozen.

The encoder: CNN or ViT

A CNN (convolutional neural network) slides small learned filters across the image, so it assumes from the start that nearby pixels belong together and that an object means the same thing wherever it appears. A ViT (vision transformer) cuts the image into a grid of fixed-size square patches, treats each patch as a token, and runs the same attention machinery a language model uses over that sequence, with no built-in assumption about locality.

The decisive row is “output shape.” A ViT’s output is already the object a text decoder consumes: a sequence of vectors. That is what makes the prefix-projection join below possible at all.

A ViT-L/14 at 336px (“L” for large, “/14” for 14-pixel patches, on a 336-pixel-square image) produces (336/14)^2 = 576 patch tokens, plus one CLS token, an extra learned position holding a summary of the whole image, for 577 total. A forward pass costs about 382 GFLOP/image: roughly 350 GFLOP of matrix multiplies (each layer is 12·d^2 weights over L=24 layers at width d=1024, at 2 FLOP per weight per token) plus a 33 GFLOP attention term that grows with T^2. Attention is only ~9% here because at 577 tokens the quadratic term has not yet caught up with the linear one, which is why the hi-res path later, running the encoder five times, stays affordable. (Vision papers quote 191 “GFLOPs” for this config because they count multiply-accumulates, one multiply plus one add as a single op, the same quantity, halved.)

The join: cross-attention vs prefix projection

There are two ways to let a language model see an image:

• Cross-attention (Flamingo-style). Insert new layers into the decoder that attend from text positions out to the vision features. They are gated: each starts with a learned multiplier at zero, so the untouched language model is the starting point and image influence is dialed up during training.

• Prefix projection (LLaVA-style). A small MLP (multilayer perceptron, two matrix multiplications with a nonlinearity between them) maps each patch embedding from the vision width of 1,024 to the language width of 4,096, and the resulting vectors are spliced into the token sequence where token embeddings would otherwise sit.

Why the second one works: the LLM’s first operation on a token id is a lookup in an embedding table returning a vector in R^4096, and nothing downstream knows or cares where that vector came from. Any module that emits R^4096 vectors is a legal token source. So the projector’s whole job is to land image features in the region of R^4096 that the frozen decoder already reads as content words about a scene. That is why you can freeze the LLM: the knowledge about horses, beaches, and kitchens already exists in the frozen weights; training only has to learn the translation into a vocabulary the model already has.

The projector is two matrices, 1024→4096→4096, about 21M params. Against an 8B decoder that is 21e6 / 8e9 = 0.26%. You train 0.26% of the parameters and get most of the capability, which is why the alignment stage costs GPU-hours, not GPU-months. Freezing a component means holding its weights fixed and passing no gradient to them; here both ends are frozen and only the 21M-parameter bridge moves.

Three terms for the comparison. The KV cache is the store of already-computed key and value vectors that lets a decoder avoid re-reading the whole prompt on every token. Prompt caching is the provider-side reuse of that store for a prefix it has seen before, billed at a fraction of the normal rate. Structured output means forcing the text to conform to a declared shape such as a JSON object.

For one image and one caption, prefix projection wins on everything operational. Cross-attention only pays off when many images per sample make the 576-token context cost the binding constraint.

flowchart LR
    IMG(["Image<br/>336 x 336"]) --> P["Patchify 14x14<br/>576 patches"]
    P --> V["ViT-L encoder<br/>302M params · frozen<br/>382 GFLOP"]
    V --> R["Token reducer<br/>pool or resampler<br/>576 -> 64"]
    R --> PROJ["Projector MLP<br/>1024 -> 4096 -> 4096<br/>21M params · TRAINED"]
    PROJ --> SEQ["Token sequence<br/>64 image vectors<br/>+ prompt tokens"]
    TXT(["Prompt"]) --> SEQ
    SEQ --> LLM["Decoder LLM · 8B<br/>FROZEN in stage 2"]
    LLM --> OUT(["Caption"])

    style PROJ fill:#2d6a4f,color:#fff
    style LLM fill:#1d3557,color:#fff
    style R fill:#bc6c25,color:#fff

The token reducer (a pool or a resampler, both defined below) compresses 576 patch vectors down to 64 before the projector maps them into the LLM’s space. Only the projector is trained in stage 2. That is the design.

Contrastive pretraining, and why one shared space buys zero-shot

The way the encoder was trained is what makes the 21M-parameter bridge sufficient. CLIP (contrastive language-image pretraining) trains an image encoder and a text encoder so they land in the same space:

1. Take a batch of N image-text pairs.
2. Encode both sides: N image vectors and N text vectors.
3. Compute similarity of every image against every caption, an N x N grid.
4. Treat the grid as an N-way multiple-choice question run both ways: given this image, which caption is its own, and given this caption, which image.

L_i2t = - (1/N) sum_k  log[ exp(cos(i_k, t_k)/tau) / sum_j exp(cos(i_k, t_j)/tau) ]
L     = (L_i2t + L_t2i) / 2

The fraction is a softmax: numerator is image k’s similarity to its own caption, denominator sums over every caption in the batch, so it is the probability assigned to the right answer, and the negative log is zero when the model is certain and correct. cos is cosine similarity, tau a learned temperature, and the two directions are averaged.

The negatives are the other items in the batch, so batch size is a hyperparameter of the loss, not just of the optimizer. N = 32,768 gives 32,767 negatives per positive; N = 256 gives 255. The task is an (N-1)-way discrimination, so the gradient signal scales with N, which is why contrastive training uses enormous batches, and why gradient accumulation (summing gradients over small batches before one update) is not a substitute: it buys more steps of an easy problem, not one step of a hard one.

The payoff is zero-shot classification, classifying into categories the model was never trained on. Because CLIP’s objective is cross-modal, what survives in the embedding is exactly what an image and its caption share. Classification collapses into retrieval: embed "a photo of a {class}" for every candidate class, embed the image, and take the highest cosine similarity. No classifier head, no labels, classes changeable at runtime.

Two consequences for the captioner:

1. A CLIP-pretrained encoder’s outputs are already text-shaped, so the projector’s job shrinks from “learn a modality translation” to “learn a near-linear remap.” That is why a frozen encoder + 21M-param MLP + frozen LLM converges on a few hundred thousand pairs instead of a few hundred million.
2. CLIPScore comes for free as a reference-free evaluation metric (it scores a caption against the image itself, not against a human answer key), and it inherits every weakness of the space it measures in.

Data and labels

Hundreds of millions of image-caption pairs have to come from somewhere, filtering throws most of them away, and one purchase buys more quality per dollar than any architecture change.

Web alt text is the only source at scale, and it is mostly garbage

Three terms for the funnel. A perceptual hash is a fingerprint of an image’s visual content, so a re-encoded or lightly cropped copy produces a nearly identical fingerprint. NSFW means not safe for work. PII is personally identifiable information: names, faces, addresses, plates.

raw crawled image + alt-text pairs                  ~5,000 M
drop empty / under 5 chars / filename-shaped        -2,200 M   ->  2,800 M
dedupe: perceptual hash on image, exact on text     -  900 M   ->  1,900 M
drop non-target-language and gibberish              -  400 M   ->  1,500 M
CLIP similarity filter, cos(img, txt) >= 0.28       -1,100 M   ->    400 M
NSFW / face / PII / watermark filters               -   40 M   ->    360 M
                                                                 -------
                                                    survival rate  7.2 %

The CLIP similarity filter is the sharpest cut and by far the most dangerous. Sharpest: it discards 73% of what reaches it (1,100 of 1,500 M), where the earlier filters only removed obvious junk nobody argues about. Most dangerous: you are filtering with a model trained on the data you are filtering. It keeps what the current model already understands and discards what it does not, silently narrowing the distribution toward the model’s existing competence, and the discarded pairs leave no trace. Two mitigations: filter with a model trained on a different, cleaner source, and always retain an unfiltered random sample so you can measure what you threw away.

Synthetic recaptioning, and what it costs

Surviving alt text is still short and non-descriptive: blue dress for a photo of a woman in a blue dress on a beach at sunset. Recaptioning runs a strong VLM, the teacher, over your images to write dense descriptions, which become the training targets for your smaller model, the student. This is distillation: training a small model to imitate a large one.

Order of magnitude: 100M images through a 70B teacher at ~8 img/s/GPU is about 3,500 GPU-hours, roughly $8.7k of compute at $2.50/GPU-hr but $25-40k all-in once egress, storage, orchestration, and reruns are counted. That is about one engineer-month, and it moves quality more than any architecture change made in that month.

The quality/quantity tradeoff is not monotone, because the two sources are not nested. Original alt text is the only place rare proper nouns live: a purely-synthetic student writes “a red sports car” where the alt text said “2019 Alfa Romeo Giulia Quadrifoglio.” Synthetic is the only place full-sentence grounded description lives. Mixing is a union of two non-overlapping capabilities, not a compromise between two grades of the same thing.

One hard ceiling: a student distilled from teacher captions cannot beat the teacher on the axis being distilled, only on cost. If the teacher hallucinates appliances in kitchens, the student now does too, fluently. Measure the teacher’s CHAIR before spending the $30k.

The eval set is where the human labels go

You do not need human labels to train; you need them to decide. Buy 2,000 images, stratified by domain (deliberately sampled so each category is represented) across people, documents, products, scenes, charts, and screenshots. Each gets 5 independent reference captions and a per-object presence annotation, a list of which objects are actually in the image, so that CHAIR is computable. Budget around $15k and treat it as infrastructure.

Training

Training runs in four stages. Stage 1 is the only one that costs a cluster, and the only one you can skip by downloading someone else’s checkpoint. Everything after it is small because of what is not in the Trainable column.

InfoNCE (information noise-contrastive estimation) is the contrastive loss written out above; CE is cross-entropy. LoRA (low-rank adaptation) freezes the original weights and trains a small pair of thin add-on matrices, so a fraction of a percent of the parameters carries the update and one base model can host many adapters. DPO (direct preference optimization) trains on pairs where something declared A better than B, not on a single correct answer, exactly the shape you need when the question is “which of these two captions is more truthful,” not “what is the right caption.”

Why stage 2 freezes the decoder: if you unfreeze an 8B decoder while the projector still emits noise, its gradient is dominated by adapting to that noise. It degrades its own language ability to accommodate a bad image representation, and you do not get it back. Frozen, the only path to lower loss runs through the 21M parameters that actually need to learn.

Stage 3 uses LoRA for a memory argument, not a quality one. Training a parameter holds four things in GPU memory: the weight, its gradient, and the optimizer’s two running statistics. In bf16 a weight or gradient is 2 bytes; Adam keeps two 32-bit moments, 8 bytes for the pair. For 8B parameters that is 16 GB weights + 16 GB gradients + 64 GB Adam moments = 96 GB before a single activation, which does not fit an 80 GB H100, the optimizer state alone is 4x the weights. LoRA at rank 16 trains ~20M parameters instead, so the moments are ~160 MB, the frozen base still costs its 16 GB, and the whole job fits on one card with room for activations and a real batch. That is the difference between iterating in an afternoon and iterating in a week.

Why stage 4 exists: cross-entropy has no truth term. Running DPO on pairs of (accurate caption, hallucinated caption) for the same image installs the missing asymmetry directly. Building the pairs is fully automatic, which is why it is affordable. An open-vocabulary detector, an object detector you can query with any noun in plain English instead of a fixed class list, returns a confidence and a box:

for each image:
    sample 4 captions at T = 1.0
    run an open-vocabulary detector over the image
    label each caption: clean if every named object is detected above 0.35
    emit (clean, hallucinated) pairs where both exist

Sampling at T = 1.0 (drawing each word from the model’s own distribution untouched) produces four genuinely different captions, not four near-copies. The detector is the label function; no annotators. This same detector is also the serving-time gate below, so its accuracy is load-bearing twice.

Offline metrics, and why they all disagree with humans

Every automatic caption metric ranks models differently from humans. Two mechanisms account for the whole disagreement.

The first five metrics are reference-based. They compare your caption against human answer keys, usually five per image:

• BLEU (bilingual evaluation understudy) counts shared runs of consecutive words (n-grams).
• METEOR does the same with a looser matcher that accepts stems and synonyms.
• ROUGE-L scores the longest sequence of words in both, in order but not necessarily adjacent.
• CIDEr weights n-grams by TF-IDF (term frequency times inverse document frequency), raising phrases common in this image’s references and rare across the corpus, so generic phrases count for little.
• SPICE parses both into a scene graph ((object, attribute) and (object, relation, object) tuples) and scores tuple overlap with F1.

The last three need no answer key:

• CLIPScore compares the caption to the image directly in CLIP’s shared space.
• CHAIR (caption hallucination assessment with image relevance) counts objects the caption named that a detector cannot find, over the annotated object vocabulary only.

Mechanism 1 is that reference-based metrics measure agreement with a sample, not correctness. Five references cover a vanishing fraction of the thousands of valid captions, so n-gram overlap is largely measuring stylistic conformity. A metric that rewards matching the reference rewards the generic caption, which is exactly the one-to-many failure mode. Optimizing CIDEr actively pushes the model toward the thing you were trying to fix.

Mechanism 2 is that reference-free metrics measure similarity, not entailment. CLIPScore asks “is this text plausibly about this image,” which is invariant to the differences that decide whether a caption is true:

image: a dog chasing a man across a lawn

A  "a dog chasing a man across a lawn"      CLIPScore 0.79
B  "a man chasing a dog across a lawn"      CLIPScore 0.77
C  "a dog, a man, and a lawn"               CLIPScore 0.75

Two points separate a correct caption from its exact inverse. A bag-of-concepts representation, one that records which things are present and discards the relations between them, is enough to win the contrastive task on 400M web pairs, so nothing in CLIP’s objective ever required the text encoder to represent who did what to whom. The metric cannot see a fact and its reversal as different, because the space it lives in cannot.

CHAIR_i and CHAIR_s are two different numbers

They share a name and a detector pass and they are not interchangeable. CHAIR_i divides by object mentions; CHAIR_s divides by captions. One caption that invents five objects moves CHAIR_i five times as much as one that invents one, and moves CHAIR_s identically. Quote a CHAIR number without the subscript and nobody can reproduce it:

def chair(captions):
    """Both CHAIR numbers from one detector pass over the same annotations.
    captions: (object mentions, hallucinated mentions) per caption, counted
    over the annotated object vocabulary only."""
    return {
        "CHAIR_i": sum(h for _, h in captions) / sum(m for m, _ in captions),
        "CHAIR_s": sum(1 for _, h in captions if h) / len(captions),
    }

# a five-caption toy set: same detector, same annotations, one factor of two
TOY = [(7, 2), (6, 3), (4, 0), (4, 0), (4, 0)]
both = chair(TOY)
assert round(both["CHAIR_i"], 3) == 0.200            # 5 bad mentions of 25
assert round(both["CHAIR_s"], 3) == 0.400            # 2 bad captions of 5
assert both["CHAIR_s"] == 2 * both["CHAIR_i"]

Two bad captions of five is 0.400; five bad mentions of twenty-five is 0.200, on identical data. Which one you report decides whether a model that concentrates its hallucinations into a few very bad captions looks better or worse than one that spreads them thinly, genuinely different products. Report both, or report CHAIR_s and say so.

The disagreement, on two real candidates

model   CIDEr   SPICE   CLIPScore   CHAIR_s    human       human
                                    (lower=      correct   "useful as alt text"
                                     better)
  A      1.21   0.211     0.762      0.181       71 %          44 %
  B      1.08   0.226     0.781      0.066       89 %          78 %

Model A learned the reference style. Model B is more specific and hallucinates a third as often. CIDEr picks A; correctness, CLIPScore, and CHAIR_s all pick B. Gate on the easiest metric to compute and you ship the worse model. Gate on CHAIR plus human correctness, and keep CIDEr only as a regression tripwire.

Human eval, on separated axes

The most common mistake is asking raters to “rate this caption 1-5.” One number blends four independent properties, two of which move in opposite directions.

Correctness and specificity are a dial, not a bug. A captioner tuned to say less is more correct and less useful. Report the pair, or report correctness at a fixed specificity band, the same discipline as reporting precision at a fixed recall (see the evaluation-metrics lesson).

Online metrics and A/B

Offline metrics gate the launch; live traffic is the other half. The two objectives differ not only in metric but in whether an online experiment can decide them at all.

Search indexing A/Bs cleanly. The primary metric is search success rate: the user issues a query, clicks a result, and does not reformulate within 30 seconds. The guardrail, a metric that must not move, is CTR (click-through rate) on results that previously had no caption text, so you can detect having added noise instead of signal.

Sizing, order of magnitude: detecting a 1-point absolute move on a 62% baseline at alpha 0.05 (accepted false-positive rate) and power 0.8 (chance of catching a real effect) needs about 38k queries per arm, ~75k total. The sample size is 16 · p(1-p) / delta^2: the 16 is the standard constant for comparing two proportions at that alpha and power, p(1-p) is the variance of the metric (worst near 50%), and delta^2 is the squared effect size, squaring is what hurts, so halving the effect you want to detect costs 4x the traffic. At 1M queries/day (~11.6/s) that is about two hours of traffic if the whole stream is in the experiment, 18 hours at 10%.

One trap: captions change a shared index, so a user-level split leaks between arms. If both variants write into the same index, a control-arm user can still be served a document the treatment arm recaptioned. You need two indexes with a query-level split, or an interleaving design where both systems’ results are mixed into one list per user and you measure which system’s items get clicked.

Alt text does not A/B. There is no online event meaning “the description was correct.” A screen-reader user who hears a false description produces no click, scroll, or reformulation that means “wrong.” So you work with proxies, and only one is strong enough to gate a launch:

Edit distance is the number of single-character edits to turn one string into another, and it is the underrated signal. A 12% edit rate at a median distance of 30 characters is a completely different system from a 12% rate at a median of 3: the first means authors are rewriting content, the second means they are fixing a comma. Report the distribution, not the rate. This is why making alt text editable is a decisive product choice: it is the difference between having a decision metric and having none.

Serving, scale, and cost

One upload’s path has six stages, and only two of them are the model:

flowchart TD
    U(["Upload"]) --> H["Content hash + pHash<br/>dedupe cache"]
    H -->|hit| CACHED(["Cached caption"])
    H -->|miss| SAFE{"Pre-filter<br/>NSFW · CSAM hash · face count"}
    SAFE -->|block| REJ(["No caption · flag"])
    SAFE -->|pass| TXT{"Text-presence<br/>detector · 3 ms"}
    TXT -->|no text · 92%| LOW["Encode at 336<br/>64 visual tokens"]
    TXT -->|text · 8%| HI["Dynamic tiling<br/>4 crops + thumbnail<br/>2,880 visual tokens"]
    LOW --> DEC["Projector + LLM decode<br/>batched"]
    HI --> DEC
    DEC --> CD["Contrastive decode<br/>subtract language prior"]
    CD --> GATE{"Post-filter<br/>noun-phrase grounding<br/>attribute lexicon<br/>3-sample agreement"}
    GATE -->|unsupported span| STRIP["Strip or hedge the span"]
    GATE -->|clean| OUT(["Caption + confidence"])
    STRIP --> OUT
    OUT --> S1[("Alt-text store")]
    OUT --> S2[("Search index<br/>text + embedding")]

    style TXT fill:#bc6c25,color:#fff
    style GATE fill:#2d6a4f,color:#fff
    style SAFE fill:#1d3557,color:#fff

The dedupe stage hashes the image twice, a content hash on the exact bytes and a perceptual hash on the visual content, and a hit returns the cached caption for free. The safety pre-filter (NSFW classifiers, a CSAM hash match, a face count) runs before the captioner, because a captioner has no notion of policy and will describe anything neutrally. The router, a 3 ms text-presence detector, sends the 92% of images with no text down a cheap 64-token lane and the 8% with text to dynamic tiling at 2,880 tokens. The guards subtract the language prior and check grounding. Then results are written to both stores. The two decisions that matter: the router decides cost, the post-filter gate decides truth.

Latency budget

The flat 576-token configuration, batch 8 on the synchronous path, is the baseline the router improves on. Two phases split the model’s work:

• Prefill reads the whole input in one parallel pass. It is limited by arithmetic, so its cost is 2 · params · tokens.
• Decode emits the caption one token at a time. It is limited by how fast the weights can be pulled from memory, not by arithmetic: every token requires reading all 8B weights. So halving the model halves decode, and doubling the token count does not change per-token decode time.

fetch + JPEG decode                                    40 ms
resize + normalize                                      5 ms
vision encoder   8 x 382 GFLOP = 3.1 TFLOP @ 300 TF/s  10 ms
projector                                              ~0 ms
LLM prefill      (576 + 40) x 8 = 4,928 tokens
                 2 x 8e9 x 4928 = 78.8 TFLOP @ 300 TF/s  263 ms
LLM decode       25 tokens x 4.8 ms (memory-bound)    120 ms
post-filter (detector + lexicon)                       15 ms
                                                      -------
                                                      453 ms  p50
                                                     ~900 ms  p95 with queueing

Prefill dominates, and prefill is 94% image tokens (576 of 616). Prefill is symmetric in params · tokens, so a 2x cut on either factor buys the same prefill, and halving the decoder actually saves more per unit ratio (191 ms vs 123 ms), because it takes decode with it and cutting tokens does not. So why cut tokens? Because tokens have far more slack. You can cut visual tokens 9x (576→64, saving 218 ms of prefill) for a 0.012 move in CHAIR_s, and there is no 9x cut of the decoder that costs that little: an 8B model cut 9x is a 0.9B model that will not write a usable caption. On the 64-token path this budget lands at ~235 ms p50.

What visual tokens actually buy

Average pooling merges each 2x2 block of patches into one vector (576→144). A Perceiver resampler holds a fixed number of learned query vectors, here 64, that attend over all 576 patches, so output length is fixed by design. OCR is optical character recognition, reading text printed inside an image.

Token count buys fine detail and almost nothing else. Across a 9x reduction (576→64), general captioning barely moves (CHAIR_s 0.066→0.078), but OCR collapses 5x and counting drops from 0.58 to 0.42. So the answer is not a single token budget but a route: send images that need detail down the expensive lane and everything else down the cheap one.

Blend the cost on a 92/8 split (0.92 x 13.3 + 0.08 x 373.8 TFLOP = 42.1 vs the flat 78.8), and the router is 1.87x cheaper on prefill. But routing is not better on every axis. Blending the quality columns on the same split, general captioning gets slightly worse (CHAIR_s 0.066→0.077) and counting drops (0.58→0.44), because the 92% rides the cheap lane and the 8% were selected for text, not for counting. The one real quality win is OCR for the images that contain text: those all land in the hi-res lane, which is ~1.9x better on OCR than the flat path. So routing is a cost-and-OCR win with a named price, 0.011 of CHAIR_s and 0.14 of counting accuracy, good for an alt-text or search product, bad for a catalog product that counts things (whose fix is a second route to a detector, below).

Throughput and cost per 1M images

Per-lane compute converts into images per second and then into dollars. On one H100 at 300 TFLOP/s effective, batch 64:

• Flat 576-token lane: ~27.8 img/s (prefill is 91% of the batch).
• 64-token lane: ~115 img/s.
• 2,880-token hi-res lane: ~6.1 img/s (five encoder passes plus a huge prefill).

The hi-res lane is 19x slower per image than the cheap one, which changes how you combine the rates. Do not average them: the fleet spends seconds, not images. Average seconds per image, then invert, a harmonic blend:

1 / routed  =  0.92 / 115  +  0.08 / 6.1  =  0.0080 + 0.0131  =  0.0211
routed      =  47.4 images/s

The 8% pays 62% of the time (0.0131 of 0.0211). A route to an expensive lane is only cheap if its cost ratio is smaller than its traffic ratio, and here 19x cost against 11.5:1 traffic means it wins by less than the 92/8 split suggests.

Converting images/s to dollars is one inverse relationship. A 19x slower lane costs 19x as much:

Two notes on the API row. Its ~1.5k image tokens are the API’s patch grid (ceil(w/28) × ceil(h/28), capped near 1,568), not the 576 or 64 you control when self-hosting. And $4,950 is per-image 1,500 × $3/1e6 + 30 × $15/1e6 = $0.00495, times a million. That is 340x the routed path, but the build/buy line is a volume, not a preference. Self-hosting has a large fixed cost ($21,900/yr for a round-the-clock H100 plus ~$50,000/yr for 20% of an engineer = ~$71,900/yr) and a tiny marginal cost ($0.0000147/image); the API is the reverse. Dividing the fixed cost by the per-image saving gives breakeven at ~14.6M images/yr, about 40k images/day. Below that, call the API and spend the engineer elsewhere.

Caching, and the one trap in it

Content-hash dedupe keys on the exact bytes. It catches re-uploads and CDN variants and typically removes 15-30% of traffic for the cost of a hash, and an exact match cannot be wrong.

Perceptual hashing additionally catches re-encodes and crops, strictly more hits, and where the trap lives. A pHash collision means captioning image A with image B’s description: two visually similar images collide, the cache returns a well-formed caption about a different photo, and nothing raises an error. So a pHash hit must be confirmed by a cheap embedding-distance check before reuse. Exact hash: reuse freely. Perceptual hash: verify, then reuse.

Failure modes

Every failure here has the same shape: a missing evidence channel is filled in by the model’s prior, and the output is indistinguishable from a grounded one.

Object hallucination — the dominant one

A logit is the raw score for a candidate next token before scores become probabilities. Loosely, it decomposes into two sources of conditioning:

logit(w)  ≈  f_lang(w | w_<t)  +  f_vis(w | image, w_<t)

This is a sketch, not an identity (the channels are not literally separable), but it names the competition. f_lang is the pull from “what word usually follows these words,” f_vis the pull from “what is actually in this picture,” and the model emits whichever wins. Three facts make f_lang win:

1. f_lang was trained on trillions of text tokens, while f_vis reaches the decoder through a 21M-parameter projector trained on 1-100M pairs. The language prior is far better estimated.
2. The image contributes a fixed number of key/value entries (576, or 64) while the text prefix grows by one every decode step. Softmax normalizes attention to sum to 1, so the share of attention available to image tokens falls monotonically as the caption lengthens.
3. Cross-entropy never penalized a plausible-but-absent object more than an implausible one. There was no gradient distinguishing them.

Fact 2 makes a falsifiable prediction: hallucination rate must rise with caption length. It does:

generated length      CHAIR_s     hallucinated objects per caption
 15-25 tokens          0.048                0.07
 40-60 tokens          0.091                0.19
 90-130 tokens         0.168                0.51

From ~20 to ~110 tokens (a 5.5x length increase) the hallucination rate rises 3.5x and hallucinated objects per caption rise 7.3x. So “describe this image in detail and be thorough” is a hallucination prompt. Added to improve alt text, it makes it measurably worse.

Fact 3 predicts the hallucinated objects should be co-occurrence-likely, not random. On a kitchen image (cutting board, knife, three tomatoes; no refrigerator, no stove), a “describe this in detail” prompt reliably invents a refrigerator and a stove, because in the caption corpus P("refrigerator" | "kitchen") is high. The model is sampling from the conditional distribution of kitchen captions, which contains appliances. It is doing exactly what it was trained to do; the objective is what is wrong. (On that single caption, CHAIR_i = 2/7 = 0.286 is the informative number; CHAIR_s over one caption is degenerate at 1.0.)

Four fixes follow, from free to expensive:

a) Shorten the target and permit abstention. Free, and by the table above worth ~3x. Do this first.

b) Visual contrastive decoding. Run a second forward pass on a blanked image and subtract it, with a strength knob a around 0.5:

logit_final(w)  =  (1 + a) · logit(w | image)  -  a · logit(w | blank)

The second term is the language prior, measured and not assumed, so subtracting it removes exactly the mass with no visual support. Nominally 2x prefill, but the blank-image prefix is constant across every request, so its KV cache is computed once at startup and reused forever. Real cost: one extra decode stream, ~120 ms.

c) Detector-gated verification. Chunk noun phrases out of the caption (a noun plus its modifiers, such as “a wooden cutting board”), run an open-vocabulary detector on each, and strip or hedge the unsupported ones. About 40 ms, and it hands you production CHAIR for free.

The gate must check phrases, not tokens. Querying the detector with every whitespace-split word asks it about with, sits, and the, which score zero and get flagged as hallucinations, hedging most of the caption, including the real “three ripe tomatoes.” A gate that hedges everything is off, not conservative: the reader learns to ignore the marker and the two genuinely invented objects hide in the noise. A simple chunker finds noun phrases without a part-of-speech tagger by treating closed-class function words (a, the, on, with, is, and…) as boundaries, since a noun phrase never contains one. Each phrase is queried twice, the full phrase and its bare head noun, and kept if either clears the threshold, because detectors are stronger on heads than on long modifier stacks. Quantifiers (“three”) are dropped from the query, and scene words (“background”, “scene”, “wall”) are skipped entirely since no detector can box them. On the kitchen caption this hedges exactly two spans:

A modern kitchen with light wood cabinets. On the counter sits a wooden
cutting board with three ripe tomatoes and a chef's knife. A [unverified]
stands against the far wall, and a [unverified] is visible to the left.

d) DPO on auto-labeled pairs (see Training). Highest quality, needs a training run, and the detector from (c) is already the label function.

Demographic assumption from context

Sampling the same image (a person in blue scrubs holding a clipboard in a hospital corridor) five times returns “a nurse” 4/5 and “a doctor” 1/5, with one sample asserting “young” and one “female”, none of which the pixels support. The model is completing from P(role word | scrubs, appearance) learned from web captions where those correlations are strong. The output is a demographic prior rendered in declarative grammar, and when it happens to be right, it is right because it encoded a stereotype.

Two controls, both in code, not in the prompt (a prompt reduces the rate and leaves a residue, and the residue is a fairness incident, not a quality miss, see the reliability and guardrails lesson):

• An attribute lexicon gate. Role, gender, age, and ethnicity terms are checked against visual support; unsupported ones are rewritten to the supported form (“a person in scrubs”).
• Sample disagreement as the detector. Any attribute term that does not appear in all 3 samples is unsupported. This costs 3x decode on a 25-token output, cheap, and needs no labels. Self-consistency across samples is a real uncertainty signal in a way that asking the model for a confidence score is not (see the customer support agent lesson).

OCR failure

A bigger model cannot help, because the information was destroyed before the model ran. A receipt photographed at 3024x4032 and resized to 336 is a 9x downsample: 10-point text drops from ~32 px to ~3.6 px, and one 14x14 patch covers about four lines of text before becoming a single 1024-d vector.

model output:  "A receipt from a restaurant showing a total of $12.50."
ground truth:  total $47.83

The number is not misread; it is generated from the prior over receipt totals, because glyph identity was destroyed by the resize before the model ever ran. The fix is resolution, not model size: dynamic high-resolution tiling, several native-resolution crops plus a downscaled thumbnail for global layout, routed by the cheap text-presence detector, so only ~8% of traffic pays for 5x the visual tokens.

Counting

Two mechanisms:

1. The encoder pools. Self-attention gives set membership far more readily than cardinality; there is no architectural component that counts.
2. The caption corpus almost never enumerates. Humans write “several,” “a group of,” “a handful,” so even a perfect visual representation has no supervision mapping it to “seven.”

true count     1      2      3      4      5      6      7     8+
exact-match  1.00   0.95   0.81   0.52   0.31   0.19   0.12  0.07

Accuracy collapses past 4, roughly the human subitizing limit, the number of objects a person can see at a glance without counting. That is not a coincidence: the training captions were written by humans who also stopped counting at four and switched to “several.” The fix is not to ask the captioner. Route counting queries to a detector and count boxes.

Summary

Alternatives considered and rejected

Conclusion

The whole design rests on three things:

• What the caption is for. Alt text and search indexing want opposite outputs and opposite error asymmetries, so neither can be designed until the purpose is answered. One encoder and one decoder serve both, with different decode configs, metrics, and launch processes.
• The encoder was trained against text. That is the only reason a 21M-parameter bridge between a frozen encoder and a frozen 8B decoder works: training learns a near-linear remap, not the world. It is what makes “train 0.26% of the parameters” true, and it collapses if the encoder was trained on class labels instead of captions.
• An open-vocabulary detector is accurate enough to be a label function. It is simultaneously the free DPO preference labels, the serving-time grounding gate, and the production hallucination metric. If it is wrong, all three are wrong together.

The recurring lesson is that cross-entropy has no truth term, so every hallucination control (short targets, contrastive decoding, the noun-phrase gate, DPO) exists to reinstall the truth signal the training loss omitted. And the dominant cost lever is a cheap classifier that routes traffic to unequal lanes: 92% cheap, 8% expensive, which is 1.87x cheaper on prefill but pays for it with a named quality cost, not a free lunch.

One line to remember: settle what the caption is for first, because everything after it (the model, the data, the metric, the truth term you have to add back) is downstream of that one answer.

flowchart TD
    subgraph BUILD["Offline — build the model"]
        C1["Contrastive pretrain (or download)<br/>ViT + text tower · 400M pairs"]
        C2["Align projector · frozen ends<br/>21M params · ~100 GPU-hr"]
        C3["Task-tune via LoRA<br/>500k examples"]
        C4["DPO on detector-labeled pairs<br/>installs the truth term"]
        C1 --> C2 --> C3 --> C4
    end
    subgraph SERVE["Online — one upload"]
        R["Router: text-presence detector<br/>92% cheap · 8% hi-res"]
        M["Projector + frozen LLM decode"]
        G["Contrastive decode + noun-phrase<br/>+ attribute + 3-sample gate"]
        R --> M --> G
    end
    C4 -. deploys .-> M
    C4 -. same detector .-> G
    G --> STORES[("Alt-text store + search index")]

    style C4 fill:#2d6a4f,color:#fff
    style R fill:#bc6c25,color:#fff
    style G fill:#1d3557,color:#fff

Further reading

• Radford et al., “Learning Transferable Visual Models From Natural Language Supervision” (CLIP).
• Liu et al., “Visual Instruction Tuning” (LLaVA, the prefix-projection join).
• Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning” (the cross-attention join).
• Rohrbach et al., “Object Hallucination in Image Captioning” (the CHAIR metric).
• Hessel et al., “CLIPScore: A Reference-free Evaluation Metric for Image Captioning”.
• Leng et al., “Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding”.
• Rafailov et al., “Direct Preference Optimization”.
• Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”.

Next: 06. Retrieval-Augmented Generation.