How to design image captioning

Image in, text out; the design is decided by what the caption is for, built as a frozen encoder + frozen decoder joined by a tiny trained bridge, and defended against one dominant failure: confidently naming objects that are not there.

Purpose decides everything

• “Caption” is an output format, not a goal. Settle the consumer first; it fixes model, data, and metric.
• Two opposite error asymmetries:

• Two more consumers: moderation triage (recall at fixed precision, structured output) and product catalog (per-attribute F1).
• Alt text is context-dependent (surrounding page text is an input); captioning is not.

The objective is structurally wrong twice

• Conditional LM, teacher forcing, cross-entropy: P(caption|image) = prod_t P(w_t | w_<t, image); loss = -sum_t log P(w_t* | w_<t*, image).
• Teacher forcing: each position sees the true prior words, graded against one reference string.
• Problem 1 (one-to-many): one image has thousands of valid captions; CE drives toward the mode, the generic caption (“a man riding a horse on a beach”). Blandness minimizes loss.
• Problem 2 (no truth term): a hallucinated object and a stylistic variation cost the same. Nothing distinguishes wrong from different. Every downstream control reinstalls this missing term.

Architecture: frozen ends, tiny bridge

• Encoder ViT (not CNN) because its output is already a token sequence. ViT-L/14 @ 336px: (336/14)^2 = 576 patch tokens + 1 CLS = 577; ~382 GFLOP/image (~9% attention).
• Join = prefix projection (LLaVA), not cross-attention. An MLP maps each patch 1024 -> 4096 -> 4096, spliced in where token embeddings sit. Works because the LLM reads any R^4096 vector as a token; projector only learns the remap.
• Projector ~21M params = 21e6/8e9 = 0.26% of an 8B decoder. Freeze both ends, train only the bridge -> GPU-hours, not months.
• Token reducer compresses 576 -> 64 before the projector (average pool or Perceiver resampler).

Why the bridge is enough: CLIP

• CLIP trains image + text encoders into one space via contrastive loss (InfoNCE); negatives are the other batch items, so batch size is a loss hyperparameter (N=32,768 -> 32,767 negatives). Gradient accumulation is not a substitute.
• Payoff: zero-shot classification collapses to retrieval; and the encoder’s outputs are text-shaped, so the projector learns a near-linear remap and converges on ~100k-1M pairs, not 100M.
• CLIPScore comes free as a reference-free metric (and inherits CLIP’s weaknesses).

Data

• Web alt text is the only source at scale and mostly garbage. Funnel ~5,000M -> ~360M (7.2% survival).
• CLIP-similarity filter (cos >= 0.28) is the sharpest cut (~73%) and most dangerous: you filter with a model trained on the same data, narrowing toward its competence. Mitigate with a cleaner filter model + keep an unfiltered random sample.
• Recaptioning = a strong VLM teacher writes dense targets for a smaller student (distillation). ~100M images ~ $25-40k all-in; moves quality more than any architecture change.
• Best corpus = ~80/20 synthetic/original mix — a union of non-overlapping capabilities (synthetic = grounded description; original = rare proper nouns). Ceiling: student cannot beat the teacher on the distilled axis, only on cost.
• Eval set: buy ~2,000 images, stratified by domain, 5 references + per-object presence each (so CHAIR is computable), ~$15k. Labels are for deciding, not training.

Training: four stages

• Stage 2 freezes the decoder: unfreezing while the projector emits noise makes the LLM degrade its language ability to fit bad image reps, unrecoverably.
• Stage 3 uses LoRA for memory: full 8B in bf16 = 16GB weights + 16GB grads + 64GB Adam moments = 96GB (> 80GB H100). LoRA r=16 trains ~20M params -> fits one card.
• Stage 4 (DPO) installs the missing truth asymmetry; pairs are auto-built by sampling 4 captions at T=1.0 and labeling with an open-vocabulary detector (clean if every named object detected > 0.35).

Metrics: all disagree with humans, two reasons

• Reference-based: BLEU, METEOR, ROUGE-L, CIDEr, SPICE. Reference-free: CLIPScore, CHAIR.
• Mechanism 1: reference-based metrics measure agreement with a tiny sample -> reward the generic caption (the one-to-many failure). Optimizing CIDEr pushes the model toward the bug.
• Mechanism 2: reference-free metrics measure similarity, not entailment. CLIPScore scores “a dog chasing a man” and “a man chasing a dog” almost equally — CLIP is a bag-of-concepts, blind to who-did-what.
• CHAIR_i = hallucinated mentions / all mentions; CHAIR_s = captions with a hallucination / all captions. Different numbers on identical data (CHAIR_s = 2·CHAIR_i in the toy set). Always report the subscript; headline numbers here are CHAIR_s.
• Gate on CHAIR + human correctness; keep CIDEr as a regression tripwire only. (Real case: CIDEr picks model A; correctness, CLIPScore, CHAIR_s all pick the better model B.)
• Human eval on separated axes; correctness and specificity trade directly — report the pair.

Online metrics

• Search A/Bs cleanly: metric = search success (query -> click, no reformulation in 30s); guardrail = CTR. Sizing 16·p(1-p)/delta^2; halving the detectable effect costs 4x traffic. Trap: shared index leaks between arms -> use two indexes with query-level split or interleaving.
• Alt text does not A/B (no “was it correct” event). Decision metric = author edit rate + edit distance (making alt text editable is what gives you any decision metric); a 12% edit rate at 30 chars differs from 12% at 3 chars — report the distribution.

Serving: router decides cost, gate decides truth

• Path: dedupe (content hash + pHash) -> safety pre-filter (before captioner) -> text-presence router (3ms) -> encode -> projector+LLM decode -> contrastive-decode + noun-phrase/attribute/3-sample gate -> stores.
• Router: 92% no-text -> 64-token cheap lane; 8% text -> dynamic tiling, 2,880 tokens.
• Cache: exact content hash = reuse freely; pHash hit must be verified by embedding distance (collision returns a well-formed caption about a different photo).

Latency (~453ms p50 flat 576-token): prefill dominates (94% image tokens). Prefill 2·params·tokens; decode is memory-bound (~4.8ms/token). Cut visual tokens 9x (576->64) for only +0.012 CHAIR_s -> ~235ms p50.

• Tokens buy fine detail (OCR, counting) and almost nothing else -> route, don’t pick one budget.
• Cost/1M images @ $2.50/GPU-hr: flat 576 = $25.00; 64-token = $6.04; routed (8% hi-res) = $14.66; flat hi-res = $113.90; frontier API ~$4,950.
• Routed throughput is a harmonic blend: 1/r = 0.92/115 + 0.08/6.1 -> 47.4 img/s; the 8% pays 62% of the time.
• Build vs buy is a volume: self-host fixed ~$71,900/yr, marginal ~$0.0000147/img -> breakeven ~14.6M images/yr (~40k/day). Below that, call the API.

Failure modes

Every failure: a missing evidence channel is filled by the model’s prior, output indistinguishable from grounded.

• Logit sketch: logit(w) ~ f_lang(w|w_<t) + f_vis(w|image, w_<t). f_lang wins because it saw trillions of tokens vs a 21M projector; image tokens are fixed (576/64) while text prefix grows, so image attention share falls monotonically; and CE never penalized plausible-but-absent objects.
• Hallucination rises with length (CHAIR_s 0.048 -> 0.168 from ~20 to ~110 tokens). “Describe in detail” is a hallucination prompt.
• Hallucinated objects are co-occurrence-likely (kitchen -> invents refrigerator, stove), not random.
• Fixes, free to expensive: (a) shorten + abstain (3x, do first); (b) visual contrastive decoding logit_final = (1+a)·logit(image) - a·logit(blank), a0.5, blank KV cached once (~120ms); (c) detector-gated noun-phrase verification (~40ms, gives production CHAIR) — check phrases not tokens, or it hedges everything; (d) DPO.
• Counting collapses past 4 = the subitizing limit (the humans who wrote the captions also stopped counting at four).

Rejected alternatives

• Frontier API for everything: ~340x marginal cost (wrong above ~40k/day).
• Cross-attention join: custom stack for no gain at one image/caption.
• End-to-end train the encoder: destroys CLIP alignment, ~400x GPU-hours.
• Retrieval/nearest-neighbor caption: trades hallucination for confident misattribution (worse).
• Optimize CIDEr as gate: prefers the model that hallucinates 3x more.
• Prompt “don’t hallucinate”: reduces, never removes; residue needs the detector anyway.
• Self-reported confidence: model has no access to whether the object was present; use 3-sample agreement or detector support.