05 — Harmful Content Detection

“Build a system that detects harmful content on our platform.”

Content moderation looks like a classification problem. It is really a label problem wrapped around an action problem.

By the end you should be able to:

• name every model in the design, and say what goes into it and what comes out;
• explain why a single decision threshold cannot exist here, and where each of the four thresholds this system does use actually comes from;
• trace how a label’s origin — a user’s report, a reviewer’s eight-second verdict, or a rewritten policy document — changes the metric you are entitled to quote;
• price a staffing decision against a modeling decision in the same currency.

The worked example is a platform carrying two billion posts a day. Every figure is recomputed from stated assumptions rather than asserted, so you can follow the arithmetic and disagree with the inputs.

Two ideas to say in the first thirty seconds

The policy is the label definition. Everything downstream is a consequence of it. Most of the disagreement in this domain — between teams, between reviewers, between you and the interviewer — is disagreement about policy wearing a modeling costume.

“Is this hate speech?” has no answer until someone writes down what hate speech is, in a document, with worked examples. Even then, two trained reviewers will disagree on the hard cases at a rate you can measure. That measured rate is a ceiling on every model number you will ever report.

The cost asymmetry cuts both ways, and that is what separates this problem from every other classification problem in the set. In fraud, a missed fraud costs money and a false positive costs a phone call — one error is clearly the cheap one. Here, a missed violation is real harm to a real person, and a false positive is a legitimate user silenced by an automated system with no explanation. Neither error is the cheap one. That symmetry forces the entire architecture, and Why you cannot pick one threshold derives it.

Two definitions borrowed from elsewhere in this repo

Both are used constantly below. They are spelled out here so that nothing in this chapter requires you to go and read another one first.

The precision-recall curve, usually shortened to PR curve. Sweep a score threshold from high to low. At each setting, precision is the share of the items you acted on that were genuinely violating, and recall is the share of all violating items that you caught. Plot one against the other and you get a curve; the area underneath it is the PR-AUC (area under the precision-recall curve), which summarises the whole curve in one number.

Read the PR curve rather than the ROC curve when violations are rare and you operate at a low false-positive rate, because it never counts the vast, uninformative pool of correctly-ignored benign posts. It comes with one hard restriction: precision depends on how rare violations are, so a PR-AUC is only comparable against another PR-AUC measured at the same prevalence. Quote it with the prevalence attached, and switch to ROC-AUC when you need to compare across surfaces or time periods whose prevalence differs. Pr auc vs roc auc under heavy imbalance and the comparison pr auc is not allowed to make derives both halves.

The noise ceiling. If the labels you train on and the labels you test against are themselves wrong some of the time, then a model that is perfect still disagrees with the test set exactly as often as the test set is wrong. Your measured score is capped by your label quality, not by your model. The noise ceiling derived turns that into an exact number, and The label is the hard part is about living underneath that cap.

The route through the chapter

Each section answers one question, and later sections spend numbers earlier ones derive:

The overall shape of the answer follows the interview spine in 01 — Framework, but you do not need it to read this.

The models in this design, and how you know each one works

Before any argument about thresholds or metrics, here is the concrete answer to “what did you build.”

One post goes in; one enforcement action comes out — remove, remove with an account strike, queue for a human, demote, or nothing.

In between sit seven things:

• a lookup table of fingerprints that catches re-posts of content you already enforced against;
• two classifiers arranged in a cascade, meaning a cheap one runs on everything and only what it flags reaches an expensive one;
• a calibration layer that turns raw model output into a probability you can compare against a threshold;
• and three small models that fill the human review queue, order it, and audit what enforcement then does.

Nothing here is a single “harmfulness model.”

The table below is the whole design on one page. Each row answers the same five questions about one component: what it is, what goes in and what comes out, where its training labels come from, the one number that says it is working, and whether it runs on the live publish path (online) or on a schedule over logs (offline). Do not try to absorb it now — read the first two columns, notice that only one row is the big neural network, and come back to it after Serving architecture draws the same components as a diagram.

Two absences in that table are deliberate, and both matter later.

There is no model that chooses an action. Actions are chosen by comparing scores against four fixed thresholds, and three of those four are not learned from anything: two are policy commitments and one is read off an org chart (Why you cannot pick one threshold, Review capacity is the binding constraint).

There is no model that uses the post’s own engagement history — no views, no clicks, no report count. At scoring time the post is milliseconds old and has none. What the model eats traces the three separate design consequences of that one empty row.

Assumptions this chapter runs on

Everything below rests on the volumes and rates in Framing and the objective that is not multi class and the staffing figures in Review capacity is the binding constraint. Most of them can be wrong by a factor and change only digits. Two cannot. Every section below ends with a paragraph doing the same job for its own numbers, so you always know which figure to attack.

• Load-bearing: the base rate of 0.12%. That is the share of posts that violate some policy. Every threshold, the dismissal of accuracy as a metric, and the 12.5 M wrongful-removals figure that kills the single-threshold design all scale directly with it. At 5% prevalence this is a different chapter.
• Load-bearing: 8.15 M items/day of human review capacity. That is 0.41% of the platform, and 0.41% is the fraction the review threshold cuts. (t_review is the name this chapter uses for that threshold: the score above which an item goes to a human. Why you cannot pick one threshold names all four thresholds and Review capacity is the binding constraint derives this one.) But only 7.19 M of that capacity — 0.36% of the platform — is content a human actually sees; the top 0.96 M is auto-removed with no reviewer involved. What the quantile buys keeps the two apart and prices the gap. Every argument about which content a human sees is an argument about the 0.36%.
• Not load-bearing, though they look it: the GPU cost model, the exact parameter count, and the video frame count. The cascade priced shows the entire machine bill is a fraction of a percent of the human bill, so being wrong about it by 10x changes no decision.

1. Framing, and the objective that is not multi-class

Start by fixing the shape of the answer — what goes in, what comes out, at what volume — and settling one design question before any model exists: whether the ten policies share one output or get ten of their own.

• Input: a post — text, image, video, audio, or any combination — plus author, thread context, and audience.
• Output: a per-policy score, and an enforcement action.
• Volume: 2 x 10^9 posts/day, 180 x 10^9 views/day.
• Prevalence: ~0.12% of posts violate some policy — 2.4 M/day. Prevalence is this domain’s word for the base rate, the share of the population that is actually positive; The product metric prevalence defined on views redefines it on views rather than posts and that redefinition changes what the whole system optimizes.
• Latency: classification must complete at publish time, and the enforcement it feeds has to land within a few minutes — Metrics prevalence and why accuracy is meaningless derives the target as a median time-to-action of about four minutes, at which a post has delivered 4% of its lifetime views against 22% at eighteen minutes. Minutes, not seconds: the binding number is queue latency, not model latency.

Multi-label, per policy — and why multi-class is wrong

The output layer comes first: ten scores that can all be high at once, rather than one score split across ten categories.

Two words first, because the section title turns on the difference. A multi-class output picks exactly one label out of several — the classic device is a softmax, a function that takes a set of raw scores and squashes them so they are all positive and sum to one, which forces them to compete. A multi-label output asks a separate yes/no question per label, using a sigmoid on each one: a function that squashes a single number into the range 0 to 1 independently of every other number, so all ten answers can be “yes” at once.

A post can be a violent threat and hate speech and harassment simultaneously. A softmax over policies forces those to compete for probability mass. That is statistically wrong, and it is operationally worse: each policy carries a different enforcement consequence, a different reviewer skill set, a different appeal path, and a different legal reporting obligation. A single “harmful” score cannot be routed to any of them.

Here are the ten policies, grouped by the severity their owners assign them. Severity runs 1 to 5 and it decides the consequence of a violation, not how likely one is. Read the grouping, not the individual names — the point is the spread between the top row and the bottom row:

independent sigmoid heads on a shared trunk, with a severity per policy:
    child safety 5 · violent extremism 5   auto-escalate, legal reporting
    credible violent threat 4
    adult nudity 3 · hate speech 3 · harassment 3
    graphic violence 2 · regulated goods 2
    spam 1 · health misinformation 1        demote only, no removal

Severity is not the score. The score is P(violates policy k) — how confident the model is. Severity is a policy-owner’s judgement about how much harm one violating view does. They are different quantities from different places, and they multiply in the action decision (Why you cannot pick one threshold).

Confusing the two is the most common design error here. A 0.99 spam score should not out-rank a 0.60 child-safety score in a review queue, even though 0.99 > 0.60.

One trunk, ten heads

Two words for the shape of the network, because the rest of the chapter uses them constantly.

• A trunk is the shared body of a network. It turns the raw input into one internal summary vector.
• A head is a small layer sitting on top of that summary, answering one specific question.

Ten heads on one trunk means the expensive work of reading the post happens once and is then asked ten separate questions.

That output shape is fixed here, before any model exists, because it is a policy decision rather than a modeling one: ten heads because there are ten enforcement paths.

What the trunk here actually is — one fused encoder, 241 M parameters, ten linear heads on a 768-dimensional pooled representation — is The trunk named. How it is trained is the end of From those sources to a training set, because the training recipe is a consequence of where the labels come from, and that is the next section’s subject.

Assumptions here, and which one is load-bearing. The volumes are stated by the interviewer and the severity ladder is stated by policy; neither is derived. The load-bearing one is the 0.12% base rate: it is what makes precision collapse at any usable recall, and every threshold argument in Why you cannot pick one threshold is downstream of it. The ten-policy split is not load-bearing for the arithmetic — merge them into three and every number changes while every argument survives — but it is load-bearing for the architecture, because it is why there are ten heads rather than one score.

2. The label is the hard part

Where does a harmful-content label physically come from, and what does each origin do to the numbers you are allowed to report? The answer arrives in three movements: how much two trained humans agree and why that caps every metric; the five distinct label sources, each with a different bias; and the recipe that turns those sources into an actual training set with an actual loss function.

Inter-annotator agreement is the ceiling

The ceiling comes first: how often two trained reviewers give the same verdict on the same item, and what that implies about how often either of them is right.

Krippendorff’s alpha measures how much two or more people annotating the same items agree, corrected for the agreement they would have reached by chance alone. It runs from 1.0 (they always agree) down through 0 (they agree no more often than two people guessing independently), and it can go negative for systematic disagreement.

The table below is measured on a 4,000-item sample where every item was labeled twice, independently. Read the alpha column first and notice the range: the policies at the top are nearly solved definition problems and the ones in bold are barely definitions at all. The middle column is derived two blocks down; take it on faith for a moment.

Where the implied-accuracy column comes from

Derive it, because it is the number that reframes the whole project. It takes two steps, and the first is the one people skip.

Step 1: alpha is not the probability that two reviewers agree. In its general form alpha is 1 - D_o/D_e, where D_o is the disagreement actually observed and D_e is the disagreement you would expect if the same labels were shuffled at random. On a binary yes/no call that reduces to a chance-corrected agreement rate, so you have to undo the correction before you can say anything about reviewers. P_chance below is the probability two reviewers land on the same answer purely by luck, computed from how often each label appears in this sample:

alpha      =  (P_agree - P_chance) / (1 - P_chance)

rearranged =>  P_agree = alpha + (1 - alpha) × P_chance

P_chance   =  0.36            measured from this sample's label marginals

hate speech, alpha 0.54:
P_agree    =  0.54 + (1 - 0.54) × 0.36
           =  0.54 + 0.46 × 0.36
           =  0.54 + 0.1656
           =  0.7056          two reviewers agree 70.6% of the time

Step 2: turn an agreement rate into an accuracy. Model two reviewers who each get the answer right with probability q and err independently. They agree when both are right (q × q) or both are wrong ((1-q) × (1-q)) — and note that “both wrong” counts as agreement, which is why agreement always overstates accuracy. Set that equal to the 0.7056 above and solve the quadratic:

P(agree)   =  q^2 + (1 - q)^2  =  2 q^2 - 2 q + 1

0.7056     =  2 q^2 - 2 q + 1
0          =  2 q^2 - 2 q + 0.2944
0          =    q^2 -   q + 0.1472

quadratic formula, a=1, b=-1, c=0.1472:
q          =  (1 ± sqrt(1 - 4 × 0.1472)) / 2
           =  (1 ± sqrt(0.4112)) / 2
           =  (1 ± 0.6412) / 2
           =  0.8206   or   0.1794

take the root above 0.5 -- a reviewer worse than a coin flip is not the
case being modeled

Run those same two steps on every row at the same P_chance = 0.36 and you get the middle column of the table: 0.9703, 0.9459, 0.8965, 0.8206, 0.7474, 0.7272.

Now check how much that column depends on P_chance being exactly 0.36. Sweep it across 0.30-0.38, the range this sample’s label marginals plausibly span. On the spam row q moves by 0.004; on the health-misinformation row it moves by 0.059. That is a fifteen-fold difference in sensitivity, and the reason is mechanical: at low alpha almost all of the observed agreement is the chance correction, so the back-solve is dividing by a small number and amplifies any error in it.

So the implied-accuracy column is trustworthy exactly where you did not need it and shaky exactly where you did. Treat 0.97 as a number and 0.73 as an order of magnitude.

What the ceiling does to your metrics

For hate speech a single reviewer’s label is right about 82% of the time. The noise ceiling derived derives what that does to a measured metric: when both your training labels and your evaluation labels come from single reviewers at 82% accuracy, a model that is perfect agrees with the evaluation set only 82% of the time.

F1 is the harmonic mean of precision and recall — the usual one-number summary of a classifier. A measured F1 of 0.80 on hate speech may therefore be a model already sitting at the ceiling. Every additional parameter you spend past that point is spent fitting reviewer noise.

The consequence is the most useful sentence in this chapter: the largest single-quarter quality win in a content-integrity system usually comes from editing a document, not from training a model. Rewriting a policy raises the agreement rate, which raises the ceiling, which raises every measured number underneath it — with no change to the model at all.

Here is one such rewrite, with what it bought. AHT in the block is average handle time: the mean number of seconds a reviewer spends disposing of one item, and the unit review capacity is bought in (Review capacity is the binding constraint). Watch the third row in particular — the model is byte-for-byte identical across it.

hate speech policy v3 -> v4:
    added 40 worked examples, split "slur" into "slur, directed" and
    "slur, reclaimed / in-group", added an explicit satire carve-out

    Krippendorff alpha    0.54  ->  0.68
    reviewer AHT          45 s  ->  38 s      (less deliberation on the newly
                                               specified cases)
    model F1, SAME MODEL, relabeled eval set
                          0.61  ->  0.70

The model did not change. The measurement got less noisy and the training targets got more consistent. If you cannot articulate the policy precisely enough for two trained humans to agree, you cannot train a model to do it and you certainly cannot evaluate one.

Where labels come from, and what each source is good for

A harmful-content label is never a fact you looked up — it is the output of a process, and there are five different processes producing labels in this system. This is the ground an interviewer should press hardest on: each process has a different bias, the biases point in different directions, and each one damages a different metric. There is no single reweighting that repairs all five.

Two words used throughout the table:

• Adjudicated means the item was judged by more than one reviewer and disagreements were resolved by a senior specialist. The label is therefore near the noise ceiling rather than at a single reviewer’s 0.73-0.97.
• Stratified means the sample was drawn deliberately unevenly, over-sampling the hard and rare cases. That makes it excellent for measuring per-policy quality and useless for measuring how much violating content exists.

In the table, the column to read closely is Bias, not Volume. The two biggest rows by volume are the two least usable, and the two smallest rows are the only unbiased ones in the system.

Now take the three sources that dominate the volume, one at a time, and say what each one’s bias does to the number you would otherwise quote.

User reports: 4 M/day, answering a different question than the one you asked

A report fires when a viewer disagrees with or dislikes a post. That overlaps with “violates a policy” only partly, which is where the precision of about 0.06 comes from: nineteen of every twenty reports are not violations at all.

The bias is not random noise, and that is what makes it dangerous. Reports scale with a post’s reach and with how unpopular its author is, so the reported set over-represents visible, controversial and minority-viewpoint content. Organized brigading — a group agreeing to mass-report one account — can manufacture the signal outright.

What that does to the metric: treat a reported-and-then-removed post as a recall success and you inflate recall on exactly the content the reporting population dislikes. You have imported their preferences into the definition of harm.

Worse, a report count is a tempting feature, and a model that learns it learns to enforce popularity.

The control is structural rather than statistical. A report routes an item into the review queue and never becomes a training positive. Its only effect on the system is which item a human looks at.

Human review: 8 M decisions/day, carrying two stacked biases

The first bias is selection: the queue is filled by the classifier, so a reviewer only ever sees items the model already scored above t_review.

The second is annotation: a single reviewer is right 0.73-0.97 of the time depending on policy, and that error is not symmetric across communities. Dialect and demographic bias in enforcement the serious one measures annotators labeling identical semantic content as toxic 2.2x more often when it is written in one dialect than another.

What selection bias does to the metric: recall computed on queue labels is not recall at all. The denominator becomes “violations the model already found,” so the number is guaranteed to look good — it would read 1.00 for a model that finds nothing. That is precision wearing recall’s name.

What annotation bias does to the metric: the per-community numbers are wrong in a direction. A model trained on those labels reproduces the direction, and a model evaluated on those labels is scored by the same skewed ruler that trained it. The error cancels in the measurement and compounds in production, which is the worst possible combination.

Policy changes: a redefinition of the target under the whole label set

This one is not a volume at all. The other four rows sample a fixed population; a policy revision changes what the population is.

When the hate-speech policy went from v3 to v4 — 40 worked examples, a split between directed and reclaimed slurs, an explicit satire carve-out — agreement rose from 0.54 to 0.68 and the same model measured 0.61 to 0.70 on a relabeled evaluation set.

What that does to the metric is the least appreciated bias of the three: any quality time series that spans a policy revision is comparing answers to two different questions. A model that “improved 9 points” across that boundary may have improved by zero. A model that regressed may simply have been asked something harder.

Two controls, and you need both:

1. Version the policy the way you version the model, and stamp every stored label with the policy version that produced it.
2. On the revision date, re-adjudicate a frozen slice of the gold set under the new policy and publish both numbers for the same model. That separates the size of the definitional jump from the size of the modeling change.

Why no single correction fixes all three

The three biases do not point the same way. Reports over-sample what is visible and disliked; review over-samples what the model already believes; a policy change moves the target underneath both. There is no reweighting that repairs all three at once.

Only two rows in the table are designed to be unbiased, and they are small for a reason: adjudication is expensive.

• The gold set is stratified on purpose and answers “how good is the model, per policy.”
• The prevalence sample is uniform over views on purpose and answers “how much harm is being delivered.”

The reviewer queue is your biggest label source and it is sampled by the model you are trying to evaluate. The uniform-random prevalence sample exists to break that circularity, and it is the most valuable 12,000 items per day in the system.

Assumptions in this section, and the load-bearing one. The implied-accuracy column assumes two reviewers err independently, which is the load-bearing assumption and is optimistic: reviewers share a training programme, a guideline document and a workload, so their mistakes correlate. Correlated errors inflate the observed agreement, which inflates the back-solved accuracy, so 0.82 for hate speech is a ceiling on the ceiling — the true single-reviewer accuracy is at or below it, never above. Secondary and not load-bearing: P_chance = 0.36 from the sample’s label marginals, which the sensitivity sweep above already prices, and the 4,000-item sample size.

From those sources to a training set

Now turn the five sources into an actual training run: which rows are positive, which are negative, what the loss function is, and how the data is split. The table above is provenance; what follows is the recipe.

There are six decisions, and not one of them is a modeling preference. Every one is forced by a row in that table or by a constraint another section already derived.

1. Positives: adjudicated decisions only, 890 k/day

That 890 k is Review capacity is the binding constraint’s review band — the violations found inside the 7.19 M items reviewed each day. (The label is the hard part’s table rounds the whole reviewer stream to 8 M decisions/day; 7.19 M of that is the proactive queue, and the balance is appeals and audit work, which produce labels but not a fresh proactive sample.)

Two things are deliberately not positives:

• User reports, excluded at 0.06 precision. The label table says so and Alternatives considered and rejected repeats it.
• Recidivism hash matches, excluded even though they are near-1.0 precision. Serving architecture catches 31% of violations by asset hash, and those are near-duplicates of positives already in the set. Adding them would train a 59-GFLOP encoder — a GFLOP is a billion floating-point operations, so fifty-nine billion of them per item — to recognise the assets a 2 ms lookup already handles.

2. Negatives: three sources, and the cheap one decides whether the model works

The block below lists them with their daily volume and, more importantly, where in the score distribution each one is drawn from. That last part is what makes them different from each other:

hard negatives   reviewer "benign" verdicts in queue          6.30 M/day
                 near-boundary BY CONSTRUCTION -- they scored above
                 t_review, so this is a free hard-negative mine

easy negatives   the tier-1 clear audit sample (section 7)      920 k/day
                 uniform over the 1,840 M posts tier 1 disposes of

unbiased anchor  the prevalence sample                           12 k/day
                 uniform over VIEWS, importance-weighted

Train on the queue’s benigns alone and every negative the model has ever seen scored above t_review. The model is then fitted on 0.41% of the input space and deployed on all of it. That is The label feedback loop’s feedback loop arriving through the negatives instead of the positives.

The tier-1 audit sample is already being paid for in Serving architecture as a measurement, and it is simultaneously the only source of negatives drawn from the 92% of traffic the model is actually run on. One line item, two jobs. Count it in both places, or it looks like a luxury in one of them.

3. The loss: masked multi-label binary cross-entropy

BCE (binary cross-entropy) is the standard penalty for a yes/no prediction. It charges you the negative logarithm of the probability you assigned to the answer that turned out to be true — so being confidently wrong is expensive and being confidently right is nearly free.

The mask is where the multi-label output of Multi label per policy and why multi class is wrong gets paid for. A reviewer who disposes of an item as spam in 8 seconds has not ruled on the other nine policies. Nine of that row’s ten labels are therefore unobserved, which is not the same thing as negative.

How much does that matter? On the gold set — the only source where every item is adjudicated against all ten policies — violating items violate 1.34 policies on average. So of every 1.34 violated (item, policy) pairs, 0.34 sit on a policy the queue never ruled on:

0.34 / 1.34  =  0.254  =  25% of true positive (item, policy) pairs
                          are on a head the queue never looked at

Treat unobserved as negative and you train a quarter of your positives as negatives — concentrated on exactly the co-occurrences Framing and the objective that is not multi class built a multi-label head layout to represent.

4. The prior shift: the training mixture is ~100x too positive

Every threshold in Why you cannot pick one threshold assumes a probability computed at the production base rate. The training mixture is nowhere near it, so the raw model output has to be corrected before any threshold means anything.

Three words first:

• The prior is how common violations are before the model looks at anything — the base rate.
• The posterior is the updated probability after it looks.
• The logit, or log-odds, is log(p / (1 - p)). This is the scale a classifier actually works on, and it has one useful property here: multiplying the odds by a constant becomes adding a constant in log-odds. So a base-rate correction is one subtraction rather than a reshaping of the whole distribution.

Now the arithmetic. The mixture above is 890 k positives against three kinds of negative:

negatives  =  6.30 M  +  0.92 M  +  0.012 M   =  7.232 M
positives  =  0.890 M
                                                 --------
total                                            8.122 M

training prior  =  890 / 8,122     =  0.10958   =  11.0%
serving prior   =  section 1's base rate        =  0.12%

An 11.0% base rate against production’s 0.12% — about 91x in probability, 102x in odds. Convert both to log-odds and the whole correction is one number:

train prior 0.10958  ->  logit  log(0.10958/0.89042)  =  -2.0950
serve prior 0.00120  ->  logit  log(0.00120/0.99880)  =  -6.7242
                                                         -------
offset to subtract                                        4.6292   (odds x 102)

Concretely: a raw sigmoid of 0.50 out of the trained head has a logit of 0, so its production posterior is sigmoid(0 - 4.6292) = 0.0097. Under 1%. It does not clear t_demote’s 1.96% break-even (The demote threshold falls out of a ratio you can measure), let alone t_remove.

That offset is what the “calibration per policy per language” node in Serving architecture’s diagram is doing.

Calibration means exactly this: when the model says 0.30, roughly thirty items in every hundred it says that about really are violations. It is a separate property from ranking — a model can order items perfectly and still be systematically 100x too high on every one of them — and it is the property every threshold in this chapter silently depends on. Calibration drift across languages and time is what happens when calibration moves and the thresholds do not.

5. Split by time, then by asset and author — a row split leaks up to 10 recall points

Time first. Train on everything up to T, validate on (T, T + 7 days], and evaluate on a gold set drawn after T. The multimodal problem and the arms race’s whole argument is that the adversarial distribution moves, and a random split scores the model on July’s traffic using July’s training data.

Then the duplicate trap, which is specific to this domain and follows from the recidivism infrastructure The multimodal problem and the arms race builds. 31% of violations are near-duplicates of an already-enforced asset. A row-level split puts perceptual-hash neighbours of the same image on both sides of the split, and the model “recalls” them by memorisation.

So split by asset-hash cluster and by author, not by row. Strip out the memorised 31% and what is left is the honest number:

row-split holdout recall                      0.78
of which memorised near-duplicates            0.31
                                              ------
honest recall  =  (0.78 - 0.31) / (1 - 0.31)
               =   0.47 / 0.69
               =   0.68

Ten points of recall that were never earned.

6. Retrain weekly, because the decay rate says so

Fresh-traffic recall falls from 0.78 to 0.51 over six months (Why static benchmarks decay). Assume exponential decay and read off the monthly rate:

monthly factor  =  (0.51 / 0.78) ^ (1/6)
                =  0.6538 ^ 0.1667
                =  0.932          -> you lose 6.8% of your recall per month

weekly factor   =  0.932 ^ (7/30.4)  =  0.984   -> 1.6% per week

Weekly retraining on a rolling 90-day window with time-decayed sample weights holds the loss inside 2%.

The cadence is bounded from below by Calibration drift across languages and time: every retrain moves the calibration, so every retrain must recompute all four thresholds, per policy and per language. thresholds_from_curve and review_threshold are both scheduled jobs rather than config values, for two unrelated reasons — t_review because capacity moves, t_remove because calibration does.

Log the propensity, because review capacity moves it daily

One discipline costs almost nothing to adopt and is impossible to retrofit: recording, next to every labeled row, the probability that the row was allowed to be labeled at all.

That probability is called the propensity.

If your system only produces a label when it decided to act, the labeled set is not a sample of the world. It is a sample of your own past decisions, and the propensity is the only record of how that sampling was done. The general rule: when the label only arrives if the system acted, write the acting policy’s probability next to the row.

This chapter needs the discipline more than most. The label-generating policy is a quantile that Review capacity is the binding constraint recomputes from live reviewer capacity. The queue’s inclusion probability is therefore different today than it was yesterday, and nothing in the historical label set records which.

A concrete case: a reviewer decision from the week when 1,100 reviewers were on loan to a new market was drawn from t_review = 0.62, not 0.55. Mixed into training with no propensity attached, it silently reweights the score region the model learns hardest.

Logging a threshold rule’s propensity is not enough on its own. A deterministic cut gives every row a propensity of exactly 0 or exactly 1, and no amount of reweighting recovers a region that was never sampled at all. So the queue policy has to be slightly stochastic below the cut.

Here is that exploration budget, with what it costs. p is the propensity — the chance any one post in the band gets pulled into the queue — and the last two lines convert the cost into the unit that matters, which is recall:

exploration budget   1% of the 8.15 M proactive capacity  =  81,500 items/day
drawn uniformly from the [0.38, 0.55] band  =  42.0 M posts
logged propensity    p  =  81,500 / 42.0 M  =  0.00194        (1/p = 515)

cost, in the unit that matters: those 81,500 items come OUT of the
proactive queue, off its low-score end where precision is below the
band average of 0.124
    violations forgone   <=  81,500 x 0.124   =  10.1 k/day
    system recall cost   <=  10.1 / 2,400     =  0.0042  =  0.42 percentage points

0.42 points of recall buys a measurement Why you cannot pick one threshold otherwise cannot make, and without which its central question stays open.

Why you cannot pick one threshold ends by saying t_demote has to be found by re-slicing the score range into eight 0.02-wide bins from 0.38 up. That is one resolution, used identically in three places: here, in The demote threshold falls out of a ratio you can measure’s FINE table, and in What the quantile buys.

Ask where the labels for those eight bins would come from and the choice of resolution stops looking arbitrary. Compare the two candidate label sources. The percentages on the right are relative standard error — the spread of a count of rare events divided by the count, which for a count of k is sqrt(k)/k = 1/sqrt(k). Under about 10% you can act on the number; at 19% you cannot:

below t_review, the only adjudicated items are the prevalence sample
    12 k/day x (42.0 M / 2,000 M)          =    252 items/day in the band
    x 0.98% in-band precision              =    2.5 violations/day
    over a quarter (90 days)               =    222 finds, 6.7% relative
    ...but split across 8 bins             =     28 finds, 19% relative

with the exploration sample, stratified across the same 8 bins
    81,500 / 8                             = 10,188 items per bin per day
    x 1.96% at the break-even              =    200 finds, 7.1% relative
                                                PER BIN, PER DAY

The prevalence sample takes a quarter to put one honest number on the whole band, and never reaches a per-bin number at all. The exploration sample delivers a per-bin number every day. That is the difference between “the demote tier is somewhere in 0.38-0.55” and an answer.

Log each bin’s own propensity, because the bins are stratified and their inclusion probabilities are therefore different by construction.

Then the inverse-propensity estimate reproduces the band total as a free check. It is the simplest possible use of a logged propensity: if an item had a 1-in-515 chance of being sampled, each violation you find inside the sample stands for 515 violations in the band it came from, so multiply your finds by 1/p to recover the population total. Run it both ways:

forward:  81,500 sampled x (410 k / 42.0 M) violation rate  =  796 finds/day
backward: 796 finds x (1/p) = 796 x 515                     =  410 k

410 k is What the quantile buys’s own number for that band, arrived at from the other direction.

Assumptions in the training recipe, and the load-bearing one. The load-bearing assumption is that the 4.6292 log-odds offset is the whole difference between the training mixture and production — that is, that downsampling changed the base rate and nothing else about the score distribution. It is exactly true only if the negatives you kept are a uniform sample of the negatives you dropped, and they are not: hard negatives are drawn from above t_review by construction. So treat the offset as the first-order correction and the monotone calibration map fitted on the gold set as the part that cleans up what the offset misses. Secondary assumptions, none of which change a conclusion: exponential decay in the retraining-cadence calculation, the 90-day rolling window, and the 1.34 policies per violating item measured on a 3 k/day gold set.

The code below is this section’s three mechanisms written out — the masked loss, the prior shift, and the inverse-propensity estimate — followed by assertions that re-derive every number quoted above from the stated volumes. The assertions are the point: if you change a volume at the top, the arithmetic downstream has to still hold.

import math

# --- section 2's label sources and the recipe above, executed -----------
REVIEWED_PER_DAY   = 7.19e6      # section 4's review band
POSITIVES_PER_DAY  = 890e3       # violations adjudicated inside it
AUDIT_PER_DAY      = 920e3       # section 7's tier-1 clear audit sample
PREVALENCE_PER_DAY = 12e3        # the uniform-over-views sample
SERVING_PRIOR      = 0.0012      # section 1


def masked_bce(logits, labels, mask, pos_weight):
    """Multi-label BCE over the heads a reviewer ACTUALLY ruled on.

    `mask[k] == 0` means "policy k was not adjudicated for this item",
    which is NOT the same as "policy k was not violated". A reviewer
    closing a spam item at 8 s AHT leaves nine heads unobserved, and on
    the gold set violating items violate 1.34 policies -- so folding
    unobserved into negative mislabels a quarter of the positive
    (item, policy) pairs, on exactly the co-occurring policies the
    multi-label output exists to represent.

    `pos_weight` is per head because label volume is anti-correlated with
    severity (section 5): spam sees millions of adjudications a day and
    child safety rounds to zero of reviewer time.
    """
    total, n = 0.0, 0
    for z, y, m, w in zip(logits, labels, mask, pos_weight):
        if not m:                         # unobserved: contributes nothing
            continue
        p = 1 / (1 + math.exp(-z))
        total += -(w * y * math.log(p) + (1 - y) * math.log(1 - p))
        n += 1
    return total / max(n, 1)


def prior_shift(logit, train_prior, serve_prior=SERVING_PRIOR):
    """Undo the training mixture's base rate, in log-odds.

    Every threshold in section 3 is a cut on P(violates k) at the
    PRODUCTION prior. A model trained on a queue-heavy mixture is
    calibrated to the mixture, so its raw sigmoid is ~100x too high and
    a 0.94 cut on it means nothing.
    """
    return logit - (math.log(train_prior / (1 - train_prior))
                    - math.log(serve_prior / (1 - serve_prior)))


def ipw_violations(found, sampled, band_items):
    """Inverse-propensity estimate of violations in an unreviewed band.

    `sampled / band_items` is the logged propensity. It is only usable
    because the queue policy is deliberately stochastic below t_review:
    a hard threshold gives propensity 0 or 1, and 1/0 is not an estimate.
    """
    return found / (sampled / band_items)


# an unobserved head contributes nothing, however wrong its logit
_w = [1.0] * 3
assert (masked_bce([2.0, 2.0, -9.0], [1, 1, 1], [1, 1, 0], _w)
        == masked_bce([2.0, 2.0, 99.0], [1, 1, 1], [1, 1, 0], _w))

# the training mixture and the offset it forces
_neg = (REVIEWED_PER_DAY - POSITIVES_PER_DAY) + AUDIT_PER_DAY + PREVALENCE_PER_DAY
_train_prior = POSITIVES_PER_DAY / (POSITIVES_PER_DAY + _neg)
assert abs(_train_prior - 0.1096) < 5e-4
assert abs(prior_shift(0.0, _train_prior) + 4.6292) < 1e-3
assert abs(1 / (1 + math.exp(-prior_shift(0.0, _train_prior))) - 0.0097) < 5e-4

# the exploration sample: cost in recall, and what it buys
EXPLORE = 0.01 * 8.15e6
assert abs(EXPLORE - 81_500) < 1
assert abs(EXPLORE * 0.124 / 2400e3 - 0.0042) < 5e-5      # <= 0.42 recall points
assert abs(ipw_violations(796, EXPLORE, 42.0e6) - 410e3) < 1e3
# per bin, one day, at the 1.96% break-even -- 8 bins of 0.02 from 0.38,
# the same eight the FINE table in section 3 is built on
_per_bin = EXPLORE / 8
assert abs(_per_bin - 10_188) < 1
assert abs(100 / math.sqrt(_per_bin * 0.0196) - 7.1) < 0.1  # % relative error

# what the prevalence sample alone can do down there, for comparison
_prev_in_band = PREVALENCE_PER_DAY * 42.0e6 / 2000e6
assert abs(_prev_in_band - 252) < 1
_finds_per_quarter = _prev_in_band * 0.0098 * 90
assert abs(_finds_per_quarter - 222) < 1                   # whole band, 90 days
assert abs(100 / math.sqrt(_finds_per_quarter) - 6.7) < 0.1
assert abs(100 / math.sqrt(_finds_per_quarter / 8) - 19.0) < 0.2   # per bin

# the duplicate-leakage correction: 31% of violations are hash near-dupes
assert abs((0.78 - 0.31) / (1 - 0.31) - 0.681) < 5e-4

3. Why you cannot pick one threshold

Where does an enforcement threshold come from when the textbook rule refuses to produce one? Run the standard cost-matrix calculation to its conclusion and the number it returns is unshippable — not because the arithmetic is wrong but because its central assumption is false here. The four thresholds this system does use come from three different origins, which is why “where did your threshold come from” has to be answered four separate times, with a different answer each time.

A scope note: every number below pools the ten policies

Every number in Why you cannot pick one threshold and Review capacity is the binding constraint is computed on the ten policies pooled, and Framing and the objective that is not multi class just spent its strongest paragraph arguing the pool is meaningless. Say this out loud rather than letting the interviewer find it. 0.0012 is Framing and the objective that is not multi class’s all-policy prevalence — 0.12% of posts violate some policy — and 2.4 M/day is the all-policy violation count. Review capacity is the binding constraint’s band table sums to the same 2,400 k. There is exactly one score distribution in this chapter and it belongs to no policy.

That is a deliberate exposition choice with a real cost, and the two halves are worth separating:

• The shape of every argument transfers. That a precision floor strands the middle of the distribution, that in-band precision and not cumulative precision pays for a marginal tier, that capacity sets a quantile — none of those depend on which policy’s curve you run them on. They are statements about the geometry of a PR curve against a cost structure.
• Not one of the resulting numbers transfers. t_remove = 0.94 is where the pooled curve reaches precision 0.95. Spam reaches 0.95 far lower and hate speech may never reach it at all — which is precisely the remove: None branch the code guards, and The label is the hard part says why: a policy whose single-reviewer accuracy is 0.82 has a measurement ceiling that a 0.95 precision floor can sit above.

So thresholds[policy] in the code is not decoration. Production carries ten score distributions, ten PR curves, and ten sets of four thresholds; this chapter derives one set on the pool because one worked instance teaches the method and ten repetitions teach nothing. Where a per-policy number would change a conclusion rather than a digit, the text says so.

The single-threshold argument, run to its conclusion

Take the standard recipe for choosing a threshold, apply it honestly, and look at what it actually asks you to ship.

The textbook rule (Choosing a threshold from the cost matrix) says: act when P(violation | x) > C_FP / (C_FP + C_FN), where C_FP is what a false positive costs you and C_FN what a false negative costs.

Take the pooled distribution at prevalence 0.0012, and suppose policy leadership says a missed violation is 8x as bad as a wrongful removal — so C_FN = 8 × C_FP. Substitute:

t*  =  C_FP / (C_FP + C_FN)
    =  C_FP / (C_FP + 8 C_FP)
    =  1 / 9
    =  0.111        on a calibrated posterior

Now read the operating point off the PR curve at that threshold and turn it into daily volumes. Recall tells you how many of the 2.4 M real violations you catch; precision tells you what share of everything you removed those catches are, so dividing by precision recovers the total you removed:

at t = 0.111:   recall 0.92,  precision 0.15

violations caught  =  2.4 M × 0.92           =   2.2 M
removals/day       =  2.2 M / 0.15           =  14.7 M
of which wrong     =  14.7 M - 2.2 M         =  12.5 M legitimate posts
                                                removed, per day

No organization will run that system, and the reason is not that the arithmetic is wrong — it is that the arithmetic assumed something false.

A single exchange rate C_FN / C_FP = 8 asserts that you would trade eight wrongful removals for one prevented harm, at the margin, indefinitely. Nobody will sign that. The two costs are borne by different parties; one is a harm you caused and the other is a harm you failed to prevent. They are not fungible in the way an expected-cost minimization requires.

One tier escapes this, and it is worth flagging now, because the derivation two subsections down will otherwise read as a contradiction of the paragraph you just finished.

Exchangeability fails at the removal tier because the two sides of the ledger are different kinds of object. On one side, a harm you failed to prevent, borne by a viewer. On the other, a harm you caused, borne by an author, delivered as a notification, a strike, and an account-level consequence that outlives the post.

Demotion has none of that. It is reversible, invisible, notifies nobody, and its entire effect is views delivered or views not delivered — the same unit on both sides of the trade.

When both sides are views, an exchange rate between them is a claim a survey can actually put to people, and expected-value arithmetic is legitimate again. That is why t_demote gets a break-even and t_remove gets a floor, and it is the only reason the two are treated differently.

So the real formulation is constrained, not scalarized:

maximize   recall
subject to precision >= P_min           (a floor, not a price)
           reviewed_items <= C_review    (a capacity, not a price)

And a precision floor forces the threshold to the high end of the curve, which throws away all the recall in the middle of the distribution. The middle is not worthless — it is just not automatically actionable. That is what creates the third action.

Actions have costs that differ by two orders of magnitude

The single act/do-not-act decision now becomes a menu of six actions, each with a priced false positive — and only one of those prices turns out to be a real price. FP below is false positive: an action taken against a post that did not violate anything.

Two rows of the table are jargon, and they are the two that create most of the cost spread, so gloss them first:

• An interstitial is a full-screen click-through warning — “this post may contain graphic violence · view / go back” — that replaces the content until the viewer opts in.
• An age-gate is the same mechanism conditioned on the account’s declared or inferred age, so a minor cannot opt in at all.

Both are viewer-side, both are instant, and crucially neither touches the author: no notification, no strike, no removal, no appeal. That is why that row costs 0.08 and the removal row costs 1.0. The actions differ far less in what the viewer sees than in what happens to the person who posted.

In the table, the two columns to read together are Relative FP cost c and Precision floor it needs. c prices one wrongful action relative to a wrongful remove-with-strike, which is fixed at 1.0. The c/(1+c) column in between is a formula applied to c, and the next paragraph is about how badly it disagrees with the column beside it.

The c/(1+c) column is there to be checked against the one beside it, and it fails on three rows out of four. That column is the break-even the next subsection derives — plug a relative FP cost c into precision > c/(1+c) and you get the precision at which taking the action stops destroying value. Read the two columns together:

demote     c = 0.02   ->  c/(1+c) = 0.020    floor stated: ~0.02   agrees
age-gate   c = 0.08   ->  c/(1+c) = 0.074    floor stated:  0.55   7.4x apart
remove     c = 0.6    ->  c/(1+c) = 0.375    floor stated:  0.92   2.5x apart
remove+    c = 1.0    ->  c/(1+c) = 0.500    floor stated:  0.95   1.9x apart

So the chapter derives one floor and chooses three, and the code says so out loud one screen down: REMOVE_PRECISION_FLOOR = 0.95 # policy decision, not a tuned number. Do not let a reader — or an interviewer — assume the formula generated the column.

The reason it cannot is the reason from three paragraphs above, now cashed out per row.

c/(1+c) is an expected-value break-even, and expected-value arithmetic needs both sides denominated in the same thing. Only the demote row satisfies that: its cost is suppressed legitimate views and its benefit is prevented violating views.

The moment an action reaches the author, the cost side acquires terms that are not views at all:

• a notification;
• a strike;
• an appeal the user must find and file;
• an account-level penalty that suppresses everything they post afterwards;
• the standing risk that a wrongful removal becomes a public incident.

c = 0.6 is a summary of a bundle, not a price, and you cannot invert a summary.

So the three upper floors come from where Framing and the objective that is not multi class said they would: policy leadership, legal exposure, and what the organization is willing to defend. The 0.95 says “one wrongful removal-with-strike in twenty is the most we will tolerate” — a commitment, not an optimum.

That leaves a clean division worth stating as the answer to “where did your threshold come from”: the invisible reversible tier is derived, the visible irreversible tiers are chosen, and t_review is neither — it is measured off an org chart (Review capacity is the binding constraint). Three origins for four numbers.

Because false-positive cost varies 50x across actions, the optimal threshold varies with it, and the thresholds nest. That is the derivation the interviewer is waiting for. One classifier, one score, four thresholds:

t_remove  = 0.94     set by a precision floor of 0.95
t_review  = 0.55     set by REVIEW CAPACITY, not by the PR curve  (§4)
t_demote  = ?        set by a harm-ratio break-even, and the published
                     0.38 fails it — see below
below                no action

The demote threshold falls out of a ratio you can measure

One threshold in the design is genuinely computed rather than chosen — and the same derivation that computes it shows the value the chapter started with is wrong.

This is the row that earns an expected-value treatment, for the reason set out above: both sides of its ledger are views, so a single number w can honestly stand between them. Nothing that follows would be legitimate one row up.

Demotion at 2% precision sounds absurd until you price it.

Let one violating view cost 1 and one suppressed legitimate view cost w. w is small because demotion is a reach penalty on one post, not a removal. Demotion cuts reach by a factor d, and E[views] below is the average number of views a post in this band would otherwise get.

benefit  =  d × (violating posts in band)   × E[views]  × 1
cost     =  d × (legitimate posts in band)  × E[views]  × w

net > 0  <=>  d × V × E[views] × 1  >  d × L × E[views] × w

Cancel d and E[views] from both sides — they appear identically on each — and you are left with a comparison of two counts:

              V  >  L × w
              V / L  >  w

One more step converts a violating-to-legitimate ratio into precision, which is what you can actually measure. Precision in the band is V / (V + L), so:

V / L > w   <=>   V / (V + L)  >  w / (1 + w)

precision_in_band  >  w / (1 + w)   ≈  w   (for small w)

The d cancelling is the useful part: how hard you demote does not change where the threshold goes, only how much the action is worth.

With w = 0.02, measured from a survey-calibrated harm scale, demotion pays wherever in-band precision exceeds:

w / (1 + w)  =  0.02 / 1.02  =  0.0196  =  1.96%

Now check that against the band table Review capacity is the binding constraint measures, because this is where the derivation kills the number it was supposed to justify:

demote band 0.38 - 0.55:   42.0 M posts,  410 k violations
in-band precision       =  410e3 / 42.0e6  =  0.0098  =  0.98%
break-even              =  w / (1 + w)     =  0.0196  =  1.96%

Demoting the 0.38-0.55 band destroys value, by the same arithmetic that justifies demotion at all. t_demote = 0.38 is half the precision it needs. Three exits, and only three:

• Move the threshold up. In-band precision is 0.98% across [0.38, 0.55] and 12.4% across [0.55, 0.94], so it crosses 1.96% somewhere inside the lower band. Four bands is not enough resolution to say where — re-slice the curve into eight 0.02-wide bins from 0.38 up (the last running 0.52-0.55, since 0.38 + 8 × 0.02 = 0.54 leaves a remainder) and read off the crossing. Eight is not arbitrary: it is what Log the propensity because review capacity moves it daily’s exploration budget can supply a per-bin number for. Everything below the crossing goes to no-action.
• Lower w. w/(1+w) < 0.0098 requires w < 0.0099: a suppressed legitimate view would have to cost under 1% of a violating view. That is a claim about harm, and the survey either supports it or it does not.
• Drop the tier. Forfeiting the middle of the distribution is strictly better than paying for it.

The tempting fourth exit is to quote the cumulative precision above 0.38 — 2,212 k / 50.2 M = 4.4% — and declare the bar cleared. It is a different quantity. The derivation cancels d precisely because it is marginal: the posts above 0.55 are already being removed or reviewed, so their precision cannot pay for an action taken on the band beneath them. Cumulative precision makes a marginal tier look about 4.5x better than it is, and it is the most common way this section of the interview goes wrong.

So the honest state of the design is: the demote tier is right in principle and the published threshold is wrong, and the thing that decides it is a harm ratio somebody has to write down. That is the honest version of “where did your threshold come from” — sometimes the answer it gives you is no.

And the code has to be able to say no. t_demote = None is the sentinel for “no band under t_review clears the break-even,” and on the four-band table it is the correct answer — which means the search for t_demote has to be restricted to bands strictly below t_review, or it returns 0.55, stacks the demote tier exactly on top of the review tier, and reports a threshold where the honest output was a refusal.

The tier map

The four thresholds assemble into a single decision path that every score actually travels, and every volume printed on it can be checked against the band table that produced them.

flowchart TD
    S(["Per-policy score<br/>P(violates k)"]) --> T1{"score >= 0.94<br/>precision >= 0.95"}
    T1 -->|yes| REM["AUTO-REMOVE<br/>+ strike if severity >= 4<br/>0.96 M/day"]
    T1 -->|no| T2{"score >= t_review<br/>SET BY CAPACITY"}
    T2 -->|yes| Q["HUMAN REVIEW QUEUE<br/>prioritized by expected harm<br/>7.2 M/day"]
    T2 -->|no| T3{"score >= t_demote<br/>IN-BAND precision >= 1.96%<br/>not 0.38 — see §3"}
    T3 -->|yes| DEM["DEMOTE · reach -70%<br/>no notification"]
    T3 -->|no| NA(["No action<br/>>= 97.5% of posts"])

    REM --> AP["Appeals path<br/>4.2% appeal · 11% overturn"]
    Q --> RD{"Reviewer decision"}
    RD -->|violating| REM
    RD -->|benign| NA
    RD -->|unclear| ESC["Escalate to<br/>policy specialist"]
    AP --> RD

    style T2 fill:#bc6c25,color:#fff
    style Q fill:#1d3557,color:#fff
    style DEM fill:#2d6a4f,color:#fff
    style REM fill:#9d0208,color:#fff

Read the diagram as one score falling through four gates in order:

1. At or above 0.94 — auto-remove. A strike is attached only when the policy’s severity is 4 or higher, which is why severity appears here and nowhere in the threshold arithmetic.
2. Below that but at or above t_review — go to a human. That cut is set by capacity, not by the curve.
3. Below that but at or above t_demote — demote. A reach penalty of roughly 70%, applied silently, with no notification to the author.
4. Below that — nothing.

On the human side, a reviewer returns one of three verdicts, and the third is the one people forget. Violating sends the item down the same removal path the model uses. Benign clears it. Unclear escalates it to a policy specialist — a senior reviewer who owns the policy document. That specialist’s ruling both settles the item and becomes a worked example in the next policy revision, which is the mechanism by which Inter annotator agreement is the ceiling’s agreement rate improves at all.

Every volume printed on that diagram comes from What the quantile buys’s band table:

AUTO-REMOVE        0.96 M/day   the >= 0.94 band
HUMAN REVIEW QUEUE 7.2  M/day   the 0.55 - 0.94 band, 7.19 M rounded
                   ----------
                   8.15 M/day   = section 4's entire daily reviewer capacity,
                                  because the capacity is what put the edge
                                  at 0.55

Only the 7.2 M reaches a reviewer, though. The auto-remove branch leaves the diagram without a human on it. So 0.96 M/day of that capacity — 11.8% of it — is bought and not spent, which What the quantile buys prices at about $61 M/year.

4.2% appeal · 11% overturn on the appeals arrow is Appeals are part of the design not a support function’s pair: 40,300 appeals and 4,430 overturns a day off that 0.96 M.

No action >= 97.5% of posts is the one to check, and the >= is load-bearing. 97.5% is 1,950 / 2,000 — the count below 0.38 — so it is the no-action share only if the demote tier takes the whole [0.38, 0.55] band. Why you cannot pick one threshold has just shown it must not. If the demote tier is dropped entirely, no-action is 1,991.85 / 2,000 = 99.6%; a t_demote somewhere inside the lower band lands between the two. The diagram is drawn at the optimistic end of a range whose width is the unresolved threshold, which is why it reads >= and not =.

Note also the arrow from appeals back into the review queue. Appeals consume the same finite reviewer capacity as proactive review, which makes the two tiers coupled in a way Review capacity is the binding constraint has to account for.

Assumptions in this section, and the load-bearing one. The load-bearing assumption is the harm ratio w = 0.02, because t_demote is the only threshold that is derived and it moves with w almost one-for-one. It comes from a survey putting a harm scale on suppressed legitimate views against violating views; it is a stated preference, not a measurement, and the whole demote tier stands or falls on it. Three assumptions that look load-bearing and are not: the precision floors of 0.95, 0.92 and 0.55 are chosen commitments, so calling them assumptions understates them — they are decisions, and the text says who makes them; the 8-to-1 cost ratio exists only to be refuted; and the pooled score distribution changes every number and no argument, which the scope note above already prices.

The code below is the tier map as two functions. action_for turns one score into one action; thresholds_from_curve produces the cut points those comparisons use. The docstrings carry the two traps — that t_demote must be searched for strictly below t_review, and that an empty tier has to return None rather than raise — and the assertions at the bottom run this chapter’s own tables through both.

from collections import namedtuple

REMOVE_PRECISION_FLOOR = 0.95     # policy decision, not a tuned number
DEMOTE_HARM_RATIO = 0.02          # cost of a suppressed legitimate view,
                                  # relative to a violating view


def action_for(score, policy, thresholds):
    """One calibrated score, four thresholds, and severity kept separate.

    `thresholds[policy]` carries per-policy, per-version, per-language cut
    points -- ten policies, ten score distributions, ten sets of four cut
    points. There is no global threshold and no global score.

    Severity selects the CONSEQUENCE once a bar has been cleared; it never
    moves the bar. That is why the only place it appears below is choosing
    between "remove" and "remove_with_strike", after the comparison against
    t["remove"] has already succeeded. The ordering half of the lesson --
    that a 0.99 spam score must not outrank a 0.60 child-safety score --
    lives in `queue_priority` in section 4, not here: this function decides
    WHICH action, and that one decides IN WHAT ORDER.

    Both `remove` and `demote` may be None: None means no cut point on that
    tier clears its bar, which is a legitimate state (see below) and not an
    error, so every tier is guarded rather than assumed present.
    """
    t = thresholds[policy]
    if t["remove"] is not None and score >= t["remove"]:
        return "remove_with_strike" if policy.severity >= 4 else "remove"
    if score >= t["review"]:                  # set by capacity, see below
        return "queue"
    if t["demote"] is not None and score >= t["demote"]:
        return "demote"                       # None => no band clears w/(1+w)
    return "none"


def thresholds_from_curve(pr_curve, band_precision, queue_capacity_quantile):
    """remove: from a precision floor.  demote: from a harm ratio.
       review: from a QUANTILE of the score distribution -- an ops number.

    `pr_curve` carries CUMULATIVE precision, which is the right quantity for
    t_remove and the wrong one for t_demote. Demotion acts on a band, so its
    break-even is on that band's own precision: reading t_demote off the
    cumulative curve borrows precision from posts that are already being
    removed or reviewed and lands the threshold far too low -- 0.38 looks
    like 4.4% cumulative and is 0.98% in-band. `band_precision` is
    [(score_low, precision_within_this_band)], ascending.

    Two things the tiers have to obey, and both are easy to lose inside a
    generator expression:

    - **t_demote must sit strictly below t_review.** A band at or above the
      review cut is already being reviewed or removed, so its precision
      cannot pay for an action taken *underneath* it -- the same
      marginal-versus-cumulative error one level up. Without the constraint
      this chapter's own four-band table returns t_demote = 0.55, which
      collapses the demote tier onto t_review and deletes it silently.
    - **an empty tier is None, not an exception.** `min()` over an empty
      generator raises, so a model whose curve never reaches the precision
      floor takes down the threshold job instead of reporting that the
      auto-remove tier does not open. Both `remove` and `demote` therefore
      pass `default=None`, and every caller has to guard for it.
    """
    breakeven = DEMOTE_HARM_RATIO / (1 + DEMOTE_HARM_RATIO)
    t_review = queue_capacity_quantile
    return {
        "remove": min((s for s, p, _ in pr_curve
                       if p >= REMOVE_PRECISION_FLOOR), default=None),
        "demote": min((s for s, p in band_precision
                       if p >= breakeven and s < t_review), default=None),
        "review": t_review,                 # NOT read off pr_curve
    }


# --- the chapter's own tables, executed ---------------------------------
# (score_low, CUMULATIVE precision above it, cumulative recall)
PR_CURVE = [(0.38, 0.0441, 0.922), (0.55, 0.2211, 0.751), (0.94, 0.950, 0.380)]
# (score_low, precision WITHIN the band) -- section 4's four-band table
BANDS = [(0.38, 0.0098), (0.55, 0.124), (0.94, 0.950)]

_t = thresholds_from_curve(PR_CURVE, BANDS, 0.55)
assert _t["remove"] == 0.94
assert _t["review"] == 0.55
assert _t["demote"] is None            # 0.98% in-band against a 1.96% bar
assert DEMOTE_HARM_RATIO / (1 + DEMOTE_HARM_RATIO) > BANDS[0][1]

# Re-sliced at 0.02, a band below t_review does clear the bar, and t_demote
# lands there -- strictly below t_review, which is the tier's whole point.
#
# ONE resolution, stated once and used everywhere: EIGHT bins, 0.02 wide,
# from 0.38 up. Eight is what section 2's exploration budget is stratified
# across (81,500 / 8 = 10,188 items per bin per day, which is the 7.1%
# relative error that makes a per-bin number possible at all), and 0.02 is
# what the text calls them. 0.38 + 8 x 0.02 = 0.54, so the last bin runs
# 0.52-0.55 and is 0.03 wide, absorbing the remainder up to t_review. That
# is the only irregularity, and it is stated rather than left to be
# discovered by subtracting the edges.
FINE = [(0.38, 0.0060), (0.40, 0.0078), (0.42, 0.0101), (0.44, 0.0131),
        (0.46, 0.0169), (0.48, 0.0207), (0.50, 0.0243), (0.52, 0.0286),
        (0.55, 0.124), (0.94, 0.950)]
assert len([s for s, _ in FINE if s < 0.55]) == 8       # the eight bins
_tf = thresholds_from_curve(PR_CURVE, FINE, 0.55)
assert _tf["demote"] == 0.48
assert _tf["demote"] < _tf["review"]

# a model whose curve never reaches the precision floor reports an empty
# tier instead of raising.
assert thresholds_from_curve([(0.94, 0.90, 0.38)], BANDS, 0.55)["remove"] is None

Policy = namedtuple("Policy", "name severity")
_hate = Policy("hate_speech", 3)
_child = Policy("child_safety", 5)
_TH = {_hate: _t, _child: dict(_t, remove=None)}
assert action_for(0.99, _hate, _TH) == "remove"
assert action_for(0.60, _hate, _TH) == "queue"
assert action_for(0.45, _hate, _TH) == "none"        # demote is None: no crash
assert action_for(0.99, _child, _TH) == "queue"      # remove tier is closed
assert action_for(0.99, Policy("cs", 5), {Policy("cs", 5): dict(_t)}) \
    == "remove_with_strike"

4. Review capacity is the binding constraint

The third threshold’s origin is not a machine learning question at all: you can compute a review threshold from a staffing table, show how far a staffing decision moves it, and order the review queue by something better than score.

This is what separates a systems answer from a modeling answer. t_review is not read off a PR curve. It is read off an org chart.

The block below is that derivation in three steps: total reviewer-seconds available per day, the average seconds one item costs, and the ratio between them. The middle step is a weighted average — each policy’s handle time times that policy’s share of the queue — so the products are written out for checking:

reviewers                                       15,000
productive hours per shift                       6.5    (mandated wellness
                                                         breaks on graphic
                                                         queues, calibration
                                                         tests, training)
reviewer-seconds per day  = 15,000 × 6.5 × 3600  =  351.0 M

average handle time, weighted by queue mix:
    spam                     8 s   × 0.31  =   2.48
    nudity                  12 s   × 0.19  =   2.28
    graphic violence        22 s   × 0.14  =   3.08
    hate speech             45 s   × 0.21  =   9.45
    harassment              58 s   × 0.09  =   5.22
    violent extremism       90 s   × 0.06  =   5.40
                                             ------
    weighted AHT                             27.91  ->  call it 28.0 s

raw capacity   =  351.0 M / 28.0     =  12.54 M items/day
minus appeals, audits, gold-set calibration, escalations   -35%
    12.54 M × 0.65                                          --------
available for the proactive queue    =   8.15 M items/day

Now express that as a share of the platform:

8.15 M / 2,000 M  =  0.0041  =  0.41 % of all content clears t_review

t_review is therefore whatever score cuts the top 0.41% of the score distribution. It is a quantile, not a precision target.

A quantile is a cut point defined purely by the share of the distribution lying above it — “the top 0.41%” — rather than by any property of the score itself.

That distinction is the whole section. A precision target says something about how good the model is. A quantile says something about how many people you employ. Everything else about the review tier follows from that one number.

What the quantile buys

Cash the 0.41% out into items, violations and precision, band by band, and every later argument about recall has an arithmetic base to stand on.

Score-band decomposition, all ten policies pooled (Why you cannot pick one threshold’s scope note), per day. The 2,400 k total row is Framing and the objective that is not multi class’s all-policy violation count, so this table is the pooled distribution and not any policy’s own:

The auto-remove and review bands together are 0.96 + 7.19 = 8.15 M items, which is exactly the capacity. That is not a coincidence — the capacity picked the band edge.

Two recall numbers fall out of the table, and they differ because reviewers are not perfect. The first is the ceiling: everything the system even looks at. The second subtracts the violations reviewers see and clear anyway, at a measured catch rate of about 88%:

reviewed recall ceiling  =  (912 + 890) / 2,400
                         =  1,802 / 2,400        =  0.751

reviewers catch ~88% of the review band's violations:
    890 × 0.88  =  783.2 k

net proactive recall     =  (912 + 783.2) / 2,400
                         =  1,695.2 / 2,400      =  0.706  ->  0.71

Now separate the two quantities that “capacity” has been standing for, because the top band of the 0.41% never reaches a reviewer. The >= 0.94 band is auto-removed by the model with no human in it at all. So the quantile and the queue are different populations:

capacity available for the proactive queue        8.15 M/day   = 0.41 % of posts
   of which auto-removed, no reviewer involved    0.96 M/day
   of which actually worked by a human            7.19 M/day   = 0.36 % of posts
                                                  ----------
unspent capacity                                  0.96 M/day   = 11.8 % of it

t_review = 0.55 is cutting the 0.41%, and the number of items a human sees is the 0.36%. The gap is not a rounding difference; it is 11.8% of the only constraint this chapter says binds, and it costs what that constraint costs:

0.118 × 35.59 M productive hours  =  4.19 M hours
4.19 M × $14.61/productive hour   =  $61 M/year of reviewer capacity
                                     bought and not spent  (§7 prices the rate)

There is no reading that rescues it. Auto-removals consume no review time by construction — that is what “auto” means. And the appeals they generate are already inside this section’s 35% deduction, along with audits and gold-set calibration, so they cannot be counted a second time here.

There are exactly two honest resolutions, and the chapter should say which it is taking.

1. State the working capacity as 7.19 M and treat 8.15 M as the quantile’s population. This is what every number below does.
2. Lower t_review until the review band alone is 8.15 M. You need 0.96 M more items out of the 42.0 M sitting in the 0.17-wide band beneath the cut, so move the cut down by that fraction of the band’s width:

0.55  -  (0.96 / 42.0) × 0.17  =  0.55 - 0.0039  =  0.546

That buys about 9 k more violations a day at the band’s marginal precision, and it is the version an operations team would actually run.

The rest of this chapter uses the first resolution, because it is the one The tier map’s diagram and The cascade priced’s reconciliation are drawn against.

The third row is the one to read twice: it is the row that fails Why you cannot pick one threshold’s break-even, and it is failing at this band resolution. That is the argument for measuring band precision in the eight 0.02-wide slices of The demote threshold falls out of a ratio you can measure rather than in quarters of the score range — a coarse table cannot locate a threshold whose whole job is to sit at a crossing.

The constraint is live, and it moves

One staffing shock, worked through the whole chain — reviewers reassigned, capacity down, threshold up, recall down — shows that the threshold is a live number rather than a configuration value.

Every line in the block below is the previous section’s formula re-run with two inputs changed: 13,900 reviewers instead of 15,000, and a handle time 9% higher because the remaining staff are covering unfamiliar queues.

scenario: a new market launches; 1,100 reviewers reassigned to build a
          language-specific queue, and AHT rises 9% while they ramp

capacity   =  (15,000 - 1,100) × 6.5 × 3600 / (28.0 × 1.09) × 0.65
           =  325.3 M / 30.5 × 0.65
           =  6.93 M items/day        (-15%)

what still fits, after the 0.96 M auto-remove band takes its share:
    review band now  =  6.93 - 0.96                =  5.97 M
    items dropped    =  7.19 - 5.97                =  1.22 M
    as a share       =  1.22 / 7.19                =  17%

violations falling out, at the band's average precision of 0.124:
    1.22 M × 0.124                                 <=  151 k/day

The new threshold is 0.62, and getting that second digit requires interpolating inside a band rather than stepping between bands.

The cut has to land wherever 17% of the [0.55, 0.94] band sits. That band is 0.39 wide, so:

0.55  +  0.17 × 0.39  =  0.55 + 0.066  =  0.616  ->  0.62

That number only exists if the threshold is interpolated inside the band. A routine that consumes whole buckets and stops can only ever return a bucket edge, so against this four-bucket table it would answer 0.94, queue 0.96 M items, and leave 5.97 M items’ worth of reviewers idle. The threshold would become a step function of capacity, which is precisely what you cannot afford when capacity is a live, noisy measurement.

The 151 k is a bound, not an estimate. The dropped 1.22 M come off the low-score end of the band, where precision is below the band average of 0.124. So multiplying by the band average is a ceiling on what falls out — anything worse would require precision to rise as score falls.

Convert that bound into recall:

recall lost  =  151 k / 2,400 k  =  0.063

system recall    0.751  ->  >= 0.688       a drop of at most 6.3 points

A staffing decision moved a model threshold by 0.07 and cost up to 6.3 points of recall, and no model changed. Say this out loud in the interview. The corollary is that t_review must be computed from live queue depth on a schedule, not configured — a static threshold with a shrinking queue silently grows a backlog until items age out unreviewed, which is the worst of both worlds because you paid the latency and got no decision.

Prioritize the queue by expected harm, not by score

Which items enter the queue is one decision; the order in which it is worked is another, and it is worth 2.3x on the metric that matters.

The obvious ordering is descending score. It is wrong, and the gap is large.

item A   score 0.90   projected remaining views     12   severity 2
item B   score 0.58   projected remaining views 400,000   severity 3

expected harm averted  =  P(violation) × projected_views × severity
    A  =  0.90 ×      12 × 2  =         22
    B  =  0.58 × 400,000 × 3  =    696,000

Reviewing in score order puts A first, and A is worth 22 units against B’s 696,000.

Reviewer capacity is a fixed budget of attention, so it should be spent where the integral of harm is largest, and that integral is dominated by projected reach. Simulated on one day’s queue at fixed 8.15 M capacity, expected-harm ordering averts 2.3x the violating views of score ordering.

The projected-views model does not need to be good. It is a simple regression on author follower count, early velocity, and recommendation eligibility, and all it has to do is separate 12 from 400,000 — which any model does.

Assumptions in this section, and the two that are load-bearing. The load-bearing pair are the weighted average handle time of 28.0 seconds and the 35% deduction for appeals, audits, gold-set calibration and escalations, because capacity is linear in both and the threshold is a quantile of capacity. Move handle time to 34 seconds and capacity falls to about 6.7 M/day, which is a larger shock than the entire staffing scenario above. Not load-bearing, though they carry big-looking numbers: the 15,000 figure is a daily on-shift staffing level rather than a headcount (The cascade priced makes that distinction and prices the difference), and the 6.5 productive hours per shift is an input to both sides of most of the ratios that use it.

The code below is the two decisions this section makes, as two functions. review_threshold turns a capacity number into a cut point by walking the score histogram top-down until the budget runs out — and the important line is the interpolation at the end, which is what keeps the threshold from being a step function. queue_priority is the ordering rule. The assertions afterwards replay the staffing shock and check the threshold moves smoothly across the whole plausible capacity range.

def review_threshold(score_histogram, capacity_per_day):
    """t_review is a quantile of the live score distribution, recomputed on a
    schedule from measured capacity. Configuring it as a constant is how a
    queue silently grows a backlog until items age out unreviewed -- you pay
    the latency and get no decision.

    `score_histogram` is [(bucket_low, bucket_high, count_per_day)],
    descending in score.

    **Interpolate inside the bucket the capacity runs out in.** Consuming
    whole buckets and stopping makes the threshold a step function of
    capacity: with this chapter's four buckets it can only ever return 0.94,
    0.55, 0.38 or 0.0, so at the stated 8.15 M it returns 0.55 only because
    7.19 M is not strictly greater than 7.19 M, and a 0.1 % capacity dip
    flips it to 0.94 -- queuing 0.96 M items and idling 5.97 M items' worth
    of reviewer capacity. Linear-in-score interpolation makes it continuous
    in capacity, which is what makes it safe to recompute from a live, noisy
    number.
    """
    budget = capacity_per_day
    cut = score_histogram[0][1] if score_histogram else 1.0
    for low, high, count in score_histogram:
        if count <= budget:
            budget -= count
            cut = low
            continue
        return high - (budget / count) * (high - low)
    return cut


def queue_priority(item):
    """Expected violating views averted, not score.

    Score cannot span much: everything in the review band is inside
    [0.55, 0.94], a ratio of 1.7x, and the worked example's own 0.90 vs 0.58
    is 1.55x. Reach spans 33,000x in that same example. So reach dominates
    the ordering by four orders to under one, and a projected-views model
    only has to separate 12 from 400,000 to capture most of the 2.3x.

    `projected_views` is LIFETIME views, not remaining -- the subtraction
    below is what makes it remaining. That distinction is the whole point:
    views already delivered are unrecoverable harm (section 6), so they buy
    no priority.
    """
    remaining = max(item.projected_views - item.views_so_far, 0)
    return item.score * remaining * item.severity


# --- section 4's own band table, executed -------------------------------
HISTOGRAM = [(0.94, 1.00, 0.96e6), (0.55, 0.94, 7.19e6),
             (0.38, 0.55, 42.0e6), (0.00, 0.38, 1950e6)]
CAPACITY = 8.15e6

assert abs(review_threshold(HISTOGRAM, CAPACITY) - 0.55) < 1e-9

# The quantile and the queue are two populations, and the gap is real money.
# t_review cuts the top 0.41%; the top band of that is auto-removed with no
# reviewer in it, so a human sees 0.36% and 11.8% of the stated capacity is
# bought and not spent.
AUTO_REMOVED = 0.96e6                        # the >= 0.94 band
HUMAN_SEEN = CAPACITY - AUTO_REMOVED         # 7.19 M, section 4's review band
assert abs(CAPACITY / 2000e6 - 0.0041) < 5e-5            # what the cut is
assert abs(HUMAN_SEEN / 2000e6 - 0.0036) < 5e-5          # what a human sees
assert abs(AUTO_REMOVED / CAPACITY - 0.118) < 5e-4       # unspent share
assert abs(AUTO_REMOVED / CAPACITY * 520e6 / 1e6 - 61.2) < 0.5   # $M/year
# and the t_review that would actually spend it, interpolated into the band
# below: 0.96 M more items out of 42.0 M spread over a 0.17-wide band.
assert abs(0.55 - (AUTO_REMOVED / 42.0e6) * 0.17 - 0.546) < 5e-4

# +/-1% of capacity moves the threshold by ~0.005, not by 0.39:
assert abs(review_threshold(HISTOGRAM, 0.99 * CAPACITY) - 0.55) < 0.01
assert abs(review_threshold(HISTOGRAM, 1.01 * CAPACITY) - 0.55) < 0.01
# and the knife edge is gone: a 0.1% dip must not move it by 0.001
assert abs(review_threshold(HISTOGRAM, 0.999 * CAPACITY) - 0.55) < 1e-3

# the staffing scenario: 6.93 M/day of capacity
_t = review_threshold(HISTOGRAM, 6.93e6)
assert abs(_t - 0.616) < 5e-4
_dropped = 7.19e6 * (_t - 0.55) / (0.94 - 0.55)
assert abs(_dropped - 1.22e6) < 0.01e6
assert abs(_dropped / 7.19e6 - 0.17) < 5e-3

# continuity: no step anywhere across the plausible capacity range
_grid = [review_threshold(HISTOGRAM, c * 1e6) for c in
         [5.0 + 0.05 * i for i in range(80)]]
assert max(abs(b - a) for a, b in zip(_grid, _grid[1:])) < 0.005

5. The multimodal problem and the arms race

Now the model itself gets built, and the order of construction is not arbitrary: the modality mix decides the architecture, the architecture decides Serving architecture’s per-item costs, and the arms race decides which parts of it are allowed to be frozen.

Modalities

What does the input actually consist of, how much does each kind cost to score, and which kind forces the architecture? A modality is just a channel of input — text, image, video, audio — and multimodal means one model reading several of them together.

Four evasion techniques appear in the table’s right-hand column, and all four are attacks on how text is turned into tokens:

• Leetspeak — replace letters with lookalike digits or symbols: h4te.
• Homoglyphs — swap in characters from other alphabets that render identically: a Cyrillic а for a Latin a.
• Zero-width joiners — invisible characters inserted between letters, so a word that looks intact to a human is several fragments to a tokenizer.
• Algospeak — community-invented substitute vocabulary, adopted specifically because the moderation system does not know it yet: “unalive” for “kill.”

All four leave the post readable to a person and unrecognisable to a word-based model. That is the mechanism the rest of this section is built around.

In the table, the column that decides the architecture is the third one, Cost/item — it spans 500x, which is why the design is a cascade — and the row that decides it is image + text.

The image-plus-text row is the one that forces a fused model rather than an ensemble of unimodal ones. A picture of a person and the caption “this one” are each innocuous. Together with the preceding post, they are a targeted threat.

Late fusion means scoring the image and the text separately and combining the two numbers at the end. It cannot represent a conjunction that exists in neither channel, because by the time the two numbers meet, everything that made them a threat together has already been discarded.

Cross-attention fusion can. It lets each image patch attend to each text token while both are still full representations rather than scores, so “this one” can be bound to the face it is pointing at.

It costs roughly 1.7x the image encoder alone. That is affordable only because it runs on the 8% of posts that survive the cheap cascade (Serving architecture), not on all two billion.

The trunk, named

Multi label per policy and why multi class is wrong said “independent sigmoid heads on a shared trunk” and did not say what the trunk is. Here it is — every component named and sized, with the per-item cost each piece is answerable for in The cascade priced’s cascade, and every cost reproducible from four design choices.

Five notations in the block need unpacking first:

• GFLOP — a billion floating-point operations. The standard unit for “how much arithmetic does one forward pass cost.”
• ViT-B/16 — a Vision Transformer of the “base” size. It cuts a 224x224 image into 16x16-pixel patches and treats each patch as a token. That is where its 197 tokens come from: (224/16)^2 = 196 patches, plus one summary token.
• Log-mel frames — short slices of audio converted into a frequency picture, on a scale that matches how human hearing spaces pitches. 1,500 of them is a few seconds of sound expressed as a sequence the same machinery can read.
• Embedding — a fixed-length list of numbers a model uses to represent one thing, here 768 per token or patch, learned so that things the model should treat alike land close together. The embedding table in the block below is the lookup that turns each of 250 k token ids into its starting 384-number row: 250,000 × 384 = 96 M parameters of pure memory and almost no arithmetic. The input embedding referred to later in this section is exactly that first vector-per-token representation — the layer a tokenizer attack lands in, and the reason a frozen trunk cannot be patched from the heads.
• fp16 — 16-bit floating point, two bytes per parameter. That is what turns 241 M parameters into 241 × 2 bytes = 0.48 GB.

Read the block as four towers plus a fusion stage, with a running cost per item on the right. The line that matters is the last one: image-plus-text costs 1.68x the image tower alone, which is the price of the cross-attention that makes the conjunction case representable.

tier 1   linear model over hashed character 3-5-grams, 2^20 buckets,
         ONE binary "route to tier 2" output, no per-policy score
                                                      0.4 ms CPU

tier 2   text tower     8 layers x 384 wide, 192 tokens      5.9 GFLOP
         image tower    ViT-B/16, 224x224, 197 tokens       34.9
         audio tower    8 layers x 512 wide, 1,500 log-mel
                        frames                             112.4
         fusion         3 cross-attention layers x 768 wide
                        over the concatenated sequence      17.9
         heads          10 x sigmoid(w_k . h + b_k)          ~0
         ------------------------------------------------------
         image + text   5.9 + 34.9 + 17.9  =  58.7  =  1.68x the image
                                                       tower alone

         (every figure above is FLOPs, i.e. MACs x 2 -- the image tower's
          17.4 G MACs is 34.9 GFLOP, which is the same number ml-sd/02
          derives for the identical ViT-B/16 at 224 px)

         241 M parameters, of which 96 M is a 250 k-token multilingual
         embedding table -- memory, not FLOPs.  0.48 GB at fp16.

Those are the chapter’s own numbers arriving from the other direction. Serving architecture’s cascade asserts 6, 35, 59, 690 and 112 GFLOP per item, and the paragraph above asserts “roughly 1.7x.” Four choices — a depth, a width, a sequence length and a frame count — reproduce all six figures.

That is the point of writing the configuration down. Serving architecture’s cost table stops being a column of constants you have to take on trust and becomes a consequence of those four choices.

Only the last of the four is worth arguing about. Video is 70% of the tier-2 bill (9,660 / 13,705), and one video is 690 / 34.9 = 19.8 frames’ worth of the same image tower. So the frame count is the entire video cost story, which is why Serving architecture’s sampling policy is the lever there and nothing else is.

The factor of two: MACs are not FLOPs

A table whose whole purpose is to stop being taken on trust cannot be off by 2x in its unit.

A MAC is one multiply plus one add, and therefore two floating-point operations. So a MAC count is not a FLOP count. Counting 12 d^2 per layer per token — the standard shortcut for a transformer layer’s six weight matrices — gives you MACs. The column above is those MACs doubled.

The check that the doubling is right is external. ml-sd/02 prices the identical ViT-B/16 at 224 px and derives 34.9 GFLOP with the doubling, which is exactly twice the 17.4 that a MAC count returns. Two chapters, one network, one number.

Nothing downstream moves much either way: the H100 line lands at 0.88 GPU-days instead of 0.44, and is still under one GPU.

Tier 1, the model that disposes of 92% of the platform

Tier 1 deserves a paragraph rather than a clause.

A character n-gram is a run of n consecutive characters — hat, ate, te — and hashing them means running each run through a hash function to get a bucket index. An unbounded vocabulary becomes a fixed-size array of 2^20 weights with no dictionary to maintain.

It is deliberately not a small version of tier 2, for two reasons.

It survives the tokenizer attacks better than a word vocabulary would. Inserting one invisible character breaks exactly one word token, but leaves most of the surrounding character runs intact.

Its output is a routing decision rather than a posterior. That is why Serving architecture has to fold its cleared items wholesale into the bottom score band instead of distributing them across the score range — and why it gets away with that only because the bottom band’s action is “nothing,” which is also tier 1’s verdict.

Its operating point is a recall target, because its misses are unrecoverable. Nothing downstream ever looks at what tier 1 clears. The chapter’s own numbers say what that recall is:

violations below 0.38 (§4)                          188 k/day
tier-1 clears' share of that row (§7)         1,840 / 1,950
tier-1 misses                                       177 k/day
tier-1 recall  =  1 - 177/2,400                     0.926

measured by the 920 k/day audit sample at that miss rate:
    88.7 finds/day, Poisson sd 9.4  =  10.6% relative, daily

Poisson sd in that block is the standard deviation of a count of rare independent events, which for a count of k is simply the square root of k — so 88.7 expected finds carries a spread of about 9.4, and 9.4 out of 88.7 is the 10.6% relative error. The 0.05% audit rate in Serving architecture is not a round number — it is sized so that tier 1’s miss rate is measurable in one day to about a tenth of itself.

One trunk rather than ten models

Framing and the objective that is not multi class spent its strongest paragraph arguing the ten policies have nothing in common, and now the architecture shares a body between them. Both are right, because what they share is upstream of the split.

The evidence — the pixels, the tokens, the thread, the author — is shared. The consequence is not. Share the evidence, separate the consequence, and the head layout of Framing and the objective that is not multi class is exactly that boundary drawn in the architecture.

The concrete payoff is label volume, and it runs the opposite way to severity. Split Review capacity is the binding constraint’s 7.19 M review band by its own queue mix, and read the two right-hand columns against each other:

spam                 0.31  ->  2.23 M reviewed items/day    severity 1
hate speech          0.21  ->  1.51 M                       severity 3
adult nudity         0.19  ->  1.37 M                       severity 3
graphic violence     0.14  ->  1.01 M                       severity 2
harassment           0.09  ->  0.65 M                       severity 3
violent extremism    0.06  ->  0.43 M                       severity 5

child safety         absent from §4's mix entirely: it rounds to zero
                     of reviewer time, and it is severity 5

Label volume is anti-correlated with severity, so the heads that matter most are the heads with the least data.

Ten independent models would give the child-safety model its own encoder and its own handful of positives to train it on. A shared trunk hands it a representation paid for by spam’s 2.23 M adjudications a day, and leaves it 768 weights + 1 bias = 769 parameters to fit.

Note that this is a data argument and not a compute one. Ten encoders would multiply Serving architecture’s GPU line by ten, which is $11 M against a $520 M reviewer bill — affordable, so compute is not what decides it.

The second reason is cheaper to state and also real: Why you cannot pick one threshold’s tier map compares ten scores against ten sets of thresholds on every post, and one trunk produces all ten in one pass.

Why not a frozen general-purpose encoder

The obvious alternative is to take an off-the-shelf vision-language encoder, freeze it, and train only ten small heads on top. It is cheaper to own, needs no trunk training, and would sidestep the whole retrain cadence.

It fails on the mechanism two subsections down. Leetspeak, homoglyphs, zero-width joiners and algospeak are attacks on tokenization. They land in the input embedding, below every head, and no amount of head retraining reaches them. A frozen trunk freezes precisely the layer that has to track the adversary.

That forces two commitments rather than one: a character-aware tokenizer, and a trunk retrained on From those sources to a training set’s weekly cadence. It is also the reason the text tower is 8 shallow layers over 192 tokens rather than something deeper over a fixed vocabulary.

“Character-aware” here means byte-level byte-pair encoding (BPE): a tokenizer that starts from raw bytes and repeatedly merges the most frequent adjacent pair into a new token, so its vocabulary is learned from the data as byte sequences rather than fixed as a word list. Three consequences, all of which matter here:

• every possible input has a representation;
• an unknown word degrades into its byte fragments instead of collapsing to a single “unknown” symbol;
• a homoglyph substitution changes the byte sequence locally rather than destroying the whole word.

What the model eats

Here are the model’s actual inputs, family by family, with sizes — and the point of the list is the one family that is empty.

Framework Stage 4’s families, instantiated — and the mapping is not one-to-one, because the entity here is the post, the “user” is the author, and the cross family is the author-target relation. One row comes out empty, and the empty row reorganizes three other sections of this chapter. Two abbreviations appear in it. OCR in the first row is optical character recognition, the step that reads text baked into an image so it can be scored as text. CTR in the last row is click-through rate, the share of the people shown something who clicked it — the standard engagement counter, and the reason that row is worth naming even though it is empty.

The tabular side is 44 floats, 176 bytes, one read keyed on author id; everything else is the post itself. Tier 1 sees none of the 44 — hashed n-grams only — which is the concrete reason Serving architecture can say tier 1 and tier 2 are not co-calibrated.

The empty row is the structural fact of this problem. In every recommendation problem in this track the item has a history and the user is the variable. In integrity the item has no history at all, by construction, because you are scoring it at the moment of publication.

Three things elsewhere in the chapter are consequences of that one empty row rather than separate ideas:

• Review capacity is the binding constraint’s queue priority has to use projected views from author-side features, because measured views do not exist yet.
• Coordinated evasion’s cluster detector runs offline on a 15-minute cadence, because co-post timing is a counter that must accumulate before it exists.
• The recidivism hashes are the only item-side history available at t = 0 — and even those are the asset’s history, not the post’s.

The one row that can close the feedback loop

“Prior enforcement” is the row that can quietly close The label feedback loop’s loop, and the fix is in the word “adjudicated.”

A feature counting the author’s prior removals is trained on labels produced by the same classifier that produced those removals. A wrongly-struck account gets a higher prior next time. That is Dialect and demographic bias in enforcement the serious one’s compounding step wearing a feature’s clothes.

Two controls, and you need both:

1. Point-in-time correctness — count only enforcement whose decision timestamp precedes the post’s creation. This is the standard counter discipline from Temporal features and lookahead leakage. It is necessary and not sufficient, because a point-in-time-correct count of the model’s own mistakes is still a count of the model’s own mistakes.
2. Adjudicated enforcement only — this is the one that breaks the loop. Count reviewer-confirmed removals, which are 0.78 M/day of Metrics prevalence and why accuracy is meaningless’s 1.74 M total. You keep 0.78 / 1.74 = 45% of the signal and discard exactly the half the model generated by itself.

Then monitor the false-positive rate — FPR from here on, meaning the share of the posts you acted against that should not have been touched — by prior-enforcement bucket crossed with dialect. That cross is where the loop would surface first, and neither margin on its own would show it.

The code below is the FLOP model that produced every cost figure in this section, plus assertions tying each one back to the stated column. Three notations in it:

• MAC — multiply-accumulate, one multiply plus one add. The natural unit for matrix arithmetic, and two floating-point operations.
• qkvo — the four projection matrices inside a transformer layer: query, key, value, output.
• FFN — the feed-forward network, the two-matrix block that follows attention in every layer.

def encoder_gflop(layers, width, tokens):
    """Forward FLOPs for a transformer encoder, in G.

    Counted in MACs and then DOUBLED, because a MAC is one multiply plus
    one add and therefore two floating-point operations. Dropping that
    factor of two is how a column labelled GFLOP ends up holding MACs:
    this configuration returns 17.4 G MACs for ViT-B/16 at 224 px, and
    ml-sd/02 derives 34.9 GFLOP for the identical network. Same network,
    same quantity, and the two chapters have to agree.

    Two terms, and which one dominates decides the design: the
    projections (qkvo plus two FFN matrices, 12 d^2 per layer) are
    linear in sequence length, the attention term is quadratic. At 192
    text tokens the quadratic term is 8% of the text tower; at 1,500
    audio frames it is 33% of the audio tower, which is why the audio
    path is the one that would need windowing if it grew.
    """
    projections = 12 * width * width * layers * tokens
    attention = 2 * tokens * tokens * width * layers
    return 2 * (projections + attention) / 1e9


def params_m(layers, width):
    return 12 * width * width * layers / 1e6


TEXT = encoder_gflop(8, 384, 192)
IMAGE = encoder_gflop(12, 768, 197)            # ViT-B/16 at 224x224
FUSION = encoder_gflop(3, 768, 197 + 192)
AUDIO = encoder_gflop(8, 512, 1500)
VIDEO_FRAMES = 18                              # 1 fps over the mean clip
VIDEO = (VIDEO_FRAMES * IMAGE + TEXT
         + encoder_gflop(3, 768, VIDEO_FRAMES * 49 + 192))  # 49 = pooled 2x2


def within(computed, stated, tol=0.05):
    return abs(computed - stated) <= tol * stated


# --- the configuration reproduces section 5's cost column ---------------
assert within(TEXT, 6)                          # text-only row
assert within(IMAGE, 35)                        # image row
assert within(TEXT + IMAGE + FUSION, 59)        # image+text row
assert within((TEXT + IMAGE + FUSION) / IMAGE, 1.7)     # "roughly 1.7x"
assert within(AUDIO, 112)                       # audio row
assert within(VIDEO, 690)                       # video row

# the unit check that made every figure above double: MACs are not FLOPs,
# and ml-sd/02 prices the same ViT-B/16 at 224 px at 34.9 GFLOP.
assert abs(IMAGE / 2 - 17.45) < 0.05            # the MAC count, for reference
assert abs(IMAGE - 34.9) < 0.05                 # what ml-sd/02 derives

# --- and the frame count is the only lever that moves the GPU bill -----
VIDEO_14 = 14 * IMAGE + TEXT + encoder_gflop(3, 768, 14 * 49 + 192)
assert abs((1 - VIDEO_14 / VIDEO) - 0.219) < 5e-3           # section 7's 14 frames
assert abs((1 - VIDEO_14 / VIDEO) * 9660 / 13705 - 0.153) < 5e-3  # of the whole bill

# --- size and throughput ------------------------------------------------
TOTAL_M = (params_m(8, 384) + 250_000 * 384 / 1e6 + params_m(12, 768)
           + params_m(3, 768) + params_m(8, 512) + 10 * 768 / 1e6)
assert abs(TOTAL_M - 241.5) < 1.0
assert abs(TOTAL_M * 2 / 1000 - 0.48) < 0.01                # GB at fp16
assert abs(2000e6 * 0.08 / 86_400 - 1852) < 2               # avg items/s, tier 2
assert abs(70_000 * 0.08 - 5600) < 1                        # peak items/s

# --- tier 1's operating point, from section 4 and section 7 -------------
_t1_misses = 188e3 * 1840 / 1950
assert abs(1 - _t1_misses / 2400e3 - 0.926) < 5e-4          # tier-1 recall
_t1_finds = 920e3 * _t1_misses / 1840e6          # section 7's audit sample
assert abs(_t1_finds - 88.7) < 0.5                          # audit finds/day
assert abs(100 / _t1_finds ** 0.5 - 10.6) < 0.1             # % relative, daily

Why static benchmarks decay

A fixed evaluation set stops measuring anything in this domain, and that is a property of adversaries rather than a mistake in how the set was built.

The mechanism is uncomfortable and worth stating plainly: every enforcement action is a free oracle query against your classifier. An oracle query is one probe at a system you cannot see inside, answered honestly — you submit an input and learn the output. Enforcement is exactly that: the adversary submits a post and the platform tells them, by acting or not acting, which side of the boundary it fell on.

auto-removals 0.96 M/day  +  post-review removals 0.78 M/day
    =  1.74 M labeled boundary probes returned to adversaries, free, per day

An adversary posts a variant, observes whether it survives, and has learned one bit about your decision boundary. Across a motivated population that is a black-box attack budget of over a million queries a day, refreshed continuously, at zero cost. No red team you can staff competes with that.

The consequence is observable on any frozen set, and its signature is that your number holds still while the thing it is supposed to measure does not:

adversarial benchmark frozen in January:
    recall measured in January                  0.78
    recall measured in July, same benchmark     0.78
    recall in July on a FRESHLY SAMPLED
    adversarial set from July traffic           0.51

The benchmark did not move because a benchmark cannot move. The distribution moved. Any content-integrity eval set has a half-life, and reporting a number from a frozen set is reporting the past.

The fixes are process, not modeling:

• Rolling eval sets. Resample adversarial evaluation from the last 30 days of traffic, monthly. Report recall on content < 30 days old as the headline and the frozen benchmark as a regression tripwire only.
• Recidivism tracking. Recidivism here means the re-posting of content you have already enforced against, whether by the original author or by anyone else. Hash every enforced asset with a perceptual hash — PDQ for images, TMK for video, MinHash shingles for text. A re-post of an enforced asset is caught at precision near 1.0 for 2 ms, and the recidivism rate is a direct measure of how hard the boundary is being probed.
• Deliberate latency on the boundary. Batching enforcement for borderline items in randomized 5-45 minute windows breaks the tight probe-observe loop. And here is the genuine tension, which is worth volunteering: Metrics prevalence and why accuracy is meaningless shows that fast enforcement is the dominant lever on harm, and this says fast enforcement maximizes the adversary’s information rate. The resolution is asymmetric — act instantly above t_remove where the model is confident and the probe teaches little, and randomize timing only in the band near the boundary where each observation is worth the most.

Assumptions in this section, and why none of them are load-bearing. The architecture rests on the modality shares, the FLOP model of 12 d^2 per layer plus a quadratic attention term, and 18 sampled frames per video. Every one of these can be wrong by a wide margin without changing a decision, and The cascade priced is why: the entire machine bill is a fraction of a percent of the human bill, so a 2x error in the cost model moves nothing anyone would act on. The one number in this section that is load-bearing sits in the last subsection rather than the architecture — the decay from 0.78 to 0.51 on freshly sampled traffic, because it alone sets the retraining cadence and it is the argument against the frozen-encoder alternative.

6. Metrics: prevalence, and why accuracy is meaningless

Which number goes on the dashboard? The obvious candidates are traps, and working out why leads to the chapter’s most useful result: cutting the time between publication and enforcement buys more than two quarters of modeling work.

Accuracy, disposed of

At a base rate of 0.0012, a model that outputs “benign” for every post is right on 1 - 0.0012 = 0.9988 of them — 99.88% accurate, while catching nothing. The full argument is in Accuracy under imbalance in arithmetic. It is mentioned here only so it can be dismissed in one line and never mentioned again.

The model metric: recall at a fixed precision, per policy

The number to report about the model is not the number to report about the product — the product gets its own, next.

report, per policy, per language, per modality:
    recall @ precision 0.95        (the auto-remove operating point)
    recall @ precision 0.50        (the review-band shape)
    PR-AUC over [0.02, 1.0] precision   (the region any action uses)

Never a single F1, and never one number across policies: hate speech at alpha 0.54 and spam at alpha 0.91 do not belong in the same average, because one of them has a ceiling four times closer than the other.

The product metric: prevalence, defined on views

One number is what the platform is actually accountable for, and its definition — views rather than posts — carries the whole argument.

Prevalence is violating views over total views, not violating posts over total posts — and the baseline every row of the next table moves against comes out of four numbers already on the table:

violating posts/day                    2.4 M
mean views per violating post            340
violating views/day                    816 M
total views/day                      180,000 M
baseline prevalence     816 / 180,000  =  0.45 %  =  45 per 10,000 views

Views, not posts, because harm is delivered by viewing. A violating post removed after 5 views did approximately nothing; one removed after 2 M views did all of the harm the policy exists to prevent. And the view distribution is heavy-tailed, so the top 0.1% of violating posts carry more views than the bottom 90% combined. A recall number computed on posts weights those identically.

Which makes time-to-action the dominant lever, and here is the arithmetic

Convert recall and enforcement latency into the same unit — surviving violating views — and a modeling roadmap and an operations roadmap can be compared directly.

Views are front-loaded, meaning most of a post’s lifetime audience arrives in its first few minutes. Measured cumulative view fraction by post age, where cumfrac(T) is the share of a post’s eventual views that have already been delivered by age T:

age        4 min   10 min   18 min    1 h     6 h     24 h
cum views   0.04    0.13     0.22    0.41    0.68    0.87

Residual prevalence after enforcement, with recall r and median action time T. Derive it in one line, because it is the engine of the chapter’s best argument and it arrives too often as a bare formula: a violation you catch at time T has already delivered cumfrac(T) of its lifetime views and delivers nothing after; a violation you miss delivers all of them. Weight the two cases by r and 1 - r and you have the fraction of baseline violating views that survive enforcement:

residual  =  r × cumfrac(T)  +  (1 - r) × 1.0

And note the assumption it hides: r and cumfrac are multiplied as if independent — as if the violations you catch have the same view trajectory as the ones you miss. Hold that thought; “the metric that hides harm” at the end of this section is a measured case where it is false, and the gap it opens is 2.5 per 10 k.

Now run that formula across four changes you could make. The last column is residual × 45, using the baseline of 45 violating views per 10,000 derived above. Work the first row by hand and the rest follow the same two steps:

baseline row:  r = 0.62,  T = 18 min,  cumfrac = 0.22

residual    =  0.62 × 0.22  +  (1 - 0.62) × 1.0
            =  0.1364       +  0.38
            =  0.5164

prevalence  =  0.5164 × 45  =  23.2 per 10,000 views

The two rows to compare are the second and the third: one is six months of modeling, the other is a queue-latency project.

Ten points of recall — two quarters of modeling — buys less prevalence reduction than cutting median action time from 18 minutes to 4. And a nightly batch pipeline at the same recall is 36.1 / 23.2 = 1.55x worse than the baseline, which is the argument for classifying at publish time rather than on a schedule, stated as a number instead of an instinct.

The corollary for the review queue: an item that sits 40 minutes in a queue has already delivered a third of its lifetime views. Queue latency is a prevalence metric, and it should be on the integrity dashboard next to recall, not on an ops dashboard somewhere else.

The metric that hides harm

Here is a quarter in which every model number improved and the thing the system exists to reduce did not — and the difference was swallowed by one identifiable averaging step.

                          quarter start    quarter end
aggregate recall              0.62            0.71     +9 pts, celebrated
posts removed/day             1.53 M          1.74 M   (§4's 0.96 M auto
                                                        + 0.78 M post-review)
prevalence, views          23.2/10k        22.6/10k     -0.6, barely moved

The prevalence row is deliberately off the r × cumfrac(T) + (1 - r) formula from three paragraphs above. Run the formula at the quarter-end recall and it disagrees with what was measured:

predicted  =  (0.71 × 0.22  +  0.29) × 45
           =  (0.1562 + 0.29) × 45
           =  0.4462 × 45      =  20.1 / 10 k

measured                       =  22.6 / 10 k
                                  ------------
gap                                2.5 / 10 k

That gap is the finding. The formula treats recall as one number, but this recall is post-weighted while prevalence is view-weighted.

The mechanism: the recall gain came almost entirely from spam and nudity — high-volume, low-severity, low-reach policies where the model was already good and more data helped. Those posts barely register in a view-weighted metric.

Aggregate recall is a post-weighted average and prevalence is a view-weighted one, so a system can get substantially better at the content nobody sees. Report recall per policy, weighted by that policy’s share of violating views, or the headline number will keep telling you about spam.

Assumptions in this section, and the load-bearing one. The load-bearing assumption is the one the derivation hides and the last subsection then breaks: that recall and the view trajectory are independent, so that the violations you catch have the same audience profile as the ones you miss. When they do not — when the extra recall lands on low-reach spam — the formula over-predicts the prevalence improvement, and the measured gap between the predicted 20.1 and the observed 22.6 per 10,000 is exactly the size of that error. Secondary: the mean of 340 views per violating post, and the cumulative-view table, both of which shift every row of the comparison table by the same factor and therefore change no ranking.

7. Serving architecture

Everything above now assembles into the path a single post travels from publication to a decision — priced in machines and in people, and ending with how a new model version is safely put in front of users. The headline result is a ratio: the humans in this system cost hundreds of times what the machines do, which decides where every optimization should go.

The diagram below is that path. Read it top to bottom — a post enters at the top and each successive stage is more expensive than the one above it, which is the entire logic of a cascade.

flowchart TD
    P(["Post published"]) --> H{"Hash match<br/>PDQ · TMK · MinHash<br/>2 ms"}
    H -->|"known violating<br/>31% of violations"| ACT["Enforce immediately<br/>precision ~1.0"]
    H -->|miss| C1{"Tier 1 · linear<br/>hashed char n-grams<br/>0.4 ms · CPU"}
    C1 -->|"clears · 92%"| DONE(["No action<br/>sampled 0.05% for audit"])
    C1 -->|"suspicious · 8%"| C2["Tier 2 · fused encoder<br/>text + image + frames<br/>cross-attention · 12-210 ms"]
    CTX["Context features<br/>parent post · is-quote<br/>author-target relation<br/>audience"] -->|"joined BEFORE scoring<br/>§8.2"| C2
    C2 --> HEADS["10 sigmoid policy heads<br/>+ calibration per policy<br/>per language"]
    HEADS --> TIER{"Tier map · §3"}
    TIER --> ACT
    TIER --> QUE["Review queue<br/>ranked by<br/>P × views × severity"]
    TIER --> DEM["Demote"]
    TIER --> DONE
    QUE --> REV(["Reviewer"])
    REV --> LAB[("Labels<br/>biased by the classifier")]
    ACT --> APP(["Appeals"])
    APP --> REV
    GRAPH[["Cluster detector<br/>co-post timing · shared assets<br/>creation cohort · runs offline"]] --> QUE

    style C1 fill:#bc6c25,color:#fff
    style C2 fill:#1d3557,color:#fff
    style TIER fill:#2d6a4f,color:#fff
    style CTX fill:#7f5539,color:#fff
    style QUE fill:#7209b7,color:#fff

Five nodes are coloured, and each colour marks something the node is the only instance of.

• C1, orange — the only stage that touches all 2 x 10^9 posts.
• C2, blue — the only learned scorer.
• TIER, green — the only place an action is chosen.
• CTX, brown — the only input that arrives from outside the post.
• QUE, purple — the only node that spends a human.

The last two are the pair to keep apart: the tier map is a comparison against four constants and costs nothing, while the queue is the $520 M constraint the whole chapter is written around. They are as far apart as any two boxes on the page.

Walking the diagram

A post published enters at the top and hits the perceptual-hash table first, because it is the cheapest test in the system. A known violating asset — 31% of all violations are re-posts of something already enforced — lets the system enforce immediately, at precision near 1.0, with no model involved.

A miss falls through to tier 1. Tier 1 clears 92% of everything, adjudicates a twentieth of a percent of those clears (the branch labelled No action sampled 0.05% for audit), and routes the remaining 8% to tier 2.

Tier 2 produces ten sigmoid policy heads, then calibration per policy and per language. Those ten calibrated scores enter the tier map of The tier map, which sends each post to enforcement, the review queue, demotion, or nothing.

Reviewer verdicts flow back into the store marked labels biased by the classifier — the whole subject of Where labels come from and what each source is good for, and the reason that store cannot be used to compute recall.

The cluster detector enters from the side, feeding the queue directly, because it acts on groups rather than items.

The CTX arrow: where context is joined is a design decision

Its label reads joined BEFORE scoring, and the emphasis is earned. Context features are joined before the encoder runs, not after the heads produce scores.

Context collapse’s whole diagnosis is that counter-speech and hate speech are indistinguishable in the 280 characters being scored: “no amount of model capacity fixes a missing input.” A post-hoc adjustment applied to a finished score is not an input.

Trace Context collapse’s example through the diagram. “Someone replied to my photo with [slur]” enters at P, misses the hash table at H, and is flagged suspicious by C1 on the slur token alone. At C2 it is concatenated with its is-quote flag and its parent post before cross-attention runs — which is what takes counter-speech FPR from 0.31 to 0.09.

Route the same flag into a node after HEADS and the encoder never sees it. The score it would have to correct was computed without the evidence that decides the case.

The sampled 0.05% for audit edge: the only measurement of the largest silent failure

C1 disposes of 1,840 M posts/day with no second opinion, and nothing else in the system ever looks at them. The label is the hard part’s label table has no row for tier-1 clears, and Review capacity is the binding constraint’s reviewers only ever see things the model scored highly.

So 0.05% of that stream is drawn uniformly and adjudicated. What that costs, in the currency this chapter cares about:

1,840 M × 0.05%  =  920 k items/day sampled from tier-1 clears
    at ~8 s AHT  =  2,044 reviewer-hours/day  /  6.5  =  315 reviewers
                 =  2.1 % of the 15,000 on shift

That 315 is inside Review capacity is the binding constraint’s 35% deduction for “appeals, audits, gold-set calibration, escalations,” not on top of it.

It is 77x the volume of the prevalence sample (920 k / 12 k) and 306x the gold set (920 k / 3 k), and it is cheap per item only because these are overwhelmingly obvious non-violations at 8 seconds each.

It buys the one number no other channel can produce: tier 1’s miss rate. Cut it and the cascade’s 92% clear rate becomes an article of faith.

The cascade, priced

Every stage of that path now gets a cost, in machines and then in people. The comparison between the two decides where engineering effort should go for the rest of the chapter.

Three units appear below:

• PFLOP — a quadrillion (10^15) floating-point operations. TFLOP/s is a trillion (10^12) of them per second.
• H100 — a datacenter GPU, rented here at $2.50 per GPU-hour.
• Utilization — the fraction of a chip’s theoretical arithmetic rate a real workload actually achieves. Taken as 60%.

Read the block as one multiplication per row — items per day times GFLOP per item — summed into a daily arithmetic total, then divided by what one GPU delivers in a day. The row to notice is video: 9% of the items and 70% of the bill.

2,000 M posts/day  =  23,150 /s average, 70,000 /s peak

hash match       2,000 M × 2 ms CPU   = 1,111 CPU-hours/day  ->  ~50 cores
tier 1 linear    2,000 M × 0.4 ms     =   222 CPU-hours/day  ->  ~10 cores
tier 2 fused        160 M items, split by §5's modality shares
                    (GFLOP, not MACs -- §5 doubles, and 35 GFLOP for
                     ViT-B/16 is what ml-sd/02 derives for the same net)
    text-only  74 M ×   6 GFLOP =    444 PFLOP     46%
    image      38 M ×  35 GFLOP =  1,330 PFLOP     24%
    image+text 29 M ×  59 GFLOP =  1,711 PFLOP     18%  <- 1.7x the image
                                                          encoder, per §5
    video      14 M × 690 GFLOP =  9,660 PFLOP      9%  <- dominates
    audio       5 M × 112 GFLOP =    560 PFLOP      3%
                                  ----------
                                  13,705 PFLOP/day     (items sum to 160 M)
at 300 TFLOP/s effective per H100, 60% utilization
    one H100 delivers  3.0e14 × 0.6 × 86,400  =  1.5552e19 FLOP/day
    13,705 PFLOP/day   =  1.3705e19 FLOP/day
    1.3705e19 / 1.5552e19  =  0.88 H100-days/day  ->  under 1 GPU

(a FLOP floor, not a bill -- real serving is batch- and bandwidth-bound, so
call it tens of GPUs. The point below survives either way: 27,000x at the
floor, still 470x at 50 GPUs.)

Reconcile the 160 M here against Review capacity is the binding constraint’s 2,000 M band table before moving on, because as stated they look like two different systems.

Only 160 M posts/day ever receive a tier-2 score. The other 1,840 M are disposed of by a linear model on hashed character n-grams that emits a pass/fail, not a posterior. Review capacity is the binding constraint’s table nonetheless assigns score bands to all 2,000 M.

The two are consistent, and the arithmetic shows exactly how — the 50.15 M below is the three scored bands above 0.38 (0.96 + 7.19 + 42.0):

tier-1 clears, never scored                       1,840 M
tier-2 scored below 0.38   =  160 - 50.15      =    110 M
                                                  -------
§4's "< 0.38" row                                 1,950 M

The tier-1 clears are folded into the bottom band, and that is safe for exactly one reason: the bottom band’s action is “nothing,” which is also tier 1’s verdict.

Nowhere else could you do this. Tier 1 is not co-calibrated with tier 2 — a hashed-n-gram linear model and a fused cross-attention encoder do not share a probability scale, and Calibration drift across languages and time insists calibration is a per-model-version artifact. The moment you wanted to demote something out of tier 1’s cleared set, the band table would be fiction, because you would need a calibrated tier-1 score you do not have.

One consequence for Review capacity is the binding constraint’s headline. There are three different denominators in play and all three appear in this chapter:

8.15 / 2,000  =  0.41 %   the quantile, as a share of ALL content
8.15 /   160  =  5.1  %   the quantile, as a share of SCORED content
7.19 / 2,000  =  0.36 %   what a human sees, of all content
7.19 /   160  =  4.5  %   what a human sees, of scored content

All four are right and they answer different questions. review_threshold is fed the 2,000 M histogram and returns the same cut either way, because the entire discrepancy lives in the bottom bucket, far below any capacity the org chart will ever buy.

GPU floor:  0.88 H100 × 24 h × $2.50/GPU-hour   =  $53/day    =  $19 k/year
real fleet: 50 H100 × 24 h × 365 × $2.50/GPU-hour           =  $1.1 M/year

The reviewer line is the one to derive rather than quote, because it is the number every prioritization argument below leans on:

First, what the 15,000 in Review capacity is the binding constraint actually is. It is the daily on-shift staffing level, not a headcount — Review capacity is the binding constraint puts 15,000 reviewers on the queue every calendar day, because the queue is 24/7 and 2 x 10^9 posts/day do not take weekends. Reading it as a headcount is the single most common error in this estimate, and it is a 1.46x error.

productive hours/year  = 15,000 × 365 × 6.5      =  35,587,500

blended rate           = $520 M / 35,587,500     =  $14.61 per productive hour
per PAID hour          = 14.61 × 6.5/8           =  $11.87

headcount behind it    = 15,000 × 365 / 250      =  21,900 people
                                                    ($520 M / 21,900
                                                     = $23,744 each)

Two assumptions to state out loud, and they are different in kind:

The $14.61 is not a sourced wage — it is what the stated $520 M implies. I am given a $520 M budget and 35.6 M productive hours, and the rate is the quotient. It is the number to challenge: it is fully loaded (wage, benefits, wellness programme, site and vendor overhead) and blended across markets that differ by more than 3x, so any single-market check will disagree with it. Sensitivity on the rate, ±20%:

$11.69/productive hour  ->  $416 M/year
$14.61/productive hour  ->  $520 M/year     (stated)
$17.53/productive hour  ->  $624 M/year

The 250 shifts/year is the assumption behind the headcount, not behind the bill. It is a 5-day week net of holiday, leave, sickness and training weeks. It converts the 15,000-person staffing level into the ~21,900 people you actually hire to sustain it, and it moves the per-head figure — but the $520 M does not move with it, because the bill is set by hours on the queue, not by how many bodies are rostered to cover them.

Side by side, every line with its rate and its unit count:

GPU floor,  0.88 H100 at $2.50/GPU-hour            ≈  $19 k/year
real fleet, 50 H100   at $2.50/GPU-hour            ≈  $1.1 M/year
15,000 on shift daily at $14.61/productive hour    ≈  $520 M/year

The humans cost 27,000x the GPU FLOP floor and 470x even a deliberately generous 50-GPU fleet:

$520 M / $19 k   =  27,000x     against the FLOP floor
$520 M / $1.1 M  =     470x     against a 50-GPU fleet

That ratio is the single most important number for prioritization in this system. Price two candidate projects against it:

cut queue volume 10% (a tier-1 precision win):
    0.10 × 35.59 M productive hours  =  3.56 M hours
    3.56 M × $14.61                  =  $52 M/year
    (10% of the queue is 10% of the bill -- that is the check that this closes)

cut the GPU bill 20%:
    0.20 × $1.1 M   =  $220 k/year   at the most generous fleet estimate
    0.20 × $19 k    =  $1,900/year   at the FLOP floor

Optimize the constraint, and the constraint is people.

Within the GPU side, video dominates, so the video sampling policy is the only lever worth touching. Uniform 1 frame per second over a 45-second clip would be 45 frames. Scene-change sampling, plus a mandatory dense pass over the last 8 seconds — the standard bait-and-switch placement, where a clip opens benignly and turns at the end — is 14 frames and catches more.

Against The trunk named’s 18-frame baseline that is a 21.9% cut in the per-video cost and 15.3% off the whole tier-2 bill. That is the largest single saving available on the GPU side, and it is still two orders of magnitude below one percent of the reviewer bill.

Assumptions in this section, and the load-bearing one. The load-bearing assumption is that the $520 M reviewer budget covers 15,000 people on shift every calendar day at 6.5 productive hours each, because the 470x ratio that drives every prioritization decision below is that budget divided by a GPU line. It survives abuse well: the whole conclusion holds if the GPU estimate is 100x too low or the reviewer budget is halved. What it does not survive is the category error of reading 15,000 as a headcount rather than a daily staffing level, which is a 1.46x error in the per-person figures and is the one an interviewer will probe. Secondary and unimportant: $2.50 per GPU-hour, 60% utilization, and the 50-GPU “real fleet” figure, which is deliberately generous precisely so the conclusion does not depend on it.

Rollout: shadow, canary, ramp, and a holdback that is legal on exactly one tier

A new model version has to reach production without repeating the regression Calibration drift across languages and time describes — and the release path ends on the one measurement question in this chapter that is settled on ethical grounds rather than statistical ones.

Calibration drift across languages and time’s regression — a better model shipped against a stale threshold, 14 points of system recall, invisible on the offline dashboard — is not something an offline gate can catch, because nothing offline knows where the threshold sits. It is caught by the first stage of a release, and a release here has four.

Stage 1 — Shadow: the new version scores 100% of traffic and the incumbent still acts. A shadow deployment runs the candidate model on live traffic and throws its output away. It is observed under exactly the production distribution while affecting nobody.

What shadow sees is the score distribution, which is precisely Calibration drift across languages and time’s failure mode. v12 would show 0.61 M items/day above the incumbent’s 0.94 threshold, against v11’s 0.96 M — a 36% drop on a million-item base, unmistakable within an hour, with zero users affected.

Thresholds are recomputed at this stage, on the gold set, per policy and per language, before anything is acted on. A canary run against un-recomputed thresholds measures the threshold bug rather than the model.

Run shadow for a full week even though the volume is decisive in minutes, because the content mix is weekly-periodic and one Tuesday is not a sample of it.

Stage 2 — Canary: 1% of traffic acted on. A canary is the first slice of real traffic the new version is allowed to affect, sized so that if it is wrong the damage is bounded and recoverable.

Shadow is blind to everything downstream of an action — no appeals, no reviewer verdicts, no user response — because none of those exist until the system does something, and Appeals are part of the design not a support function is entirely about signals of that kind.

Price the canary’s sensitivity off Appeals are part of the design not a support function’s own numbers rather than guessing. z in the last line is the number of standard deviations the observed appeal count sits from the expected one, using the square-root-of-the-count spread of a rare event:

1% of 0.96 M auto-removals/day          =  9,600 removals/day
at §9's 4.2% appeal rate                =    403 appeals/day

§9's regression case (removal precision 0.95 -> 0.88, appeals x2.4):
                                        =    968 appeals/day
z  =  (968 - 403) / sqrt(403)           =   28.1

28 standard deviations is not a marginal call.

So a regression that would add 0.96 M × (0.12 - 0.05) = 67,200 wrongful removals a day at full exposure is resolved in under a day at 1% exposure, against 9,600 removals of risk. That ratio is the argument for the canary existing at all.

Stage 3 — Ramp 1 -> 5 -> 25 -> 100, with Dialect and demographic bias in enforcement the serious one’s per-dialect FPR as an independent gate at every step, not only the last.

Dialect and demographic bias in enforcement the serious one’s whole finding is that a 4.7x disparity moves the aggregate metric by 0.4 points. So an aggregate gate clears at every rung of the ramp whatever is happening to the slice. A gate that cannot see the failure is not a gate.

Stage 4 — Holdback, and this is the one that needs an argument rather than a template. A permanent holdback is a population the system deliberately treats with the old policy so you can keep measuring the new one.

At the removal tier that is not a defensible thing to build. You would be choosing to leave real harm in front of real people in order to measure yourself, and the reason it is indefensible is exactly Why you cannot pick one threshold’s: a removal’s two sides are not the same kind of object, so there is nothing to trade.

At the demote tier it is fine, and Why you cannot pick one threshold already said why. Demotion is reversible, invisible, notifies nobody, and both sides of its ledger are views.

So the holdback is demote-only. 0.1% of posts skip the demote tier entirely, carrying 816 M × 0.001 = 816 k violating views/day — which resolves a prevalence difference many times over. The binding constraint on the size of this holdback is ethical, not statistical, so take the smallest one that resolves the effect in a week.

That holdback also closes a hole Appeals are part of the design not a support function names: the demotion tier generates zero appeals by construction and therefore has no feedback channel at all, so Appeals are part of the design not a support function asks for a deliberate audit sample of demoted content.

The two measurements are complements and you need both. The audit sample tells you how many demotions were wrong. The holdback tells you whether demoting achieved anything at all — which matters, given that Why you cannot pick one threshold has just shown the published band is net-negative.

8. Failure modes

Five ways this system breaks in production, ordered by how much harm they do rather than by how interesting they are. The first is the only one in this chapter where the model working exactly as designed is itself the failure.

8.1 Dialect and demographic bias in enforcement — the serious one

A single failure can be traced from the annotator’s desk to an account-level penalty in three measurable steps — and the standard dashboard cannot see any of it.

The bias runs through three stages, and each one is a separate measurement rather than a stage of one argument: it starts in the labels, is amplified by ordinary gradient descent, and is then compounded by enforcement.

Step 1 — annotation. Annotators who are not members of a speech community rate that community’s speech as more offensive. African-American English (AAE) is a rule-governed dialect of English with its own grammar and vocabulary; Standard American English (SAE) is the prestige variety, and the shorthand AAE and SAE is used from here on. On a controlled set where the same semantic content was written in both:

identical content, toxic-label rate:
    SAE phrasing        0.081
    AAE phrasing        0.178        2.2x

Step 2 — the model learns the token, not the speaker. A reclaimed slur used in-group is the single strongest lexical predictor of the “hate speech” label in the training set. It is high-frequency and consistent. The contextual signal that distinguishes in-group from directed use is low-frequency and inconsistent.

Gradient descent is the procedure that sets a model’s weights by repeatedly nudging every one of them a small step in whichever direction most reduces the loss. It is how every model in this chapter is trained. Each of those steps goes where the loss falls fastest, and the loss falls fastest on the signal that is both frequent and consistent.

So on a fixed step budget the model learns the token first, and may never get to the context at all. The model is not making an error; it is reproducing the labels it was given, efficiently.

Step 3 — enforcement compounds it. Fix the denominator before reading any number here, because three different quantities in this chapter are called “false-positive rate.” The first two below are the same three words over denominators that differ by roughly 800x (0.019 / 0.000024 = 792):

The block below uses the first row. Same three words, three denominators, and the slice comparison is only meaningful inside one of them.

1 - precision on hate-speech removals at t_remove = 0.94:
    SAE-classified posts        0.019
    AAE-classified posts        0.089          4.7x

downstream: removals carry strikes, strikes carry reach penalties and
account restrictions, so a 4.7x FPR becomes a 4.7x rate of account-level
penalty on one community

Now aggregate the two slices, because this is the step that shows why nobody caught it. AAE-classified posts are about 5.7% of hate-speech-classified traffic, so the aggregate is a weighted average with 94.3% of the weight on the clean slice:

aggregate  =  0.943 × 0.019  +  0.057 × 0.089
           =  0.017917       +  0.005073        =  0.0230

if AAE were fixed to the SAE rate:
           =  0.943 × 0.019  +  0.057 × 0.019   =  0.0190
                                                    ------
move in the aggregate metric                        0.0040  =  0.4 points

A 4.7x gap on one community is 0.4 points on the dashboard, because the affected slice is 5.7% of the denominator.

That is not a reporting oversight, it is arithmetic. Any minority slice, by construction, is a small weight in an aggregate. So the aggregate metric is structurally incapable of showing a disparity that lands on a minority — and the smaller the community, the more invisible the harm.

Slicing is not a nice-to-have here. It is the only instrument that can see the effect at all.

Four mitigations, in order of measured effect:

1. Recruit annotators from the speech community and give them speaker context. AAE false-positive rate 0.089 -> 0.041 with no model architecture change. This is the largest single lever and it is a staffing decision.
2. Report FPR per dialect slice as an independent launch blocker, at the same standing as aggregate recall. This requires a dialect classifier — which raises the obvious question, answered next.
3. Use the dialect signal for measurement only; never as a model feature. If dialect is an input, the model can and will learn different effective thresholds per dialect, and you have built explicit differential enforcement into the weights with an audit trail that is indefensible in every forum it will eventually be examined in. Measure by the slice, never condition on it.
4. Counterfactual consistency training. Generate dialect-paired rewrites and add a penalty on the score difference. Residual gap after: 0.041 -> 0.028. Real, but the two levers are not close: labels bought 0.089 - 0.041 = 0.048 and the training penalty bought 0.041 - 0.028 = 0.013, so the modeling fix is just over a quarter of the annotation fix — 3.7x smaller — and it is the one that requires a research project. That is the ordering lesson.

8.2 Context collapse

Context collapse is what happens when a post that only makes sense against something outside its own text is scored as if the text were all there is. It produces three recurring cases, and all three are a missing-input problem rather than a model-quality problem.

Three posts, each scored high and auto-removed. Read them and ask what a human would need in order to get the call right — the answer is never “a bigger model”:

post   "someone replied to my photo with [slur]. this is what i deal with daily."
score  hate_speech 0.91  ->  auto-removed

post   "[slur] is a word with a history most people do not know. In 1948 ..."
score  hate_speech 0.87  ->  auto-removed

post   quote-tweet of a public official's statement, quoting it verbatim
score  hate_speech 0.94  ->  auto-removed, and it was journalism

Mechanism: the evidence that distinguishes these from the violating case is outside the model’s input window. The quotation structure, the parent post, the author’s relationship to the target, the account’s history — none of it is in the 280 characters being scored. This is a missing evidence channel, and no amount of model capacity fixes a missing input.

Supply the channel and measure what happens to each probe set. Two of the three move a lot; the third barely moves, and the third is the interesting one:

add to the input: parent post, is-quote flag, author-target follow relation,
                  account age and prior enforcement, thread role

counter-speech FPR   0.31  ->  0.09
news/quotation FPR   0.28  ->  0.11
satire FPR           0.34  ->  0.24      <- barely moves

Satire barely moves, and it should not be surprising: the distinguishing evidence for satire is often not in the thread at all — it is in whether the reader knows the account. When the evidence does not exist in any available channel, the correct design response is to route to a human rather than to raise the threshold, which is one more reason the review tier exists.

8.3 Coordinated evasion

Sometimes every individual item is honestly borderline and the violation exists only in the relationship between them — the one failure in this chapter that requires changing what the system makes decisions about. pHash below is a perceptual hash of an image, and “pHash distance <= 6” means the two images differ in at most six bits of that fingerprint — visually the same picture.

200 accounts created in a 6-day window, each posting an image that is
pHash-identical modulo a 3-pixel border and a 1-degree rotation.

per-item score    0.41 - 0.47   below every threshold; 200 × nothing = nothing
cluster evidence  creation cohort · 94% co-post within 40 s · pHash distance
                  <= 6 · identical outbound domain     ->  posterior 0.998

You cannot threshold your way to this, because every individual item is genuinely ambiguous. The unit of decision has to change from the item to the cluster. A separate offline detector over co-posting timing, shared-asset hashes, account-creation cohorts, and follow-graph overlap produces cluster scores, and a cluster hit promotes every member into the review queue regardless of item score. It runs offline on a 15-minute cadence because graph features need a window to accumulate — which is a deliberate trade of latency for a signal that does not exist at publish time.

8.4 Calibration drift across languages and time

This failure catches out the most people in an interview: a strictly better model, shipped correctly, that destroys enforcement because a number in a configuration file did not move with it.

Thresholds are set on a calibrated score. A model retrained on a new data mix shifts its score distribution, and every threshold silently moves with it.

Read the block below top to bottom. The first line says the model got better; the second says the operating point moved; the third says what happens when you ship the first and ignore the second.

model v11 -> v12, same architecture, +14% training data
    PR-AUC 0.681 -> 0.694 (better);  score at precision 0.95: 0.94 -> 0.89
    t_remove left at 0.94:  removals/day 0.96 M -> 0.61 M,
                            SYSTEM recall 0.71 -> 0.57

Which recall is that, and where does the 14 points come from? The bare word “recall” is doing damage across four sections here, so pin it down.

This is Metrics prevalence and why accuracy is meaningless’s system recall: every enforcement path summed, post-weighted. The auto-remove tier is one component of it, and its size is Review capacity is the binding constraint’s top band:

auto-remove's contribution  =  912 k / 2,400 k  =  0.38

Cutting removals from 0.96 M to 0.61 M at a threshold that has not moved scales that contribution down in proportion. Nothing else in the system changes:

new contribution  =  0.38 × (0.61 / 0.96)  =  0.38 × 0.635  =  0.2415
lost              =  0.38 - 0.2415                          =  0.1385

system recall     =  0.7063 - 0.1385  =  0.568   ->  0.71 - 0.14 = 0.57

Both endpoints have to come from the same quarter, and this is the easy place to mix them.

0.38 is read off Review capacity is the binding constraint’s band table, whose 1.74 M removals/day and (912 + 890 × 0.88)/2,400 = 0.7063 recall are the state Metrics prevalence and why accuracy is meaningless labels quarter-end. So the starting point that 0.38 is a component of is 0.71, not Metrics prevalence and why accuracy is meaningless’s quarter-start 0.62. Subtracting a quarter-end component from a quarter-start total gives a pair of endpoints that describe no state the system was ever in.

The regression itself is unaffected either way, because 0.38 - 0.24 = 0.14 is a property of the tier and not of the baseline it sits in. It is the endpoints that were the artefact. Quote 0.71 -> 0.57 if you are working from Review capacity is the binding constraint’s table, or run the whole thing on the quarter-start numbers, but never one of each.

The auto-remove tier lost 36% of itself; the headline lost 14 points. A model-version change moved one tier and the number everyone reports moved by less than half as much — which is exactly why the regression survives an offline dashboard that reports curve-level metrics.

A better model shipped with a stale threshold is a 14-point recall regression that looks like nothing on the offline dashboard. Thresholds are per-model-version artifacts and must be recomputed on the gold set as part of the release, and re-derived per language because calibration differs by language far more than PR-AUC does. See Calibration what it means and when it matters.

8.5 The label feedback loop

The circle now closes back to Where labels come from and what each source is good for: the classifier chooses which items get labeled, and those labels train the next classifier, so the system can quietly stop learning anything it does not already believe.

The queue is filled by the classifier, labeled by reviewers, and those labels train the next classifier. Content the model scores low is never reviewed, never labeled, and never learned. Your training distribution converges to the model’s own current beliefs.

The break is the uniform-random prevalence sample: 12 k items/day drawn uniformly over views, adjudicated, and mixed into training with an importance weight — a multiplier on the row’s contribution to the loss that compensates for how rarely rows like it were sampled, so that a deliberately small sample can still speak for the population it was drawn from. It is 0.15% of the label volume and it is the only part of the label set that can discover a violation type the model has never scored highly. Sample uniformly, weight by the inverse of the sampling rate, and treat the cost as insurance rather than as training data.

It is a break and not a fix, and the difference is where the two samples land. Uniform-over-views spreads 12 k items across the entire score range, so it puts 252 of them a day in the band Why you cannot pick one threshold cannot resolve — enough for a quarterly number on the whole band and never enough for a per-bin one. The exploration band in The label is the hard part is the targeted version of the same instrument: 81.5 k items/day drawn only from below t_review, with the propensity logged because Review capacity is the binding constraint makes it move. One sample tells you the model has a blind spot; the other tells you where its edge is.

Summary of the failure modes

Every failure above, plus five smaller ones, with the mechanism that causes it, the instrument that would catch it, and the control that prevents it. p95 in the queue-starvation row is the 95th percentile: the age that 95% of queued items are younger than, which is the right statistic for a backlog because a mean hides the tail that actually ages out.

9. Appeals are part of the design, not a support function

What can an appeals stream measure? The short answer: it is a terrible estimator of how often you are wrong and an excellent alarm for when you suddenly got worse — and both halves have numbers.

An enforcement system without an appeals path is a system whose false-positive rate is unobservable. Appeals are the only channel through which a wrongly-removed user can reach you.

Start with what the channel actually produces in a day:

auto-removals/day                        0.96 M
appeal rate                              4.2%      ->  40,300 appeals/day
overturn rate on appeal                  11%       ->   4,430 overturns/day
overturns / removals                     0.46%

Now compare that against the truth, measured independently. The audited false-positive rate at t_remove is 5.0%:

real wrong removals  =  0.96 M × 0.050   =  48,000/day
surfaced by appeals                      =   4,430/day
                                            ---------
sensitivity          =  4,430 / 48,000   =  0.092  =  9%

Sensitivity here means the share of the real errors this channel manages to detect at all. Appeals surface 9% of them. The other 91% never appeal, because appealing requires noticing the removal, caring enough to contest it, and knowing how.

Appeals are a quality signal with terrible, but measurable, sensitivity. Use them as a regression canary, never as your false-positive estimate. The FP estimate comes from the audited gold sample; the appeals stream tells you fast when something broke, because a 3x jump in appeal rate on a policy shows up within hours of a bad release while the audit sample takes a week to reach significance.

Three further uses that are underrated:

• Overturn rate sliced by dialect, language, and country is one of the cleanest bias detectors available, because it is real people re-judging real enforcement on real accounts rather than a synthetic probe set. If AAE-classified removals are overturned at 19% against 9% overall, that is Dialect and demographic bias in enforcement the serious one confirmed on production traffic.
• The demotion tier generates zero appeals, because nobody is told. That invisibility is exactly why demotion has a low FP cost, and it is also why the demotion tier has no feedback channel at all. The only way to measure demotion FP is a deliberate audit sample of demoted content. Build it, or the cheapest tier is also the blindest one.
• Appeals cost capacity, and the coupling has a sign. Raising auto-remove recall raises appeals, which consumes the reviewer capacity that sets t_review:

+1 M auto-removals/day
    -> +42,000 appeals/day at 90 s AHT   =  1,050 reviewer-hours/day
    =  162 reviewers  =  1.1% of capacity  ->  t_review rises slightly

In the steady state that coupling is modest: one extra million removals a day costs about a percent of the review workforce, which nudges t_review up and nothing more.

The case that matters is the bad release, and it should be priced off the same exchange rate rather than guessed at, because a guess here is easy to get wrong by 3x. A release that drops removal precision from 0.95 to 0.88 takes the wrong-removal rate from 5% to 12% — and 12 / 5 = 2.4, so 2.4x the wrongly-removed users, and 2.4x the appeals they file:

baseline appeals            0.96 M × 4.2%    =  40,320/day
appeals scaled 2.4x                          =  96,768/day
increment                                    = +56,448/day
    × 90 s AHT  =  1,411 reviewer-hours  /  6.5  =  217 reviewers
                                             =  1.45 % of the 15,000 on shift

So the honest number is 1 to 1.5% of proactive capacity, and it is worth resisting the temptation to inflate it — reaching 3.5% would need appeals to rise 4.4x, which a precision move from 0.95 to 0.88 does not support. The point is sharper for being small: a bad release quietly takes a couple of hundred reviewers off the proactive queue and the extra appeals are concentrated in exactly the policy that regressed, so the capacity loss lands where you least want it. That is the argument for the precision floor being a hard gate rather than a target — it protects the capacity budget, not just the users.

10. Alternatives considered and rejected

Every design also has a negative space: for each plausible alternative, the reason someone would propose it and the specific number that rules it out.

Two abbreviations in the table. VLM is a vision-language model, one network trained on images and text together. LLM is a large language model.

The rows in bold are the ones an interviewer is most likely to propose. Read the middle column first — every alternative here is genuinely tempting, and an answer that cannot say why is not an answer.

11. Interviewer pushback

Here are the ten hardest follow-up questions this design attracts, each with the answer said out loud and a note on what the question is actually testing. Read them as a self-check: if any answer surprises you, the section it draws on is the one to re-read.

“Just train a classifier and pick a threshold. Why is this complicated?” Testing: whether you understand the two-sided asymmetry or just recite it. Because the threshold does not exist. Take the textbook rule with an 8-to-1 cost ratio at 0.12% prevalence: the optimum lands at 0.111, which is 0.92 recall at 0.15 precision, which is 14.7 M removals a day of which 12.5 M are legitimate posts. Nobody signs that, and the reason is not bad arithmetic — it is that a single exchange rate asserts I would trade eight wrongful removals for one prevented harm indefinitely, when the two costs land on different people and one is a harm I caused. So the real problem is constrained: maximize recall subject to a precision floor and a review-capacity budget. A precision floor pushes the threshold to the high end, which strands all the recall in the middle of the distribution, and that is what forces a third and fourth action rather than a threshold.

“Fine. Where do your four thresholds come from?” Testing: whether each number has a different origin. Three different places, which is the point — and only one of the four is derived, which I would say before you ask.

t_remove = 0.94 comes from a precision floor of 0.95, read off the PR curve. That 0.95 is a policy commitment, not an optimum: it says one wrongful removal-with-strike in twenty is the most we will defend. You can check that it is not derived. My action table prices a wrongful removal-with-strike at 1.0 relative to a violating view, and the break-even formula I am about to use would turn that into a floor of 1.0/2.0 = 0.50, not 0.95; the age-gate row prices at 0.08 and implies 7.4% against a stated floor of 0.55. Those gaps are not errors — they are the formula refusing to apply, because it needs both sides of the ledger in the same unit, and any action that reaches the author puts a strike, a notification and an appeal on the cost side.

t_demote comes from a harm ratio. If a violating view costs 1 and a demoted legitimate view costs w, demotion is net positive wherever in-band precision exceeds w/(1+w), and the demotion strength cancels out entirely. With w = 0.02 that break-even is 1.96%. And I will say the uncomfortable part, because it is my own table that says it: the 0.38-0.55 band is 410 k violations in 42.0 M posts, which is 0.98% — half the break-even. So 0.38 is not the crossing point, demoting that band is net-negative, and the threshold has to move up to wherever in-band precision reaches 1.96%, which my four-band table is too coarse to locate. The one thing I will not do is quote cumulative precision above 0.38, which is 4.4% and clears the bar by borrowing precision from posts I am already removing.

t_review = 0.55 comes from neither. It comes from the org chart: 15,000 reviewers at 6.5 productive hours and a 28-second weighted handle time is 12.5 M items a day, minus 35% for appeals and audits, so 8.15 M — which is 0.41% of 2 billion posts. The review threshold is the score that cuts the top 0.41% of the distribution. It is a quantile, not a precision target.

And I would separate two numbers people run together there. The cut is on 0.41%, but the top 0.96 M of that is auto-removed with no human in it, so what a reviewer actually sees is 7.19 M, or 0.36%. That difference is 11.8% of my only binding constraint — about $61 M a year of reviewer capacity bought and not spent — and the version an ops team would run drops t_review to about 0.546 so the review band alone is 8.15 M.

One caveat I would volunteer rather than be caught on: every number I just gave is computed on the ten policies pooled, and I opened this interview arguing the pool is meaningless. The shape of all three arguments transfers to any single policy’s curve; none of the four numbers does. In production there are ten curves and ten sets of thresholds, and spam’s 0.95-precision point is nowhere near hate speech’s — which may not exist at all, given a label ceiling of 0.82.

“What happens to that when you lose reviewers?” Testing: whether you know the constraint is live. The threshold moves and recall falls, with no model change. Reassign 1,100 reviewers to a new market and let AHT rise 9% while they ramp: capacity goes to 6.93 M, down 15%, t_review has to rise from 0.55 to 0.62 — that is 0.55 + 0.17 × 0.39, read by interpolating inside the band, because a routine that consumes whole score buckets can only ever return a bucket edge and would answer 0.94 — and 1.22 M items, 17% of the review band, fall out. They fall out of the low-score end, where precision is below the band average, so at most 151,000 violations a day go with them and system recall goes 0.751 to no worse than 0.688. A staffing decision moved a model threshold by 0.07 and cost up to 6.3 points of recall. Which is why t_review has to be computed from live queue depth on a schedule rather than configured — a static threshold against shrinking capacity grows a backlog until items age out unreviewed, and then you have paid the latency and gotten no decision.

“You have spent ten minutes on thresholds and org charts and still have not told me what the model is.” Testing: whether there is a system underneath the policy layer, or only a policy layer. Fair. Two tiers.

Tier 1 is a linear model over hashed character 3-to-5-grams, 0.4 ms on CPU, one binary route-or-clear output and no per-policy score. Character n-grams and not a word vocabulary, because homoglyphs and zero-width joiners are attacks on tokenization. It disposes of 92% of traffic at recall 0.926, and I can say 0.926 because 0.05% of its clears are adjudicated every day: 920,000 items, about 89 finds, ten percent relative error daily.

Tier 2 is one fused encoder on the 8% that survive — an 8-layer 384-wide text tower over 192 tokens, ViT-B/16 for images and sampled frames, an audio tower over log-mel frames, three cross-attention fusion layers at 768 wide, and ten sigmoid heads. 241 million parameters, half a gigabyte at fp16. It reproduces the cost column I quoted earlier: 6, 35 and 59 GFLOP — FLOPs, so twice the MAC count, which is how it agrees with the 34.9 GFLOP ml-sd/02 derives for the same ViT-B/16 — with image-plus-text at 1.68x the image tower alone.

One trunk rather than ten models, for a data reason and not a compute one. Ten encoders would be ten times a $1 M GPU line, which I can afford. The real reason is that label volume runs opposite to severity: spam gets 2.23 million adjudications a day and child safety rounds to zero of reviewer time, so the severity-5 heads survive only on a representation spam paid for. Fusion is early because the image-plus-text case is a conjunction that exists in neither channel.

What it eats is the post, the parent post, an is-quote flag, the author-target relation, account age, and prior adjudicated enforcement — 44 tabular floats, one key-value read on the author. The interesting row is the empty one: no item counters, no views, no CTR, because the item is milliseconds old. That one missing family is why queue priority uses projected reach, why the cluster detector runs offline on a 15-minute cadence, and why a recidivism hash is the only item history I have at t = 0.

Training is a masked multi-label BCE, masked because a reviewer who closes a spam item in eight seconds has not ruled on the other nine policies, and folding unobserved into negative mislabels a quarter of my positives. Positives are adjudicated decisions. Hard negatives are reviewer benigns, which are near-boundary by construction. Easy negatives come from that same tier-1 audit sample, which is the only negative I have drawn from the 92% of traffic I actually run on. And I split by time, and then by asset hash and author rather than by row, because 31% of violations are near-duplicates of something already enforced and a row split would hand me ten points of recall I did not earn.

“Your recall went from 62% to 71% this quarter. Good quarter?” Testing: whether you know what prevalence is. Prevalence barely moved — 23.2 per ten thousand views to 22.6. Aggregate recall is a post-weighted average and prevalence is a view-weighted one, so the gain came almost entirely from spam and nudity, which are high volume and low reach. The metric that matters is violating views over total views, because harm is delivered by viewing and the view distribution is heavy-tailed enough that the top 0.1% of violating posts outweigh the bottom 90%. And once you write prevalence as recall × cumfrac(T) + (1 - recall), something uncomfortable falls out: those nine points of recall bought less than cutting median action time from 18 minutes to 4 would have. Two quarters of modeling against a queue-latency project.

“So latency is the lever. Enforce instantly on everything?” Testing: whether you notice the tension you just created. Not uniformly, because instant enforcement is also how the adversary learns. Every enforcement is a labeled boundary probe returned free — 1.74 M a day across auto-removals and post-review removals, which is a black-box attack budget no red team matches. That is why a frozen benchmark reads 0.78 in January and 0.78 in July while fresh July traffic reads 0.51. So the policy is asymmetric: act instantly above t_remove, where the model is confident and the probe teaches little, and randomize enforcement timing in the narrow band near the boundary where each observation is worth the most. And report recall on content less than 30 days old as the headline, with the frozen set demoted to a regression tripwire.

“African-American English posts are being removed at four times the rate. What happened and what do you do?” Testing: whether fairness is a mechanism to you or a slogan. This is the question. Three steps, each measurable. Annotators who are not members of the community rate identical semantic content as toxic 2.2x more often when it is written in AAE — that is the label. Then the model learns the token rather than the speaker, because a reclaimed slur is high-frequency and perfectly consistent in the training labels while the context that distinguishes in-group from directed use is low-frequency and inconsistent, and gradient descent learns the consistent signal first. Then enforcement compounds it: removals carry strikes and strikes carry account restrictions, so a 4.7x false-positive gap becomes a 4.7x gap in account-level penalties. Aggregate FPR moves 0.4 points across all of that, so the aggregate cannot see it. The fixes in measured order: recruit annotators from the speech community and give them speaker context, which takes AAE FPR from 0.089 to 0.041 with no model change; make per-dialect FPR an independent launch blocker; and add counterfactual consistency training, which gets 0.041 to 0.028. What I would not do is put dialect in the model as a feature — that builds differential enforcement into the weights explicitly, and there is no forum in which that is defensible.

“A user posts a screenshot of a slur being used against them and gets banned. Why?” Testing: whether you diagnose a missing input or blame the model. Because the evidence that distinguishes counter-speech from hate speech is not in the model’s input. It is scoring 280 characters; the quotation structure, the parent post, the author’s relation to the target, and the account’s history are all outside the window. That is a missing evidence channel, and capacity does not fix a missing input. Adding parent post, an is-quote flag, the author-target relation, and account history takes counter-speech FPR from 0.31 to 0.09 and quotation from 0.28 to 0.11. Satire only moves from 0.34 to 0.24, and that is the honest limit — the disambiguating evidence for satire often is not in the thread at all, it is in whether the reader knows the account. Where the evidence does not exist in any channel, the right answer is to route to a human, not to raise the threshold.

“Two hundred accounts each posting something that scores 0.44. Nothing fires. Fix it.” Testing: whether you can change the unit of decision. No threshold catches this, because each item genuinely is ambiguous — the evidence is not in any item, it is in the relationship between them. So the unit of decision has to move from the item to the cluster. An offline detector over co-post timing, shared asset hashes with pHash distance, account-creation cohorts, and follow-graph overlap gives the cluster a posterior near 1.0 while every member sits at 0.44, and a cluster hit promotes all 200 into the review queue irrespective of item score. It runs offline on a 15-minute cadence because graph features need a window to accumulate, which is a deliberate trade: I give up publish-time latency to get a signal that does not exist at publish time.

“You shipped a model with better PR-AUC and enforcement collapsed. Debug it.” Testing: whether you know thresholds are versioned artifacts. Calibration. Same architecture, 14% more training data, PR-AUC 0.681 to 0.694 — genuinely better — but the score at which precision hits 0.95 moved from 0.94 to 0.89. The threshold config did not move, so I was operating 5 points too high: removals fell from 0.96 M to 0.61 M a day, and since the auto-remove tier is 0.38 of the 0.71 system recall my band table describes, scaling it by 0.61/0.96 takes system recall from 0.71 to 0.57. Fourteen points, and I would be careful to take both endpoints off the same quarter — my band table is the quarter-end state, so pairing its 0.38 against a quarter-start 0.62 describes a system that never existed. Nothing on the offline dashboard shows this, because the offline dashboard reports curve-level metrics and the regression is entirely in where I sit on the curve. Thresholds are per-model-version artifacts, recomputed on the gold set as a release step, and recomputed per language, because calibration varies across languages much more than PR-AUC does.

“How do you know your false-positive rate?” Testing: whether you trust appeals. Not from appeals. At t_remove the audited FP rate is 5%, which is 48,000 wrong removals a day, and appeals surface 4,430 of them — 9% sensitivity, because appealing requires noticing, caring, and knowing how. So the estimate comes from a stratified audited sample, and appeals are a regression canary: a 3x jump in appeal rate on one policy shows up within hours of a bad release, while the audit sample needs a week to reach significance. Appeals earn their place two other ways. Overturn rate sliced by dialect and country is the cleanest bias detector I have, because it is real people re-judging real enforcement rather than a synthetic probe. And the demotion tier generates zero appeals by construction — nobody is told — which means the cheapest tier is also the only one with no feedback channel, so it needs a deliberate audit sample of demoted content or it is unmeasured forever.

“Where do you spend the next headcount — engineers or reviewers?” Testing: whether your cost model produces a decision. The numbers say reviewers, and then say something more interesting.

The inference bill is startlingly small. 13,705 PFLOP/day against an H100-day of about 1.6e19 FLOP is under one GPU of pure arithmetic — and that is FLOPs, doubled off the MAC count, so it is the number ml-sd/02 would recognise for the same ViT. Even allowing one to two orders of magnitude for batching and bandwidth — call it 50 H100 at $2.50 a GPU-hour, $1.1 M a year — it is single-digit millions at most.

The reviewers I would price out loud rather than assert, and the first thing to say is what the 15,000 is: the number on shift every calendar day, not a headcount, because the queue runs 24/7. So it is 15,000 × 365 × 6.5 = 35.6 million productive hours a year, and against the stated $520 M that implies $14.61 per productive hour — $11.87 per paid hour, since only 6.5 hours of an 8-hour shift are on the queue. I would flag that $14.61 as derived rather than sourced: it is the budget divided by the hours, and it is the number to push back on. Behind those 15,000 daily seats are about 21,900 people at 250 shifts each, roughly $23,700 a head.

So humans are 470x a generous GPU fleet and 27,000x the FLOP floor. A change that cuts queue volume 10% at constant recall frees 3.56 million reviewer-hours and is worth $52 M — 10% of the queue is 10% of the bill. A 20% GPU saving is worth $220 k.

But the highest-leverage spend is neither. Rewriting the hate-speech policy with 40 worked examples and an explicit satire carve-out took Krippendorff’s alpha from 0.54 to 0.68, cut handle time from 45 to 38 seconds, and moved model F1 from 0.61 to 0.70 with no model change at all. If two trained reviewers agree only 70% of the time, single-reviewer labels are right about 82% of the time and a perfect model measures 82%. A lot of what looks like modeling headroom on hate speech is label noise, and the cheapest way to buy it back is a document.

Next: 06 — Video Recommendation.