How to design harmful-content detection

In this lesson, we’ll design a system that catches harmful content at the scale of a large platform, from the post arriving to the enforcement action landing. Content moderation looks like a classification problem (flag the harmful posts, let the rest through) but it is really a labeling problem wrapped around an action problem. That gap is what makes detecting harmful content hard, and it drives every decision below. By the end you’ll be able to place each threshold, name the constraint that actually binds, and defend the whole design in an interview.

The worked example is a platform carrying two billion posts a day. The numbers exist to show where each decision comes from, not to be memorized.

Two ideas that frame everything

The policy is the label definition. “Is this hate speech?” has no answer until someone writes down what hate speech is, in a document, with worked examples. Even then, two trained reviewers disagree on the hard cases at a measurable rate, and that rate is a ceiling on every model metric the system can report. Most disagreement in this domain (between teams, between reviewers) is disagreement about policy expressed as a modeling problem.

The cost asymmetry cuts both ways. In fraud, a missed case costs money and a false positive costs a phone call; one error is clearly cheaper. Here, a missed violation is real harm to a real person, and a false positive is a legitimate user silenced by an automated system with no explanation. Neither error is cheap, and that is what forces the architecture below.

Two definitions borrowed from the metrics material

The precision-recall (PR) curve. Sweep a score threshold from high to low. At each setting, precision is the share of items acted on that were genuinely violating, and recall is the share of all violating items that were caught. Plotting one against the other gives a curve; the area under it is the PR-AUC, a one-number summary.

Prefer the PR curve to the ROC curve when violations are rare, because it never counts the vast pool of correctly-ignored benign posts. The catch: precision depends on how rare violations are, so a PR-AUC is only comparable against another measured at the same prevalence. To compare across surfaces or time periods with different prevalence, use ROC-AUC instead. The metrics chapter derives both.

The noise ceiling. If the training and evaluation labels are themselves wrong some of the time, a perfect model still disagrees with the test set exactly as often as the test set is wrong. The measured score is capped by label quality, not by the model. The model-debugging chapter turns that into an exact number.

The models in this design

One post goes in; one enforcement action comes out: remove, remove with a strike, queue for a human, demote, or nothing. There is no single “harmfulness model.” Seven components sit in between, and only one is a large neural network.

A few terms used in the table:

• A perceptual hash is a short fingerprint computed so that a re-encoded, cropped, or slightly rotated copy of an asset hashes to nearly the same value (PDQ for images, TMK for video, MinHash shingles for text).
• A cascade runs a cheap model on everything and passes only what it flags to an expensive one.
• Calibration turns raw model output into a probability you can compare against a threshold.
• Online means on the live publish path; offline means on a schedule over logs.

Two absences matter later. No model chooses an action: actions come from comparing scores against four fixed thresholds. No model uses the post’s own engagement history: at scoring time the post is milliseconds old and has none.

Two numbers drive every conclusion: the base rate of 0.12% (the share of posts that violate some policy) and the human review capacity of ~8.15 M items/day (0.41% of the platform). Everything else (the GPU cost model, the exact parameter count, the video frame count) can be off by a factor and change only digits.

Framing: the objective is multi-label, not multi-class

• Input: a post (text, image, video, audio, or any mix) plus author, thread context, and audience.
• Output: a per-policy score, and an enforcement action.
• Volume: 2 x 10⁹ posts/day, 180 x 10⁹ views/day.
• Prevalence: ~0.12% of posts violate some policy, 2.4 M/day. Prevalence is this domain’s word for the base rate.
• Latency: classification runs at publish time, and enforcement has to land within a few minutes. The binding number is queue latency, not model latency.

The output layer is ten scores that can all be high at once, not one score split across ten categories.

• A multi-class output picks exactly one label using a softmax, which forces the classes to compete for probability mass.
• A multi-label output asks a separate yes/no question per label using a sigmoid on each, so all ten answers can be “yes” at once.

A post can be a violent threat and hate speech and harassment simultaneously, so softmax is statistically wrong here. It is also operationally wrong: each policy carries a different enforcement consequence, reviewer skill set, appeal path, and legal reporting obligation. A single “harmful” score cannot be routed to any of them.

The ten policies, grouped by the severity their owners assign (severity decides the consequence of a violation, not how likely one is):

independent sigmoid heads on a shared trunk, with a severity per policy:
    child safety 5 · violent extremism 5   auto-escalate, legal reporting
    credible violent threat 4
    adult nudity 3 · hate speech 3 · harassment 3
    graphic violence 2 · regulated goods 2
    spam 1 · health misinformation 1        demote only, no removal

Severity is not the score. The score is P(violates policy k), the model’s confidence. Severity is a policy-owner’s judgement about how much harm one violating view does. They come from different places and they multiply in the action decision. A 0.99 spam score should not out-rank a 0.60 child-safety score in a review queue, even though 0.99 > 0.60.

One trunk, ten heads. A trunk is the shared body of the network that turns raw input into one summary vector; a head is a small layer on top answering one question. Ten heads on one trunk means the expensive work of reading the post happens once and is then asked ten separate questions. That shape is a policy decision (ten enforcement paths) fixed before any model exists.

The label is the hard part

Where does a harmful-content label physically come from, and what does each origin do to the metrics you can report?

Inter-annotator agreement is the ceiling

Krippendorff’s alpha measures how much two or more people agree on the same items, corrected for chance agreement. It runs from 1.0 (always agree) through 0 (agree no more than guessing).

The implied-accuracy column comes from two steps. First, undo the chance correction to get the raw rate at which two reviewers agree (for hate speech, alpha 0.54 becomes agreement ~0.71). Then model two reviewers who each get the answer right with probability q and err independently: they agree when both are right or both wrong, so agreement = q² + (1-q)². Solving for hate speech gives q ≈ 0.82.

That back-solve is trustworthy where alpha is high and shaky where it is low: at low alpha almost all observed agreement is the chance correction, so small errors amplify. Treat 0.97 as a number and 0.73 as an order of magnitude. The independence assumption is also optimistic: reviewers share a guideline document and workload, so their errors correlate, which inflates agreement. So 0.82 for hate speech is a ceiling on the ceiling.

What the ceiling does to your metrics: when both training and evaluation labels come from single reviewers at 82% accuracy, a perfect model agrees with the evaluation set only 82% of the time. A measured F1 of 0.80 on hate speech may already be at the ceiling, and every parameter spent past it fits reviewer noise.

The consequence is that the largest single-quarter quality win usually comes from editing a document, not training a model. Rewriting a policy raises the agreement rate, which raises the ceiling, which raises every measured number underneath it:

hate speech policy v3 -> v4:
    added 40 worked examples, split "slur, directed" from "slur, reclaimed",
    added an explicit satire carve-out

    Krippendorff alpha    0.54  ->  0.68
    reviewer handle time  45 s  ->  38 s
    model F1, SAME MODEL, relabeled eval set    0.61  ->  0.70

The model did not change. The measurement got less noisy and the training targets more consistent. If you cannot articulate a policy precisely enough for two trained humans to agree, you cannot train or evaluate a model to do it.

Where labels come from, and what each is good for

A label is the output of a process, and five processes produce labels here, each with a different bias pointing in a different direction. Two terms: adjudicated means judged by more than one reviewer with disagreements resolved by a senior specialist (near the noise ceiling); stratified means sampled deliberately unevenly, over-sampling hard and rare cases (good for per-policy quality, useless for measuring how much violating content exists).

The Bias column matters more than Volume: the two biggest sources are the two least usable.

User reports fire when a viewer dislikes a post, which overlaps only partly with “violates a policy”, hence ~0.06 precision, nineteen in twenty not violations. The bias is not random: reports scale with reach and with how unpopular the author is, and organized brigading (a group mass-reporting one account) can manufacture the signal. Treating a reported-then-removed post as a recall success imports the reporting population’s preferences into the definition of harm. A report count is also a tempting feature, and a model that learns it learns to enforce popularity. The control is structural: a report routes an item into the queue and never becomes a training positive.

Human review carries two stacked biases. Selection: the queue is filled by the classifier, so recall computed on queue labels would read 1.00 for a model that finds nothing, precision wearing recall’s name. Annotation: a single reviewer’s error is not symmetric across communities (labeling identical content as toxic 2.2x more often in one dialect than another). A model trained on those labels reproduces the direction, and a model evaluated on them is scored by the same skewed ruler: the error cancels in measurement and compounds in production.

Policy changes are not a volume; a revision changes what the population is. Any quality time series spanning a revision compares answers to two different questions, so a model that “improved 9 points” across that boundary may have improved by zero. Two controls, both needed: version the policy like you version the model and stamp every label with its policy version; and on the revision date, re-adjudicate a frozen slice of the gold set under the new policy and publish both numbers for the same model.

No single reweighting fixes all three: reports over-sample what is visible and disliked, review over-samples what the model already believes, and a policy change moves the target underneath both. Only the gold set (stratified: “how good is the model, per policy”) and the prevalence sample (uniform over views: “how much harm is being delivered”) are designed to be unbiased, and they are small because adjudication is expensive. The uniform-random prevalence sample is the most valuable 12,000 items per day in the system, because it is the only label source not sampled by the model being evaluated.

From those sources to a training set

Six decisions, each forced by the label sources, not by modeling preference.

Positives are adjudicated decisions only (890 k/day, the violations found inside the reviewed queue). User reports are excluded at 0.06 precision. Recidivism hash matches are excluded even at ~1.0 precision, because they are near-duplicates of positives already in the set: adding them trains an expensive encoder to recognize what a 2 ms lookup already handles.

Negatives come from three sources, drawn from different places in the score distribution.

hard negatives   reviewer "benign" verdicts in queue   6.30 M/day
                 near-boundary BY CONSTRUCTION (scored above t_review)
easy negatives   the tier-1 clear audit sample           920 k/day
                 uniform over the traffic tier 1 disposes of
unbiased anchor  the prevalence sample                     12 k/day
                 uniform over VIEWS, importance-weighted

Train on the queue’s benigns alone and every negative the model has ever seen scored above t_review: the model is fitted on 0.41% of the input space and deployed on all of it. The tier-1 audit sample is the only source of negatives drawn from the 92% of traffic the model actually runs on, and it is already paid for as a measurement.

The loss is masked multi-label binary cross-entropy (BCE). BCE charges the negative log of the probability assigned to the true answer, so being confidently wrong is expensive. The mask matters because a reviewer who disposes of an item as spam in 8 seconds has not ruled on the other nine policies: those labels are unobserved, not negative. On the gold set, violating items violate 1.34 policies on average, so treating unobserved as negative would mislabel about 25% of true positive (item, policy) pairs, concentrated on exactly the co-occurrences the multi-label layout exists to represent.

A prior shift also has to be undone. Every threshold assumes a probability at the production base rate, but the training mixture is nowhere near it: ~11% positive against production’s 0.12%, roughly 100x. Because a base-rate correction is a single addition in log-odds, the whole fix is one constant: subtract ~4.63 from the logit. Concretely, a raw sigmoid of 0.50 becomes a production posterior under 1%. Calibration means exactly this: when the model says 0.30, about thirty in a hundred really are violations. It is separate from ranking (a model can order items perfectly and still be systematically 100x too high) and every threshold in this chapter silently depends on it.

Split by time, then by asset and author. Train up to T, validate on the next 7 days, evaluate on a gold set drawn after T, because the adversarial distribution moves and a random split scores the model on its own training era. Then the duplicate trap: 31% of violations are near-duplicates of an already-enforced asset, so a row-level split puts perceptual-hash neighbors on both sides and the model “recalls” them by memorization. Splitting by asset-hash cluster and author instead strips the memorized 31% out, taking holdout recall from an inflated 0.78 to an honest 0.68. Ten points of recall that were never earned.

Retrain weekly. Fresh-traffic recall falls from 0.78 to 0.51 over six months, about 6.8% of recall lost per month, 1.6% per week. Weekly retraining on a rolling 90-day window with time-decayed weights holds the loss inside 2%. Every retrain moves calibration, so every retrain must recompute all four thresholds, per policy and per language.

Log the propensity

Record, next to every labeled row, the probability that the row was allowed to be labeled at all. That probability is the propensity. When a label only arrives if the system acted, the labeled set is a sample of your own past decisions, and the propensity is the only record of how that sampling was done.

This system needs the discipline more than most, because the label-generating policy is t_review, a quantile recomputed daily from live reviewer capacity. A decision from the week when 1,100 reviewers were on loan came from t_review = 0.62, not 0.55; mixed into training with no propensity attached, it silently reweights the region the model learns hardest.

A deterministic cut gives every row a propensity of exactly 0 or 1, and no reweighting recovers a region that was never sampled. So the queue policy is made slightly stochastic below the cut: an exploration budget of 1% of capacity (81,500 items/day) drawn from the band just below t_review. That costs at most ~0.42 percentage points of recall, and it buys a per-bin measurement of the demote band that no other source can produce: the prevalence sample alone takes a full quarter to put one honest number on the whole band and never reaches a per-bin number at all. With a logged propensity, an inverse-propensity estimate (multiply each find by 1/p) reconstructs the band’s population total as a free check.

Why you cannot pick one threshold

A scope note: every number in this section and the next pools the ten policies, and the pool is meaningless as a real operating point. The shape of each argument transfers to any single policy’s curve; none of the resulting numbers does. Production carries ten score distributions, ten PR curves, and ten sets of four thresholds. One worked instance shows the method.

The single-threshold argument, run to its conclusion

The textbook rule says: act when P(violation | x) > C_FP / (C_FP + C_FN). Suppose policy leadership says a missed violation is 8x as bad as a wrongful removal, so C_FN = 8 C_FP, giving a threshold of 1/9 ≈ 0.111. Read the operating point off the pooled PR curve there:

at t = 0.111:   recall 0.92,  precision 0.15
violations caught  =  2.4 M x 0.92        =  2.2 M
removals/day       =  2.2 M / 0.15        =  14.7 M
of which wrong     =  14.7 M - 2.2 M      =  12.5 M legitimate posts/day

No organization runs that, and the reason is not bad arithmetic: it is that a single exchange rate asserts a willingness to trade eight wrongful removals for one prevented harm, indefinitely. The two costs are borne by different parties: one is a harm caused, the other a harm failed to be prevented. They are not fungible the way an expected-cost minimization requires.

One tier escapes this. A removal reaches the author as a notification, a strike, and an account-level consequence that outlives the post. Demotion has none of that: it is reversible, invisible, notifies nobody, and its entire effect is views delivered or not. When both sides of the ledger are views, an exchange rate is a claim a survey can actually put to people, and expected-value arithmetic is legitimate. That is why t_demote gets a break-even and t_remove gets a floor.

So the real formulation is constrained, not scalarized:

maximize   recall
subject to precision >= P_min           (a floor, not a price)
           reviewed_items <= C_review    (a capacity, not a price)

A precision floor forces the threshold to the high end of the curve, stranding all the recall in the middle of the distribution. The middle is not worthless (just not automatically actionable) and that is what creates the third action.

Actions have costs that differ by 50x

The single act/do-not-act decision becomes a menu, each with a priced false positive. An interstitial is a full-screen click-through warning; an age-gate is the same conditioned on account age. Both are viewer-side and instant, and neither touches the author, which is why that row costs 0.08 and the removal row costs 1.0.

The c/(1+c) column is the expected-value break-even, and it matches the stated floor on only one row, demote. Age-gate is 7.4x off, remove 2.5x off. The chapter derives one floor and chooses three. The reason is the same one from above: expected-value arithmetic needs both sides in the same unit, and only the demote row satisfies that. The moment an action reaches the author, the cost side acquires terms (a strike, an appeal, an account penalty, the standing risk of a public incident) that are not views. c = 0.6 is a summary of a bundle, not a price, and you cannot invert a summary. So the upper floors come from policy leadership and legal exposure: 0.95 says “one wrongful removal-with-strike in twenty is the most we will tolerate.”

Three origins produce four numbers: the invisible reversible tier is derived, the visible irreversible tiers are chosen, and t_review is measured off an org chart.

t_remove  = 0.94     set by a precision floor of 0.95
t_review  = 0.55     set by REVIEW CAPACITY, not the PR curve
t_demote  = ?        set by a harm-ratio break-even; the published 0.38 fails it
below                no action

The demote threshold falls out of a measurable ratio

Let one violating view cost 1 and one suppressed legitimate view cost w. Demotion cuts reach by a factor d, which appears identically on both sides and cancels: how hard the demotion is does not change where the threshold goes. What remains is a comparison of counts that converts to precision:

demotion pays  <=>  in-band precision  >  w / (1 + w)  ≈ w   (for small w)

With w = 0.02 from a survey-calibrated harm scale, the break-even is 1.96%. Check it against the measured band:

demote band 0.38 - 0.55:   42.0 M posts,  410 k violations
in-band precision  =  410k / 42.0M  =  0.98%
break-even         =  1.96%

Demoting the 0.38-0.55 band destroys value, by the same arithmetic that justifies demotion at all. The published t_demote = 0.38 is half the precision it needs. Three exits: move the threshold up (in-band precision crosses 1.96% somewhere inside the lower band; you need finer than four bands to locate it); lower w (which requires a suppressed view to cost under 1% of a violating view, a claim about harm the survey must support); or drop the tier.

The tempting fourth exit is to quote the cumulative precision above 0.38 (4.4%) and declare the bar cleared. That is the wrong quantity. The break-even is marginal: posts above 0.55 are already being removed or reviewed, so their precision cannot pay for an action taken on the band beneath them. Cumulative precision makes a marginal tier look ~4.5x better than it is. Sometimes the honest answer to “where did your threshold come from” is that the tier should not ship yet.

The tier map

The four thresholds assemble into one decision path that every score travels.

flowchart TD
    S(["Per-policy score<br/>P(violates k)"]) --> T1{"score >= 0.94<br/>precision >= 0.95"}
    T1 -->|yes| REM["AUTO-REMOVE<br/>+ strike if severity >= 4<br/>0.96 M/day"]
    T1 -->|no| T2{"score >= t_review<br/>SET BY CAPACITY"}
    T2 -->|yes| Q["HUMAN REVIEW QUEUE<br/>prioritized by expected harm<br/>7.2 M/day"]
    T2 -->|no| T3{"score >= t_demote<br/>IN-BAND precision >= 1.96%<br/>not 0.38"}
    T3 -->|yes| DEM["DEMOTE · reach -70%<br/>no notification"]
    T3 -->|no| NA(["No action<br/>>= 97.5% of posts"])

    REM --> AP["Appeals path<br/>4.2% appeal · 11% overturn"]
    Q --> RD{"Reviewer decision"}
    RD -->|violating| REM
    RD -->|benign| NA
    RD -->|unclear| ESC["Escalate to<br/>policy specialist"]
    AP --> RD

    style T2 fill:#bc6c25,color:#fff
    style Q fill:#1d3557,color:#fff
    style DEM fill:#2d6a4f,color:#fff
    style REM fill:#9d0208,color:#fff

A score falls through four gates in order: at or above 0.94, auto-remove (a strike attaches only at severity 4+, which is why severity appears here and nowhere in the threshold arithmetic); below that but at or above t_review, go to a human; below that but at or above t_demote, demote silently; below that, nothing.

On the human side a reviewer returns three verdicts, and the third is the one people forget. Violating sends the item down the removal path; benign clears it; unclear escalates to a policy specialist who owns the policy document. That specialist’s ruling settles the item and becomes a worked example in the next policy revision, the mechanism by which the agreement rate improves at all.

Two things to read off the diagram. Only the 7.2 M reaches a reviewer; the 0.96 M auto-remove band leaves without a human, so ~11.8% of the review capacity is bought and not spent. And the appeals arrow feeds back into the review queue, so appeals consume the same finite capacity as proactive review.

The load-bearing assumption here is the harm ratio w = 0.02, since t_demote is the only derived threshold and moves with w almost one-for-one. It is a stated preference from a survey, not a measurement.

Review capacity is the binding constraint

The third threshold’s origin is not a machine-learning question at all. t_review is not read off a PR curve. It is read off an org chart.

reviewers on shift daily                        15,000
productive hours per shift                       6.5   (wellness breaks,
                                                        calibration, training)
weighted average handle time                    ~28 s  (spam 8 s ... extremism 90 s,
                                                        weighted by queue mix)

raw capacity  =  15,000 x 6.5 x 3600 / 28   ≈  12.5 M items/day
minus 35% for appeals, audits, calibration, escalations
available for the proactive queue           ≈   8.15 M items/day
                                            =   0.41% of all content

t_review is whatever score cuts the top 0.41% of the distribution. A quantile is a cut defined purely by the share of the distribution above it, not by any property of the score. That distinction is the whole section: a precision target says how good the model is; a quantile says how many people you employ.

What the quantile buys

Score-band decomposition, all ten policies pooled, per day:

The auto-remove and review bands sum to exactly 8.15 M: the capacity picked the band edge. Two recall numbers fall out: the ceiling (everything the system even looks at) is (912 + 890)/2,400 = 0.751, and net proactive recall, after reviewers catch ~88% of the review band, is ~0.71.

The quantile and the queue are different populations. The cut is on 0.41%, but the top band is auto-removed with no human, so a reviewer sees 0.36%. The gap is ~11.8% of the only binding constraint, about $61 M/year of reviewer capacity bought and not spent. There are two honest resolutions: state working capacity as 7.19 M and treat 8.15 M as the quantile’s population (the version used below), or lower t_review to ~0.546 so the review band alone is 8.15 M (the version an operations team would run).

The constraint is live, and it moves

A staffing shock, worked through: 1,100 reviewers reassigned to a new market and handle time up 9% while the rest ramp. Capacity drops to ~6.93 M/day (-15%). After the auto-remove band takes its 0.96 M, ~1.22 M items (17% of the review band) fall out, off the low-score end, so at most 1.22 M x 0.124 ≈ 151 k violations/day, about 6.3 points of recall. t_review has to rise from 0.55 to ~0.62, and getting that second digit requires interpolating inside a band: a routine that consumes whole score buckets can only ever return a bucket edge, which would answer 0.94, queue 0.96 M items, and idle the rest.

A staffing decision moved a model threshold by 0.07 and cost up to 6.3 points of recall, with no model change. The corollary: t_review must be computed from live queue depth on a schedule, not configured. A static threshold against a shrinking queue silently grows a backlog until items age out unreviewed: you pay the latency and get no decision.

Prioritize the queue by expected harm, not by score

Which items enter the queue is one decision; the order they are worked is another, worth 2.3x on the metric that matters.

item A   score 0.90   projected remaining views     12   severity 2
item B   score 0.58   projected remaining views 400,000   severity 3

expected harm  =  P(violation) x projected_views x severity
    A  =  0.90 x     12 x 2  =         22
    B  =  0.58 x 400,000 x 3  =    696,000

Score order puts A first, and A is worth 22 units against B’s 696,000. Reviewer capacity is a fixed budget of attention, so spend it where the integral of harm is largest, and that integral is dominated by projected reach. Score spans ~1.7x inside the review band; reach spans four orders of magnitude. The projected-views model does not need to be good; it only has to separate 12 from 400,000, which any model does.

The two load-bearing numbers here are the ~28-second weighted handle time and the 35% overhead deduction, because capacity is linear in both and the threshold is a quantile of capacity.

The model itself, and the arms race

The modality mix decides the architecture, the architecture decides the serving cost, and the arms race decides which parts may be frozen.

Modalities

A modality is a channel of input; multimodal means one model reading several together. Four evasion techniques all attack how text becomes tokens:

• Leetspeak, lookalike digits or symbols: h4te.
• Homoglyphs, characters from other alphabets that render identically: a Cyrillic а for a Latin a.
• Zero-width joiners: invisible characters that split a word into fragments for a tokenizer.
• Algospeak, community-invented substitutes adopted because the system does not know them yet: “unalive” for “kill.”

Cost/item spans 500x, which is why the design is a cascade. The image-plus-text row forces a fused model. A picture of a person and the caption “this one” are each innocuous; together with the preceding post they are a targeted threat. Late fusion scores image and text separately and combines the two numbers at the end: it cannot represent a conjunction that exists in neither channel, because by the time the numbers meet, everything that made them a threat together is gone. Cross-attention fusion lets each image patch attend to each text token while both are still full representations, so “this one” can be bound to the face it points at. It costs ~1.7x the image encoder alone, affordable only because it runs on the 8% of posts that survive the cascade.

The trunk

tier 1   linear model over hashed character 3-5-grams, 2^20 buckets,
         ONE binary "route to tier 2" output               0.4 ms CPU

tier 2   text tower     8 layers x 384 wide, 192 tokens      6 GFLOP
         image tower    ViT-B/16, 224x224, 197 tokens       35
         audio tower    8 layers x 512 wide, 1,500 frames  112
         fusion         3 cross-attention layers x 768 wide  18
         heads          10 x sigmoid                          ~0
         image + text   6 + 35 + 18  =  59  =  1.68x the image tower
         video          ~690 (dominated by frame count)

         241 M parameters (96 M of it a 250k-token embedding table).
         0.48 GB at fp16.

A ViT-B/16 is a Vision Transformer that cuts a 224x224 image into 16x16 patches and treats each as a token (196 patches + 1 summary = 197). A GFLOP is a billion floating-point operations. One subtlety worth stating because the whole cost table depends on it: a MAC (multiply-accumulate) is two floating-point operations, so a MAC count must be doubled to be a FLOP count. The image tower is 17.4 G MACs = 34.9 GFLOP, the same number the visual-search chapter derives for the identical network. Video is ~70% of the tier-2 bill and one video is ~20 frames of the same image tower, so the frame-sampling policy is the only lever that matters there.

Tier 1 is deliberately not a small tier 2. Character n-grams survive tokenizer attacks (one invisible character breaks one word token but leaves surrounding character runs intact) where a word vocabulary would not. Its output is a routing decision, not a posterior, so its cleared items are folded wholesale into the bottom score band, safe only because the bottom band’s action is “nothing,” which is also tier 1’s verdict. Its operating point is a recall target (0.926) because its misses are unrecoverable: nothing downstream ever looks at what it clears. That recall is measurable only because 0.05% of its clears are adjudicated daily, sized so the miss rate is measurable in one day to about a tenth of itself.

One trunk, not ten models, for a data reason. Split the review band by queue mix and label volume runs opposite to severity:

spam                 2.23 M reviewed items/day    severity 1
hate speech          1.51 M                       severity 3
adult nudity         1.37 M                       severity 3
graphic violence     1.01 M                       severity 2
harassment           0.65 M                       severity 3
violent extremism    0.43 M                       severity 5
child safety         rounds to zero of reviewer time, severity 5

The heads that matter most have the least data. Ten separate models would give the child-safety model its own encoder and a handful of positives. A shared trunk hands it a representation paid for by spam’s 2.23 M adjudications a day, leaving it 769 parameters to fit. This is a data argument, not a compute one: ten encoders would be ~$11 M of GPU against a $520 M reviewer bill, which is affordable.

Why not a frozen general-purpose encoder with ten small heads on top? It would sidestep the retrain cadence, but it fails on the evasion mechanism: leetspeak, homoglyphs, zero-width joiners, and algospeak land in the input embedding, below every head, and no head retraining reaches them. A frozen trunk freezes exactly the layer that has to track the adversary. That forces a character-aware tokenizer (byte-level byte-pair encoding, where the vocabulary is learned from raw bytes so an unknown word degrades into byte fragments instead of a single “unknown” symbol) and a weekly-retrained trunk.

What the model eats

The empty row is the structural fact of this problem. In every recommendation problem the item has a history and the user is the variable; in integrity the item has no history at all, because you score it at the moment of publication. Three things elsewhere follow from that one empty row: queue priority uses projected views; the cluster detector runs offline because co-post timing must accumulate; and a recidivism hash is the only item-side history at t = 0, and even that is the asset’s history, not the post’s.

One row can quietly close the feedback loop. A feature counting the author’s prior removals is trained on labels the same classifier produced, so a wrongly-struck account gets a higher prior next time. Two controls, both needed: count only enforcement whose timestamp precedes the post (point-in-time correctness), and count only adjudicated (reviewer-confirmed) enforcement, which discards exactly the half the model generated itself. Then monitor false-positive rate by prior-enforcement bucket crossed with dialect, where the loop surfaces first.

Why static benchmarks decay

Every enforcement action is a free oracle query against your classifier. An adversary submits a post and the platform tells them, by acting or not, which side of the boundary it fell on:

auto-removals 0.96 M/day + post-review removals 0.78 M/day
    =  1.74 M labeled boundary probes returned free, per day

That is a black-box attack budget over a million queries a day, refreshed continuously, that no red team matches. The signature is a frozen number that holds still while the thing it measures does not:

adversarial benchmark frozen in January:
    recall in January                             0.78
    recall in July, same benchmark                0.78
    recall in July on FRESH July traffic          0.51

The benchmark did not move because a benchmark cannot move; the distribution moved. The fixes are process, not modeling: rolling eval sets (resample the last 30 days monthly; report “recall on content < 30 days old” as the headline and the frozen set as a tripwire only), recidivism tracking (hash every enforced asset; the recidivism rate directly measures how hard the boundary is being probed), and deliberate latency on the boundary. That last one is a genuine tension: fast enforcement is the dominant lever on harm and fast enforcement maximizes the adversary’s information rate. The resolution is asymmetric: act instantly above t_remove where the probe teaches little, and randomize timing only in the narrow band near the boundary where each observation is worth the most.

The one load-bearing number here is the decay from 0.78 to 0.51 on fresh traffic; it alone sets the retrain cadence and rules out the frozen-encoder alternative.

Metrics: prevalence, and why accuracy is meaningless

Accuracy is disposed of in one line: at a 0.0012 base rate, a model that outputs “benign” for everything is 99.88% accurate and catches nothing.

The model metric is recall at a fixed precision, per policy: reported at precision 0.95 (the auto-remove point), at precision 0.50 (the review-band shape), and as PR-AUC over the region any action uses. Never a single F1, and never one number across policies: hate speech at alpha 0.54 and spam at alpha 0.91 do not belong in the same average, because one has a ceiling four times closer than the other.

The product metric is prevalence, defined on views:

violating posts/day       2.4 M
mean views per violation    340
violating views/day       816 M
total views/day       180,000 M
baseline prevalence   816 / 180,000  =  0.45%  =  45 per 10,000 views

Views, not posts, because harm is delivered by viewing. A post removed after 5 views did almost nothing; one removed after 2 M views did all the harm the policy exists to prevent. The view distribution is heavy-tailed enough that the top 0.1% of violating posts carry more views than the bottom 90% combined, and a post-weighted recall number weights those identically.

Time-to-action is the dominant lever

Views are front-loaded; most of a post’s audience arrives in its first minutes:

age        4 min   10 min   18 min    1 h     6 h     24 h
cum views   0.04    0.13     0.22    0.41    0.68    0.87

A violation caught at time T has already delivered cumfrac(T) of its views and nothing after; a violation missed delivers all of them. Weighting by recall r and 1 - r:

residual prevalence  =  r x cumfrac(T)  +  (1 - r) x 1.0

Run that against four changes, using the 45-per-10k baseline:

Ten points of recall (two quarters of modeling) buys less prevalence reduction than cutting median action time from 18 minutes to 4. A nightly batch pipeline at the same recall is 1.55x worse than the baseline, which is the argument for classifying at publish time instead of on a schedule. Queue latency is a prevalence metric and belongs on the integrity dashboard next to recall.

The metric that hides harm

                     quarter start   quarter end
aggregate recall         0.62           0.71     +9 pts, celebrated
prevalence, views      23.2/10k       22.6/10k   barely moved

The residual formula predicts 20.1 per 10k at the quarter-end recall, but the measured value is 22.6, a gap of 2.5. That gap is the finding. The formula treats recall as one number, but aggregate recall is post-weighted while prevalence is view-weighted. The recall gain came almost entirely from spam and nudity (high-volume, low-reach policies where the model was already good) and those barely register in a view-weighted metric. A system can get substantially better at the content nobody sees. Report recall per policy, weighted by that policy’s share of violating views, or the headline keeps telling you about spam.

Serving architecture

Everything above assembles into the path a single post travels, priced in machines and in people. The headline is a ratio: the humans cost hundreds of times what the machines do.

flowchart TD
    P(["Post published"]) --> H{"Hash match<br/>PDQ · TMK · MinHash<br/>2 ms"}
    H -->|"known violating<br/>31% of violations"| ACT["Enforce immediately<br/>precision ~1.0"]
    H -->|miss| C1{"Tier 1 · linear<br/>hashed char n-grams<br/>0.4 ms · CPU"}
    C1 -->|"clears · 92%"| DONE(["No action<br/>sampled 0.05% for audit"])
    C1 -->|"suspicious · 8%"| C2["Tier 2 · fused encoder<br/>text + image + frames<br/>cross-attention · 12-210 ms"]
    CTX["Context features<br/>parent post · is-quote<br/>author-target relation<br/>audience"] -->|"joined BEFORE scoring"| C2
    C2 --> HEADS["10 sigmoid policy heads<br/>+ calibration per policy<br/>per language"]
    HEADS --> TIER{"Tier map"}
    TIER --> ACT
    TIER --> QUE["Review queue<br/>ranked by<br/>P x views x severity"]
    TIER --> DEM["Demote"]
    TIER --> DONE
    QUE --> REV(["Reviewer"])
    REV --> LAB[("Labels<br/>biased by the classifier")]
    ACT --> APP(["Appeals"])
    APP --> REV
    GRAPH[["Cluster detector<br/>co-post timing · shared assets<br/>creation cohort · runs offline"]] --> QUE

    style C1 fill:#bc6c25,color:#fff
    style C2 fill:#1d3557,color:#fff
    style TIER fill:#2d6a4f,color:#fff
    style CTX fill:#7f5539,color:#fff
    style QUE fill:#7209b7,color:#fff

A post hits the perceptual-hash table first (the cheapest test); a known-violating asset (31% of all violations are re-posts of something already enforced) is enforced immediately with no model involved. A miss falls to tier 1, which clears 92%, audits a twentieth of a percent of those clears, and routes 8% to tier 2. Tier 2 produces ten calibrated scores that enter the tier map. The cluster detector enters from the side, feeding the queue, because it acts on groups, not items.

Two nodes are worth isolating. CTX is joined before scoring, not after. Counter-speech and hate speech are indistinguishable in the 280 characters being scored, and a post-hoc adjustment to a finished score is not an input: routing an is-quote flag and the parent post into the encoder before cross-attention is what takes counter-speech false positives from 0.31 to 0.09. The 0.05% audit edge is the only measurement of the largest silent failure: tier 1 disposes of 1,840 M posts/day with no second opinion, so 0.05% (920 k items/day, ~315 reviewers, inside the 35% deduction) is adjudicated to produce the one number no other channel can, tier 1’s miss rate.

The cascade, priced

The whole tier-2 arithmetic bill is ~13,705 PFLOP/day, which is under one H100-day of pure compute. Real serving is batch- and bandwidth-bound, so call it tens of GPUs, ~$1.1 M/year at the generous end. The reviewers cost ~$520 M/year.

GPU floor    ~$19 k/year
real fleet   ~$1.1 M/year   (50 H100 at $2.50/GPU-hour)
reviewers    ~$520 M/year   (15,000 on shift daily at ~$14.61/productive hour)

humans cost 27,000x the GPU floor and 470x even a generous 50-GPU fleet.

One clarification that is the single most common error in this estimate: the 15,000 is a daily on-shift staffing level, not a headcount. The queue runs 24/7, so behind those 15,000 daily seats are roughly 21,900 people at ~250 shifts each. Reading it as a headcount is a 1.46x error. The $14.61/hour rate is derived (the $520 M budget divided by 35.6 M productive hours), fully loaded and blended across markets, and is the number to challenge, but even halving it, or inflating the GPU bill 100x, leaves the conclusion standing.

Optimize the constraint, and the constraint is people. Cutting queue volume 10% (a tier-1 precision win) frees ~3.56 M reviewer-hours, worth ~$52 M/year; a 20% GPU saving is worth ~$220 k. Within the GPU side, video dominates, so the only lever worth touching is frame sampling: scene-change sampling plus a dense pass over the last 8 seconds (the standard bait-and-switch placement) is 14 frames instead of 18, a 15% cut off the tier-2 bill, still two orders of magnitude below one percent of the reviewer bill.

Rollout: shadow, canary, ramp, holdback

A new version has to reach production without repeating the calibration regression below, which no offline gate catches because nothing offline knows where the threshold sits. A release has four stages.

Shadow runs the candidate on 100% of traffic and throws its output away, so it is observed under the production distribution while affecting nobody. It sees the score distribution (exactly the calibration failure mode) within an hour. Thresholds are recomputed here, on the gold set, per policy and per language, before anything is acted on. Run shadow a full week even though the volume is decisive in minutes, because the content mix is weekly-periodic.

Canary acts on 1% of traffic, the first slice allowed to affect real users, sized so damage is bounded. It catches everything downstream of an action that shadow is blind to: appeals, reviewer verdicts, user response. A regression that would add ~67,000 wrongful removals a day at full exposure shows up as a ~28-standard-deviation jump in appeals at 1% exposure, against only 9,600 removals of risk.

Ramp 1 → 5 → 25 → 100, with per-dialect false-positive rate as an independent gate at every step. A 4.7x disparity moves the aggregate metric only 0.4 points, so an aggregate gate clears at every rung whatever is happening to the slice; a gate that cannot see the failure is not a gate.

Holdback is a population deliberately treated with the old policy so you can keep measuring the new one, and it is demote-only. At the removal tier a permanent holdback means choosing to leave real harm in front of real people to measure yourself, and a removal’s two sides are not the same kind of object, so there is nothing to trade. Demotion is reversible, invisible, and both sides are views, so a 0.1% holdback there is fine. It also closes a hole: the demotion tier generates zero appeals, so it has no feedback channel at all. The audit sample tells you how many demotions were wrong; the holdback tells you whether demoting achieved anything, which matters, given the published band is net-negative.

Failure modes

Five ways this breaks in production, ordered by harm. The first is the only one where the model working exactly as designed is itself the failure.

Dialect and demographic bias in enforcement — the serious one

The bias runs through three measurable stages, and the standard dashboard cannot see any of it.

Step 1, annotation. Annotators who are not members of a speech community rate its speech as more offensive. African-American English (AAE) is a rule-governed dialect with its own grammar; Standard American English (SAE) is the prestige variety. On identical semantic content:

toxic-label rate:   SAE 0.081    AAE 0.178   (2.2x)

Step 2, the model learns the token, not the speaker. A reclaimed slur used in-group is the single strongest lexical predictor of the “hate speech” label, and it is high-frequency and consistent; the context that distinguishes in-group from directed use is low-frequency and inconsistent. Gradient descent nudges weights where the loss falls fastest, which is on the signal that is both frequent and consistent. So the model learns the token first and may never reach the context. It is not making an error; it is reproducing its labels efficiently.

Step 3, enforcement compounds it. Three different quantities in this chapter are called “false-positive rate,” over denominators that differ by ~800x, so fix the denominator first:

Using the first row: 1 - precision on hate-speech removals is 0.019 for SAE and 0.089 for AAE (4.7x), and because removals carry strikes and account restrictions, that becomes a 4.7x rate of account-level penalty on one community. But AAE is only ~5.7% of hate-speech-classified traffic, so the aggregate is 0.943 x 0.019 + 0.057 x 0.089 = 0.023, against 0.019 if AAE were fixed, a move of just 0.4 points. The aggregate metric is structurally incapable of showing a disparity that lands on a minority, and the smaller the community, the more invisible the harm. Slicing is the only instrument that can see it.

Four mitigations, in order of measured effect:

1. Recruit annotators from the speech community and give them speaker context. AAE false-positive rate 0.089 → 0.041, no model change. The largest lever, and it is a staffing decision.
2. Report FPR per dialect slice as an independent launch blocker.
3. Use the dialect signal for measurement only, never as a model feature. If dialect is an input, the model learns different effective thresholds per dialect, explicit differential enforcement built into the weights, indefensible in every forum it will eventually be examined in.
4. Counterfactual consistency training (penalize the score difference on dialect-paired rewrites): 0.041 → 0.028. Real, but the annotation fix bought 0.048 of gap and this buys 0.013, 3.7x smaller, and it is the one that needs a research project. That is the ordering lesson.

Context collapse

Context collapse is scoring a post as if its text were all there is, when it only makes sense against something outside itself. Three cases, each scored high and auto-removed:

"someone replied to my photo with [slur]. this is what i deal with daily."
    hate_speech 0.91  ->  auto-removed        (counter-speech)
"[slur] is a word with a history most people do not know. In 1948 ..."
    hate_speech 0.87  ->  auto-removed        (education)
quote-tweet of a public official, verbatim
    hate_speech 0.94  ->  auto-removed        (journalism)

The evidence that distinguishes these from the violating case (quotation structure, parent post, the author’s relation to the target) is outside the input window, and no amount of model capacity fixes a missing input. Supplying the channel:

counter-speech FPR   0.31  ->  0.09
news/quotation FPR   0.28  ->  0.11
satire FPR           0.34  ->  0.24      <- barely moves

Satire barely moves because its distinguishing evidence is often not in the thread at all: it is in whether the reader knows the account. When the evidence exists in no available channel, route to a human instead of raising the threshold, one more reason the review tier exists.

Coordinated evasion

Sometimes every item is honestly borderline and the violation exists only in the relationship between them:

200 accounts created in a 6-day window, each posting an image that is
pHash-identical modulo a 3-pixel border and a 1-degree rotation.

per-item score    0.41 - 0.47   below every threshold
cluster evidence  creation cohort · 94% co-post within 40 s · pHash distance
                  <= 6 · identical outbound domain  ->  posterior 0.998

You cannot threshold your way to this. The unit of decision has to change from the item to the cluster. An offline detector over co-post timing, shared-asset hashes, creation cohorts, and follow-graph overlap scores the cluster, and a cluster hit promotes every member into the review queue regardless of item score. It runs offline on a 15-minute cadence because graph features need a window to accumulate, a deliberate trade of latency for a signal that does not exist at publish time.

Calibration drift

A strictly better model, shipped correctly, that destroys enforcement because a number in a config file did not move with it:

model v11 -> v12, same architecture, +14% training data
    PR-AUC 0.681 -> 0.694 (better);  score at precision 0.95: 0.94 -> 0.89
    t_remove left at 0.94:  removals/day 0.96 M -> 0.61 M
                            SYSTEM recall 0.71 -> 0.57

The auto-remove tier is 0.38 of the 0.71 system recall; cutting its removals to 0.61/0.96 of itself scales that contribution down by 0.14. A model-version change moved one tier, and the number everyone reports moved by less than half as much, which is exactly why the regression survives an offline dashboard that reports curve-level metrics. Thresholds are per-model-version artifacts, recomputed on the gold set as a release step, and re-derived per language, because calibration differs by language far more than PR-AUC does.

The label feedback loop

The classifier chooses which items get labeled, and those labels train the next classifier, so the system can stop learning anything it does not already believe.

flowchart LR
    M["Classifier"] -->|"scores high"| Q["Review queue"]
    Q --> R["Reviewer labels"]
    R -->|"train"| M
    M -->|"scores low"| U["Never reviewed,<br/>never labeled"]
    P["Prevalence sample<br/>uniform over views"] -->|"importance-weighted"| R

    style M fill:#1d3557,color:#fff
    style P fill:#2d6a4f,color:#fff
    style U fill:#9d0208,color:#fff

Content the model scores low is never reviewed, labeled, or learned, so the training distribution converges to the model’s own current beliefs. The break is the uniform-random prevalence sample: 12 k items/day over views, adjudicated and mixed into training with an importance weight (a multiplier that compensates for how rarely such rows are sampled, so a small sample can speak for the population). It is 0.15% of the label volume and the only part that can discover a violation type the model has never scored highly. The exploration band below t_review is the targeted version of the same instrument: one sample tells you the model has a blind spot, the other tells you where its edge is.

Summary

Appeals are part of the design, not a support function

An enforcement system without an appeals path has an unobservable false-positive rate, because appeals are the only channel through which a wrongly-removed user can reach the system. But that channel is a poor estimator and a good alarm:

auto-removals/day        0.96 M
appeal rate              4.2%   ->  40,300 appeals/day
overturn rate            11%    ->   4,430 overturns/day

audited FP rate at t_remove is 5.0%:  real wrong removals = 48,000/day
sensitivity  =  4,430 / 48,000  =  9%

Appeals surface 9% of the real errors: the other 91% never appeal, because appealing requires noticing, caring, and knowing how. So the FP estimate comes from the audited gold sample; appeals are a regression canary, because a 3x jump in appeal rate on a policy shows up within hours of a bad release while the audit sample takes a week to reach significance.

Three underrated uses stand out. Overturn rate sliced by dialect and country is one of the cleanest bias detectors available, because it is real people re-judging real enforcement, not a synthetic probe. The demotion tier generates zero appeals (nobody is told), so it needs a deliberate audit sample or it is unmeasured forever. And appeals cost capacity with a sign: raising auto-remove recall raises appeals, which consumes the capacity that sets t_review. A bad release that drops precision from 0.95 to 0.88 scales appeals ~2.4x, quietly taking ~1.5% of proactive capacity and concentrating the loss in exactly the policy that regressed. That is the argument for the precision floor being a hard gate, not a target: it protects the capacity budget, not just the users.

Alternatives considered and rejected

Conclusion

The load-bearing takeaways, most of which are not about the model:

• The policy document is the highest-leverage artifact. Rewriting one policy moved model F1 from 0.61 to 0.70 with no model change, because it raised the agreement ceiling every measured number sits under. If two trained humans cannot agree, you cannot train or evaluate a model.
• There is no single threshold, because the two error costs are not exchangeable. Only the demote tier has views on both sides of its ledger and earns an expected-value break-even; the removal floors are policy commitments, and t_review is a quantile read off a staffing table, not a PR curve.
• The binding constraint is people, not machines. Reviewers cost 470x-27,000x the compute, so every optimization should aim at queue volume and reviewer time.
• Measure the product on views, not posts, and act fast. Cutting median time-to-action from 18 minutes to 4 beats two quarters of modeling, and aggregate post-weighted recall can rise while view-weighted harm does not budge.
• Every design choice has to survive an adversary and a feedback loop. Enforcement leaks a million boundary probes a day, the queue is sampled by the model it trains, and a minority-community disparity is invisible in any aggregate, so rolling eval sets, a uniform-over-views prevalence sample, and per-slice reporting are not extras.

One line to remember: in content moderation the model is the cheap part, and the decisions that matter live in the policy document, the thresholds, and the reviewer capacity around it.

Further reading

• Sap, Card, Gabriel, Choi, and Smith, The Risk of Racial Bias in Hate Speech Detection, ACL 2019: the measured AAE annotation and enforcement disparity.
• Borkan, Dixon, Sorensen, Thain, and Vasserman, Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification, WWW 2019: per-slice fairness metrics for moderation classifiers.
• Krippendorff, Content Analysis: An Introduction to Its Methodology: the definition and interpretation of the alpha agreement coefficient.
• Meta / ThreatExchange, PDQ and TMK+PDQF perceptual-hashing algorithms (open source): the recidivism-matching primitives.
• Dosovitskiy et al., An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021: the ViT-B/16 image tower.

Next: the video-recommender chapter.