How to design harmful-content detection

Content moderation is a labeling problem wrapped around an action problem: the policy is the label definition, and the two error costs (missed harm vs. a silenced legitimate user) are borne by different parties and cannot be traded.

Problem shape (2B posts/day worked example)

• Multi-label, not multi-class: ten independent sigmoid heads on one shared trunk, all high at once. Softmax is wrong because a post can be threat + hate + harassment together, and each policy has its own consequence, reviewer, appeal, and legal path.
• Prevalence (this domain’s word for base rate) ~0.12% of posts, 2.4M/day. Severity (1-5) is a policy-owner’s harm judgement, separate from the score P(violates k); they multiply in the action decision.
• Binding constraint is queue latency, not model latency. Human review capacity ~8.15M items/day = 0.41% of content.
• Item has no engagement history at scoring time (post is milliseconds old) — the empty feature row that shapes everything.

The label is the hard part

• Krippendorff’s alpha = inter-annotator agreement, chance-corrected (1.0 agree always, 0 = guessing). It caps every model metric: a perfect model still disagrees with the eval set as often as the eval set is wrong (noise ceiling).
• Implied single-reviewer accuracy q from agreement = q² + (1-q)². Trust it at high alpha, treat low-alpha values as order-of-magnitude.
• Biggest quality win is often editing the policy doc, not the model: hate-speech v3→v4 raised alpha 0.54→0.68 and F1 0.61→0.70 with the same model.

Label sources (bias > volume)

• Only the gold set (adjudicated, stratified — for eval/thresholds) and the prevalence sample (12k/day, adjudicated, uniform over views — the only unbiased prevalence estimate) are designed to be unbiased.
• User reports (~0.06 precision, brigadable): route to queue, never a training positive or feature. Queue labels are sampled by the classifier (recall on them = precision in disguise; model converges to its own beliefs).
• Training set: positives = adjudicated decisions only; exclude reports and hash matches. Negatives need the tier-1 audit sample (the only negatives from the 92% of traffic the model runs on).

Training gotchas

• Loss = masked multi-label BCE. Mask because a reviewer who ruled spam did not rule the other 9 policies — unobserved ≠ negative (would mislabel ~25% of true positive pairs).
• Prior/base-rate shift: training mix ~11% positive vs. 0.12% production (~100x). Fix = subtract ~4.63 from the logit. Calibration = when model says 0.30, ~30% really violate; separate from ranking; every threshold depends on it.
• Split by asset-hash cluster + author, then by time. 31% of violations are near-duplicates; a row-level split memorizes them (holdout recall 0.78 inflated → 0.68 honest).
• Retrain weekly: fresh-traffic recall decays 0.78→0.51 over 6 months (~1.6%/week). Every retrain recomputes all four thresholds per policy per language.
• Log the propensity (P that a row was allowed to be labeled) — t_review is a daily quantile. Keep a stochastic exploration band (1% capacity below the cut) so no region has propensity 0.

The four thresholds (you cannot pick one)

Single-threshold rule act if P > C_FP/(C_FP+C_FN) fails: an 8:1 ratio at 0.12% prevalence removes 12.5M legitimate posts/day. Real formulation is constrained: maximize recall s.t. precision ≥ floor, reviewed ≤ capacity.

• Only demote has views on both sides, so only it earns an expected-value break-even. Break-even: demote pays when in-band precision > w/(1+w) ≈ w. With w=0.02, break-even 1.96%. Published band 0.38-0.55 has in-band precision 0.98% — net-negative, must move up.
• Use marginal (in-band) precision, never cumulative — cumulative makes a marginal tier look ~4.5x better.
• t_review = the score cutting the top 0.41% (a quantile = share above it, not a score property). Recompute from live queue depth; a staffing shock moved it 0.55→0.62 and cost ~6.3 recall points with no model change.
• Prioritize queue by expected harm = P × projected_views × severity, not score. Worth 2.3x; reach spans 4 orders of magnitude, score only ~1.7x.

score >= 0.94  -> auto-remove (+strike if severity>=4)
     >= t_review(0.55, capacity) -> human queue
     >= t_demote(>=1.96%, not 0.38) -> demote, no notice
     else -> no action

Model + arms race

• Cascade: perceptual-hash recidivism match (catches 31% at precision ~1.0, 2ms) → tier-1 linear over hashed char 3-5-grams (recall 0.926, routing decision not a posterior) → tier-2 fused encoder (241M params) on the 8% that route up.
• Cross-attention fusion required for image+text: a benign image + benign caption can be a threat together; late fusion can’t represent a conjunction in neither channel. Costs ~1.7x image encoder, affordable on 8%.
• One trunk, ten heads for a data reason: label volume runs opposite to severity (spam 2.23M/day sev 1; child safety ~0 reviewer time, sev 5). Shared trunk sells severity-5 heads spam’s representation (769 params to fit). Compute is the weak argument ($11M vs $520M reviewer bill).
• Frozen encoder rejected: evasion (leetspeak, homoglyphs, zero-width joiners, algospeak) lands in the input embedding below every head; a frozen trunk freezes the layer that must track the adversary. Use char-aware byte-level BPE.
• Every enforcement is a free oracle query: ~1.74M labeled boundary probes/day. Act instantly above t_remove (probe teaches little); randomize timing only near the boundary.

Metrics

• Accuracy is meaningless: 99.88% by always predicting benign. Report recall at fixed precision, per policy (never one F1 across policies — different alpha ceilings).
• Product metric = prevalence on views (harm delivered by viewing): baseline 816M violating views / 180,000M = 0.45% = 45/10k. Top 0.1% of violating posts outweigh the bottom 90%.
• Time-to-action dominates: views are front-loaded. residual = r·cumfrac(T) + (1-r). Cutting median action 18min→4min beats 10 recall points; a 6h batch job is 1.55x worse than baseline.
• Aggregate post-weighted recall can rise +9pts while view-weighted prevalence barely moves (gains hide in high-volume low-reach policies like spam). Weight recall by each policy’s share of violating views.

Failure modes (ordered by harm)

• Dialect bias is the serious one (model working as designed = the failure): AAE labeled toxic 2.2x more than SAE on identical content → gradient descent learns the token not the speaker → hate-speech 1-precision 0.019 SAE vs 0.089 AAE (4.7x), invisible in aggregate (moves it 0.4pts). Biggest fix is staffing: community annotators drop AAE FPR 0.089→0.041, bigger than counterfactual training (0.041→0.028).
• Calibration drift: a strictly better model (PR-AUC 0.681→0.694) with a stale t_remove=0.94 cut system recall 0.71→0.57 — invisible offline.
• Three quantities all called “false-positive rate” differ by ~800x in denominator — fix the denominator first.

Numbers that anchor everything

• Reviewers cost ~$520M/year vs ~$1.1M GPU fleet: humans cost 470x-27,000x the machines. Optimize queue volume (10% cut ≈ $52M/year).
• The 15,000 is daily on-shift seats, not headcount (~21,900 people at ~250 shifts) — reading it as headcount is a 1.46x error.
• Rollout: shadow (100%, output discarded, recompute thresholds) → canary (1%, catches post-action effects) → ramp 1→5→25→100 with per-dialect FPR gate at every step → holdback (demote-only; removal holdback would leave real harm to measure yourself).
• Appeals: 9% sensitivity (91% never appeal) — a regression canary, not the FP estimator (that’s the audit sample). Overturn rate sliced by dialect is a clean bias detector.