How to design image-privacy blurring

In this lesson, we’ll design a system that finds every human face and every readable vehicle licence plate in a planet’s worth of street-level photography, and destroys that information before the imagery is published. A 13,312 × 6,656 pixel spherical photograph, taken from a moving vehicle, goes in; the same photograph comes out with every face and plate replaced by a block of destroyed pixels, and nothing else touched. By the end you’ll be able to derive the blur threshold from costs, explain why per-face recall is the wrong unit to gate on, and say why the standard detector metric misleads here.

Four ideas carry the lesson:

• How to derive an operating threshold from costs instead of guessing one.
• Why that derivation produces a degenerate answer the first time, and how to repair it.
• Why small objects are hard, for a reason that lives in a training-time assignment rule.
• Why the standard object-detection metric is the wrong one to report here.

What makes this different from ordinary detection

Three facts reshape the whole design.

1. The error asymmetry is extreme, and it is legal, not commercial. A false negative (FN) (a real face the system failed to blur) is a privacy violation with a regulatory tail. A false positive (FP) (a region blurred that contained no face) is a slightly over-blurred patch of wall. The two costs are roughly four orders of magnitude apart, so the operating point is derived, not chosen.
2. It is offline. Weeks pass between capture and publication, so nothing has a latency budget. This is a batch pipeline over petabytes, and the currency is cost per image, not milliseconds per request. Latency, autoscaling, caching, and model warm-up all drop out of the design.
3. mAP is the wrong number to report. mAP (mean average precision) is the standard detector score: for each class you sweep the confidence threshold from strict to permissive, record how precision falls as recall rises, average precision across that curve, then average across classes. Here the right number is a per-slice miss rate whose worst individual cell is the launch gate.

Two ideas from elsewhere on this site are used in a stronger form below, and both are restated in plain words where they appear: class imbalance (the imbalanced-data lesson), which here shows up as roughly 58,000 background locations for every face; and choosing a threshold from a cost matrix (the metrics lesson), which the next section applies and then watches break.

The models in this system

There are only two learned models in the whole system. Most of the recall is bought by geometry and by pipeline structure, not by modelling.

Three components are often mistaken for models and are not. Multi-view association is pure geometry: it projects a detection from one photograph into the next using the vehicle’s recorded pose and a depth map, with nothing learned. Local tone mapping is a fixed image-processing pass triggered by a histogram test. Redaction is arithmetic on pixel blocks. Recognising which parts are learned and which are not is most of what makes this design cheap.

Five assumptions carry weight; each, if false, breaks a specific argument later.

1. The annotation guideline defines what a face is. If “face” is undefined then “recall” is undefined, and every number here inherits that definition.
2. The cost of a spurious blur grows faster than linearly in total blurred area (measured as P ≈ 0.017 · (area%)^1.40). If that curve were a straight line, the whole threshold argument reverses.
3. The capture rig already produces accurate vehicle pose and a depth map, because it needs them to stitch panoramas. Multi-view association gets its benefit for free from that, and would be unaffordable otherwise.
4. Misses are correlated across views, worth about 1.80 effective independent views out of four. Assuming independence overstates the benefit enormously.
5. Model-assisted labels are 9% incomplete in exactly the hard cases. This is the most fragile assumption, and it is why evaluation uses a separate blind control.

Framing

Six facts anchor every later decision.

That last row matters most. If “face” is undefined, “recall” is undefined, and two teams can report 0.99 and 0.96 on the same model because one counts the back of a head. Write the annotation guideline first; it is the specification of the ML objective.

Deriving the operating point

The confidence threshold is derived three times before it comes out right: once by the textbook rule (which destroys the product), once with a repaired cost model (a defensible optimum the product still refuses), and once as a constrained optimization (the number that ships). Even that number cannot reach the real target, which is what forces the geometric method later.

The cost-ratio threshold, and its degenerate answer

A detector emits a confidence score in [0, 1] for each candidate box, and the threshold t is the score above which you act (here, blur). If a false positive costs C_fp and a false negative costs C_fn, expected cost is minimized by acting whenever the probability of a real face exceeds C_fp / (C_fp + C_fn). The intuition: if missing is a thousand times worse than over-blurring, act on anything with more than about a thousandth of a chance of being real.

Neither cost is a sticker price. Each is an expectation down a chain of conditional events, and every link is a place someone can argue with the estimate:

C_fn = 0.62 (real, identifiable person)   -- many "faces" are posters, mannequins, statues
     × 0.013 (noticed and reported)       -- almost nobody browses imagery of their own street
     × $1,000 (takedown $40 + amortized regulatory/legal $960)
     = $8.06 per missed face

C_fp = 0.004 (visible enough to matter) × 0.02 (complaint | visible) × $11
     = $0.00088 per spurious blurred box

t* = 0.00088 / (0.00088 + 8.06) = 1.09e-4

A threshold of 0.0001 has a predictable effect on a dense detector, which scores tens of thousands of candidate positions per tile with almost all of those scores sitting just above zero, so a threshold this low accepts essentially every candidate. Measured on real panoramas, t = 1e-4 emits about 4,100 boxes per panorama and blurs 43% of the image area. The product is destroyed.

The formula is not wrong; the input is. C_fp was modelled as a constant per box, and it is not. The cost of over-blurring is convex in total blurred area (it curves upward, so each added unit of blur hurts more than the last) because blur goes from invisible to visible to annoying to disqualifying as it accumulates. Two hundred small boxes are not two hundred times the harm of one.

The corrected cost model

Rather than assume that curve, measure it. Sample panoramas across a range of blur coverage and record how often reviewers call the image unusable:

blurred area   P(unusable):   0.1%→0.001   0.5%→0.004   1.0%→0.012
                              2.0%→0.045   5.0%→0.19    10.0%→0.48

least squares in log space (a straight line in log-log is exactly y = a·x^b):
   P ≈ 0.017 · (area%)^1.40      superlinear, as expected

At an internal cost of $0.15 per unusable panorama and n = 3.2 faces in an average one, expected cost per panorama is the privacy cost miss_rate(t) · n · C_fn plus the blur cost 0.15 · 0.017 · area(t)^1.40. Recall, FP count, and blurred area are all measured at each threshold; the cost columns follow:

The unconstrained optimum sits at t = 0.005, which blurs 3.2% of every image. That is the honest answer to the arithmetic, and it is one no imagery product will accept. 3.2% coverage means visible grey patches on most facades.

The shipped threshold comes from a constraint, not the optimum

When the cost-minimizing point is unacceptable, the shipped point comes from a constraint, and naming which constraint binds is the answer:

maximize    recall(t)
subject to  blurred_area(t) <= 0.5%          (the product's blur budget)

-> t = 0.035, recall 0.9955, 4.5 misses / 1,000 faces, 14 FP boxes

The binding constraint is the product’s blur budget, not the legal cost.

A constraint has two numbers, easy to confuse. The cap is 0.5%, a product decision. The spend is what the shipped threshold actually blurs, a measurement, and smaller. At t = 0.035 the spend is 0.40%. (Reading it off requires interpolating the area column in log space, because that column is a power law; a straight-line reading gives 0.495%, which lands suspiciously close to the cap and is simply the wrong rule for a convex column.)

The 0.10 points between spend and cap are not headroom. They are the margin the dilation step will spend. Every published box is grown 15% before blurring, which multiplies area by 1.15² = 1.32, so t = 0.035 actually publishes 0.40 × 1.32 = 0.53%, 6% over budget. That is why the threshold later moves to t = 0.045. The constraint that governs is on the dilated area, because the dilated blur is what publishes.

The cost model still earns its place: it prices the budget. Relaxing the cap from 0.5% to 1.0% drops the miss cost from about $0.116 to at most $0.064 per panorama, $0.052 × 2e9 = $103M of expected privacy cost per year, and that is a floor. The area budget is not a taste question; it has a price tag.

flowchart TD
    A["C_fn = $8.06 per missed face<br/>C_fp = $0.00088 per spurious box"] --> B["Cost-ratio rule<br/>t* = 1.09e-4"]
    B --> C{"What does that<br/>threshold actually do?"}
    C -->|"4,100 boxes/panorama<br/>43% of image blurred"| D["DEGENERATE<br/>C_fp is not linear in box count"]
    D --> E["Re-model: blur cost is<br/>convex in TOTAL AREA<br/>P_unusable = 0.017 · area^1.4"]
    E --> F["Unconstrained optimum<br/>t = 0.005 · 3.2% area"]
    F --> G{"Acceptable to the product?"}
    G -->|no| H["Constrained optimum<br/>max recall s.t. area <= 0.5%<br/>t = 0.035 · recall 0.9955<br/>0.40% raw · 0.53% dilated"]

    style B fill:#1d3557,color:#fff
    style D fill:#9d0208,color:#fff
    style E fill:#bc6c25,color:#fff
    style H fill:#2d6a4f,color:#fff

No terminal box is a guessed number: each is derived or rejected for a stated reason.

Per-face recall is the wrong unit

One line of arithmetic changes what the whole design optimizes. A per-face recall of 99.55% sounds excellent, but a user does not experience one face. They experience one photograph, which may hold many. Compound the miss rate over an image with P(≥1 miss) = 1 - (1 - m)^n:

suburban, n = 3.2, m = 0.0045:   1 - 0.9955^3.2  =  1.43%   ->  28.6M panoramas/yr with an unblurred face
dense urban, n = 40:             1 - 0.9955^40   = 16.5%    ->  one in six

To hold a dense panorama to a 1% chance of any miss you would need (1 - m)^40 = 0.99, i.e. m = 2.51e-4, 18× below the shipped 0.0045. That is not reachable by threshold tuning inside the area budget. So the recall you need cannot come from the detector; it has to come from redundancy.

Multi-view association

The highest-leverage idea in the lesson is not a modelling idea: the same person is photographed several times, so a face missed in one frame can be recovered from another using geometry the rig already computes.

The capture vehicle triggers a panorama every 8 metres, so a pedestrian appears in 4 to 6 consecutive frames, 0.72 s apart at 40 km/h. A detection in any frame can be projected into the others and the union blurred. The projection needs exactly three inputs, all already computed to stitch panoramas:

Worked through one pedestrian: a face detected at bearing 30° and depth 12 m in frame 2 back-projects to a 3D point, moves 8 m with the vehicle, and reprojects into frame 3 at bearing 68.26°, width 52.5 px, up from 28.2 px. The bearing swung 38.3°, which is 1,415 pixels, and the face nearly doubled in size between consecutive frames.

That is why this is not object tracking. Tracking matches boxes that overlap, measured by IoU (intersection over union: shared area divided by combined area, 1 for identical boxes, 0 for disjoint ones). Here the two boxes do not touch, so their IoU is exactly zero. Association happens in world coordinates or not at all.

The projection is depth-sensitive, and that decides the algorithm. Perturb only the assumed depth and re-run:

assumed depth   bearing in frame 3   error against truth
   11.0 m           74.49°               +230 px
   12.0 m (true)    68.26°                  0
   14.0 m           59.49°              −324 px

A 1 m depth error (8%) displaces the reprojected box by up to 230 px, four times the width of the 52 px face it is meant to cover. Blurring the reprojected box safely would need a 52.5 + 2·230 = 513 px square, which is a hole in the image. So the reprojection is used as a gate, not an answer:

1. Gate. Back-project every frame-3 detection, down to confidence 0.002, and keep those within R = 1.5 m of the frame-2 point. That radius is a physical budget: a pedestrian at 1.4 m/s covers 1.4 × 0.72 = 1.01 m between triggers, plus 2σ = 0.50 m of depth noise, giving 1.51 m.
2. Promote. If a below-threshold detection falls inside the gate, blur its box, not the reprojected one. This is the common path and it costs no extra area, because the box is a real detection with real localization; the geometry only supplied permission to accept it at low confidence.
3. Fall back. Only when the gate is empty is the reprojected region blurred, a quadrilateral (projecting a rectangle does not leave it rectangular), dilated by the depth uncertainty.

flowchart LR
    D2["Frame 2 detection<br/>score 0.61 · 30° · 12 m"] --> BP["Back-project<br/>bearing + depth → 3D point"]
    BP --> POSE["Apply pose delta<br/>8 m forward"]
    POSE --> G{"Any frame-3 detection<br/>within 1.5 m in 3D?"}
    G -->|"yes · 82%"| PROM["PROMOTE that box<br/>blur it · zero extra area"]
    G -->|"no · 18%"| FB["Blur reprojected quad<br/>dilated by depth σ<br/>0.00076% of image area"]
    PROM --> U["Union into the blur set"]
    FB --> U

    style G fill:#bc6c25,color:#fff
    style PROM fill:#2d6a4f,color:#fff
    style FB fill:#1d3557,color:#fff

The expensive fallback is affordable because it only fires on the 0.45% of faces the detector already missed, and only 18% of those find an empty gate, about 0.0026 fallback boxes per panorama, costing 0.00076% of image area, or 1/650th of the blur budget. That is the shape of the whole design: buy recall wherever the volume is small enough that the area cost rounds to nothing.

Two things bound it. Mis-association in a crowd is benign: a loose gate can grab the person standing next to the intended one, and that face needed blurring anyway, so the gate is loose on purpose. Depth failure on glass is not benign: lidar returns from the interior behind a shop window, so a reflected face lands metres too deep and its gate hits empty pavement. Association therefore does not rescue reflections at all.

Propagation must be thresholded too, because a false positive projects into neighbouring frames exactly as well as a true detection. A face spans about four frames, so a panorama receives from ~3 neighbours; propagating everything at t = 0.035 (14 FP each) gives 14 + 3·14 = 56 FP boxes, a 4× blow-up. Gating propagation at score ≥ 0.50 (0.4 FP each) gives 14 + 3·0.4 = 15.2, a 9% increase instead. Detections between 0.035 and 0.50 are still blurred where found; they just cannot paint boxes into frames that never saw them.

The payoff justifies the machinery. If per-frame misses were independent across 4 views, the joint miss rate would be 0.0045^4 = 4.1e-10 and the problem would vanish. It does not: a 9-pixel, backlit, half-occluded face tends to be all three in every frame, so the same views fail together. Measured on 5,000 tracked pedestrians, the joint miss rate is 6.1e-5, 150,000× worse than the fantasy. Solving m^k_eff = 6.1e-5 gives k_eff = 1.80 effective independent views. That takes the dense-urban panorama miss rate from 16.5% to 0.24%, at zero inference cost, because the frames were already captured and already scored.

This is the highest-leverage element in the lesson, and it is not a modelling idea. It requires only pose and depth, which the rig already produces for stitching.

The ML objective and the architecture

Objective

The detector takes one 1024²-pixel tile and emits a set of boxes, each with a class in {face, plate} and a confidence score. Training minimizes a loss, a single number measuring how wrong the model is, which gradient descent pushes down:

L = L_cls + lambda_box · L_box
   L_cls = focal loss (alpha 0.25, gamma 2.0)
   L_box = GIoU on positive assignments

Focal loss is used instead of plain cross-entropy for an arithmetic reason. Cross-entropy charges the negative log of the probability assigned to the correct answer. A dense detector scores every cell of every pyramid level (87,296 locations at 1024²) against roughly 1.5 real objects per tile, an imbalance near 58,000:1. Each easy negative contributes almost nothing, but 87,000 of them summed swamp the positives, and the model learns to say “background” everywhere. Focal loss multiplies the cross-entropy by a factor that shrinks toward zero as an example gets easy; gamma = 2.0 sets how aggressively, alpha = 0.25 re-weights positive against negative. (Derivation in the imbalanced-data lesson.)

GIoU (generalized IoU) is used for the box term because plain IoU is flat at zero when boxes do not overlap and so gives no gradient; GIoU adds a term from the smallest enclosing box, which keeps pulling even when the boxes are disjoint.

Set lambda_box low (around 0.3 instead of the usual 2.0) because localization precision is nearly worthless here. A box 20% too large blurs slightly more wall; a box 5% too small leaves an eye visible. The loss should be asymmetric in the same direction the product is, and the cheapest way is to under-weight the box term and then dilate every box at blur time.

Architecture comparison

Four detector families could do this job. A word on each first:

• Two-stage (Faster R-CNN): a region proposal network (RPN) suggests where objects might be, then a classifier scores each proposal. Accurate and expensive.
• Single-stage, anchor-based (RetinaNet): classifies every location directly, comparing each against a fixed set of template boxes called anchors.
• Single-stage, anchor-free (FCOS): drops the templates; each location predicts distances to the box edges directly.
• DETR (detection transformer): emits a fixed number of object “queries” that compete to claim objects, needing no duplicate removal.

NMS (non-maximum suppression) is the duplicate-removal step DETR avoids: when several boxes describe one object, keep the highest-scoring and delete the rest. GFLOP is a billion floating-point operations.

Anchor-free wins on the column that decides this problem: it removes the anchor-assignment rule that silently labels small faces as background. “Center sampling” is its replacement rule: a location is positive if it falls inside the object’s box (or a shrunken central region), with no overlap test at all. DETR is rejected on a specific ground: its 100-slot query budget cannot represent a scene with more objects than that, and crowded scenes are exactly where the panorama-level risk lives.

What the model eats, and what is withheld

There are no engineered features. The input is one tile (a 3 × 1024 × 1024 array of 8-bit RGB, cropped from the undistorted equirectangular panorama and normalized per channel) and nothing else. The learned representation is the feature pyramid, so the design work is choosing which resolutions exist, not which columns to compute.

What is available and deliberately withheld is worth defending. Every panorama carries capture time, satellite position, exposure and gain, and a region code. Feed any of these in and the model acquires a prior: a dusk capture in a low-density suburb really does hold fewer faces, so a more sceptical model scores better overall. The problem is where that scepticism lands. It lowers scores in exactly the low-light, low-density cells that are already the worst-performing ones. A geolocation-and-time feature buys mAP, and it buys it out of the worst cell of the slice grid. So the metadata is used only outside the model: exposure and gain select the tone-mapping pre-pass; pose and depth drive association. None of it reaches the part of the network that decides whether a face is present.

The small-object problem

Small faces are hard, and the cause is not “the model needs more data.” Two mechanisms produce it (a training-time assignment rule and a resolution limit), each with a different fix.

How small is a face, in pixels

The rig produces a 13,312 × 6,656 equirectangular panorama covering 360°. Equirectangular means the sphere of directions is unrolled onto a rectangle so a fixed pixel count maps to a fixed angle everywhere, the property the arithmetic relies on:

angular resolution = 13,312 / 360 = 37.0 px per degree

a 0.16 m face at distance D spans about 9.17/D degrees (small-angle);
× 36.98 px/deg:
   D =  5 m → 68 px       D = 30 m → 11 px
   D = 15 m → 23 px       D = 50 m →  7 px  (below any detection floor)

Measured over real captures, the face-size distribution is: <8 px 4.1% (out of scope, not identifiable), 8–16 px 19.3%, 16–32 px 34.8%, 32–64 px 27.1%, >64 px 14.7%. With sub-8px faces out of scope, the in-scope denominator is 95.9%, so the 8–16 px band is 19.3 / 95.9 = 20.1% of in-scope faces, and it produces 62% of the misses. Every architectural decision below is about that band.

Why anchors label small faces as background

An anchor-based detector compares each location against template boxes, and an assignment rule decides which templates count as containing an object during training:

• IoU > 0.5 with a ground-truth box → positive (train the model to fire).
• IoU 0.4–0.5 → ignore (no gradient).
• IoU < 0.4 → negative (train the model to say background).

Standard anchors are {32, 64, 128, 256, 512} px. Take a centered 11 px face against the smallest, 32 px. The small box sits inside the large one, so IoU = 11² / 32² = 0.118, which is below 0.4. The location is assigned as a negative, and the classification loss trains the model to output “background” exactly where a face is. Sweeping the face size, nothing between 8 px and 22 px ever becomes a positive: an 11 px face is 0.118, a 16 px face 0.250, and only at 23 px does IoU cross 0.5.

The assignment rule installs a hard detection floor at ~23 px, a face at 15 metres. Three fixes:

Why the feature-pyramid level matters

A convolutional network predicts from a feature map, a grid of learned descriptors, each summarizing a patch of the image. The stride is how many input pixels separate two cells, and a feature pyramid is a set of maps at several strides (P2–P6 at strides 4, 8, 16, 32, 64) so large objects are found on coarse maps and small ones on fine maps. At stride s, a face f pixels wide occupies f/s cells:

Below one cell, the face is a sub-cell signal. Every cell that sees it also sees a lot of pavement and wall, so its evidence is averaged away. An 11 px face is 0.69 cells at P4, so it needs P2, the stride-4 level. That is where the cost is: the detection head (the small network run at every cell to produce a score and box) runs once per cell, and halving the stride quadruples the grid, so a full-width head at P2 costs 619 GFLOP, three times the other four levels combined.

The fix is a cheaper head at P2: depthwise-separable convolutions (a per-channel spatial filter then a 1×1 mixing step), 128 channels instead of 256, 2 layers instead of 4. That costs 9.2 GFLOP (67× cheaper) and is defensible because finding small, low-detail objects at P2 is a genuinely easier task than at P5, where a “face” can be a mannequin, poster, statue, or reflection.

Adding it up gives the 410 GFLOP/tile figure: backbone (ResNet-50 scaled to 1024²) 171, FPN 25, heads P3–P6 205, lightweight P2 head 9. What the 9 GFLOP buys: recall on the 8–16 px band goes from 0.42 to 0.89, more than doubling recall on the band carrying 62% of misses.

Data and labels

Annotation costs, and the surprise in them

Two activities cost money: drawing a box, and scanning the image to find faces. At a $22/hour annotator rate:

DRAWING   ~7 s/box + 40% QA overhead          = $0.060 per box
SCANNING  ~4 min to sweep an 88.6 MP panorama = $1.47 per panorama

per panorama: 5 boxes × $0.060 = $0.30 of boxes + $1.47 scan = $1.77
              boxes are 17% of it

The scan dominates the drawing by 5 to 1, which reorganizes the strategy:

1. Annotate tiles, not panoramas. A 1024² tile scans in 6 s, not 4 min. You lose the exhaustive-coverage guarantee and buy it back statistically: sample tiles with a known probability and reweight by its inverse.
2. Pre-annotate with the current model. Correcting a proposed box takes 1.2 s against 7 s to draw one, a 5.8× speedup.
3. Pre-annotation has a recall of its own, and if you do not measure it your model’s recall is fiction. Annotators anchored on proposals miss whatever the model missed, because nothing on screen draws their eye to it.

Measure that with a blind 5% control: annotate a random 5% of tiles with no proposals shown.

blind annotation:           1,412 faces
model-assisted, same tiles: 1,285 faces
label-process recall = 1,285 / 1,412 = 0.910

A model measured at 0.9955 against assisted labels is measured against a ground truth that is itself 9% incomplete, in exactly the hard band the model also misses. This is the most consequential number in the lesson, and it is why the control must be permanent, not a one-time audit.

Active learning: what to label next

Active learning lets the current model choose what gets labelled next. Random sampling is the wrong default here: 78% of tiles contain nothing, and 62% of the risk sits in one size band.

Multi-view disagreement is free, self-labelling, and 18× more efficient than random, because the geometry supplies a near-certain positive: a face at 0.94 in frame 2 and 0.03 in frame 3, where projection says it should be visible, is almost certainly a face in frame 3 too. The working mix is 50% multi-view disagreement, 30% slice-targeted, 20% random, the random fifth kept so the label distribution stays estimable.

Synthetic augmentation

Labelled hard examples are scarce, so manufacture them, each technique with a specific way it can teach the wrong thing.

The pasting failure is instructive: if the model can detect the seam, it will, because the seam is an easier feature than a face. Diagnose by evaluating on real hard examples only; if recall is 0.97 on synthetic 11 px faces and 0.71 on real ones, the model learned compositing.

The train/test split

The split is the most consequential line in the lesson and it cannot be random. The same pedestrian appears in 4–6 consecutive frames of one capture run. Under an 80/10/10 split by tile, for a face with four sibling views:

P(a given sibling not in train) = 0.20
P(none of four in train)        = 0.20^4 = 0.0016
P(at least one sibling in train) = 99.84%

Essentially every test face was memorized during training at a slightly different size, pose, and exposure. Measured, a random tile split reads 0.9987 recall (1.3 misses/1,000) against a run-level split’s 0.9955 (4.5), the headline metric optimistic by 3.5×, with nothing in the training loop reporting an error. So the split is by capture_run_id, whole runs never spanning two sides, with two more cuts on top:

• By geography. Two runs down the same street a month apart share shopfronts, parked cars, and commuters. Hold out whole cities and report them separately: the only number estimating next year’s expansion markets.
• By time. Freeze the most recent six weeks. Rigs, firmware, and auto-exposure change; a random-in-time split averages a regression across the change instead of surfacing it.

The training recipe follows:

Metrics, and why mAP is the wrong headline

Three reasons mAP is misleading here

1. It averages over recall levels you never operate at. Average precision integrates precision across recall 0 to 1. You ship at recall 0.9955, so the only region you care about is the last half-percent, which AP weights as half a percent of the answer.
2. COCO-style mAP averages over IoU 0.5 to 0.95, pricing localization precision. Here a loose box is free (every box is dilated before blurring) while a tight box that clips an eye is a failure, so mAP penalizes exactly the behavior the product wants.
3. It is one number over all object sizes, and 62% of the risk lives in one band holding 19% of the objects.

Two real candidate models make it concrete:

Model B misses 4.5× fewer faces and loses on mAP by 7 points. A team gating on mAP ships A. This is what happens when one model is trained with a low box-loss weight and a small-object-heavy sampler: it localizes worse and finds more.

The precision nobody quotes is the other side of the story: the shipped point runs at precision 0.26. At t = 0.035 a panorama has (3.2 + 1.8) × 0.9955 = 4.98 true objects against 14 false positives, so 4.98 / 18.98 = 0.262. Roughly three of every four blurred regions contain nothing. That is not a deferred defect. It is priced into the area budget (those 14 boxes are the bulk of the 0.40% spend). Report the 0.26 alongside the recall; recall alone describes one side of the confusion matrix.

The metrics to actually report

The matching rule underneath every recall number is worth stating: IoU ≥ 0.35, matched greedily by descending score, one-to-one, in global panorama coordinates. “Greedy one-to-one” means: take the highest-scoring detection, pair it with the ground-truth box it overlaps most, remove both, repeat. No detection claims two faces, no face is credited to two detections.

The threshold choice matters because localization does not. A prediction s times the true width has IoU = s² if too small and 1/s² if too large, so:

IoU >= 0.50  ->  accepts 0.71× to 1.41× true width
IoU >= 0.35  ->  accepts 0.59× to 1.69× true width

The loose side is free (a 1.69× box just costs blur area, and dilation adds 15% anyway). The tight side is a real failure, and deliberately not IoU’s job. A 0.59× box covers about a third of the face and still passes at IoU 0.35. Catching that is what effective recall is for: it credits a face only when the blur covers the identity-bearing region (under-covering by ≤13% of width), and a 0.59× box fails that even after dilation (0.59 × 1.15 = 0.68, still 32% short).

IoU 0.35 asks “did you find it”; coverage asks “did you cover it”; only the first is allowed to be generous. Report recall at IoU 0.5 as a sanity column too. The gap prices what the low box-loss weight cost in localization (shipped: 0.9955 at IoU 0.35 vs 0.9908 at 0.50). A gap that grows release over release signals the box term has been de-weighted past the point a 15% dilation can cover.

The slice grid

A slice is a subset defined by an attribute; a marginal reports performance by one attribute at a time, averaging over the rest. Marginal recall by skin tone (using the Monk scale, a published ten-point skin-tone scale) shows a 1.24-point gap between the lightest and darkest bands (0.9971 vs 0.9847). That is the number a standard fairness table reports.

Cross the same data with illumination and the picture changes:

The gap is an interaction, not a main effect. A main effect is what one variable does alone; an interaction is what two do only in combination. In daylight the skin-tone gap rounds to nothing; at night it is twenty times that. The marginal averages 0.12 and 2.47 into 1.24, understating the worst cell by half and pointing at the wrong fix. It suggests a general problem with darker faces when the real problem is darker faces in low light.

The mechanism is physical. The camera auto-exposes for a scene dominated by sky and road. In low light a darker face lands in the bottom two stops of the sensor (a stop is a doubling of light, so this is the darkest quarter of the range), where its contrast against the background (the signal a detector uses) compresses toward the noise floor. This is a sensor-and-pipeline problem showing up as a model fairness metric. Four fixes, in order of return:

1. Gate on the worst cell, not the worst marginal. Free, and it makes the rest happen.
2. Exposure and gamma augmentation matched to the real capture distribution. Cheap, the largest single model-side gain.
3. Slice-targeted labeling of dusk and night captures, ~40k boxes, about $2,400.
4. Local tone mapping on tiles whose histogram shows crushed shadows, 3% of tiles, negligible.

After fixes 2–4 the night Monk 8–10 cell goes from 0.9661 to 0.9904 (gap 2.47 → 0.44 pts) and overall recall from 0.9955 to 0.9968.

There is no A/B test

An A/B test splits live traffic, gives one arm the new system and the other the old (the control or holdback), and compares. Here the A/B test is structurally forbidden: the control arm of “does blurring work” is published unblurred imagery, which is the harm itself. There is no holdback and never will be. Three things stand in:

1. Reweighting. The eval set is enriched with hard mines, so its unweighted mean reads a pessimistic 0.9931 (6.9 misses/1,000). Reweighting each slice cell by its frequency in real traffic gives 0.9955 (4.5). Publish both and the weights. An unweighted mean over a deliberately hard set is unactionable, and a reweighted one whose weights are hidden is uncheckable.
2. Shadow scoring, the closest thing to a counterfactual (what would have happened under the other choice). Run the candidate over the last 30 days of already-published captures (164M panoramas) and diff its boxes against the incumbent’s. The recall fix (0.9955 → 0.9968) yields 3.2 × 0.0013 × 164M ≈ 684,000 boxes of disagreement. You verify a stratified sample; because the diff is pre-filtered to disagreements, the yield per verified box is two orders of magnitude above a fresh random audit.
3. Staged rollout by capture run. Route 5% of runs to the candidate, keeping whole runs on one model so association never reconciles two detectors. The daily per-cell audit is the readout; the appeal rate lags by months and so cannot be the gate.

The gap none of the three closes is measurement position. The audit taps the pipeline before association, so it measures the detector, not the shipped system. The two differ by 74×. That is the right tap (the detector is what changes between releases), but it means the post-association number is derived through k_eff, which is re-measured only quarterly. A pose or depth regression would reach the appeal queue months before any offline metric. So alarm on the association fallback rate (the share of rescues finding an empty gate), which is a direct, daily, free signal that the geometry still works.

The pipeline

Everything above assembles into one batch job.

flowchart TD
    CAP[("Capture store<br/>2e9 panoramas · 36 PB")] --> SH["Shard by capture run<br/>keeps adjacent frames together"]
    SH --> DEC["Decode + undistort<br/>0.74 CPU-s each"]
    DEC --> TIL["Tile 1024² · overlap 192<br/>16 × 8 = 128 tiles"]
    TIL --> GATE{"Content gate<br/>0.6 GFLOP/tile · recall 0.9999"}
    GATE -->|"39% keep"| DET["Detector<br/>410 GFLOP/tile · anchor-free · P2-P6"]
    GATE -->|"61% skip"| NOBOX["No detections"]
    DET --> NMS["NMS in global panorama coords"]
    NMS --> MV["Multi-view association<br/>project across 4-6 frames · union"]
    MV --> DIL["Dilate boxes +15% width, +2 px floor"]
    DIL --> BLUR["Irreversible redaction<br/>decimate to block means + noise"]
    NOBOX --> PASS["Pass original bytes through<br/>NO re-encode"]
    BLUR --> ENC["Re-encode affected 2048² regions only"]
    ENC --> OUT[("Published tiles")]
    PASS --> OUT
    NMS --> AUD["Audit sampler<br/>stratified by slice cell"]
    AUD --> HR["Human recall audit"]

    style GATE fill:#bc6c25,color:#fff
    style DET fill:#1d3557,color:#fff
    style MV fill:#2d6a4f,color:#fff
    style BLUR fill:#9d0208,color:#fff

Four nodes carry more weight than their labels suggest, and three look like plumbing.

• SH: shard by capture run. The shard key is capture_run_id, a hard dependency of association, not a throughput tweak. Association is a join across the 4–6 frames of one pedestrian; shard by panorama id or a hash and those frames land on different workers, turning a local join into a global shuffle over 36 PB. The same key is also the unit of retry and of the train/test split.
• DEC: decode and undistort. Each sensor image is resampled onto the equirectangular sphere before tiling. Without it, angular resolution varies across the frame and the uniform 36.98 px/degree that every face-size, anchor-IoU, and motion-smear derivation rests on holds only near the optical axis.
• AUD taps before MV. The audit measures detector recall, not shipped post-association recall, 74× apart. That is the right tap because the detector is what changes, but it makes the launch gate a gate on the detector.
• ENC: whole 2048² regions. JPEG quantization is block-aligned, so a ragged re-encoded region leaves a visible rectangle. One 2048² region is 4.7% of an 88.6 MP panorama, and an affected one averages 3.2 of them, the “16% of panoramas re-encode ~15% of pixels” figure used in the cost estimate.

The content gate, and why it saves less than you think

Seventy-eight percent of tiles are sky, road, foliage, or upper-storey facade and hold no face or plate. A tiny classifier at one-sixteenth resolution skips those for 0.6 GFLOP against the detector’s 410, nearly 700× cheaper.

But the gate is now the first stage of a cascade (a chain where each stage passes only survivors to the next), and a cascade’s recall can never exceed its first stage’s. A face the gate drops is unrecoverable, so the gate must run tighter than the detector, not looser. At its natural point it skips 78% of tiles at recall 0.994, which would add 6 misses/1,000 on top of the detector’s 4.5. Pushed to 0.9999, the skip rate falls to 61%:

no gate:    128 × 410 GFLOP            = 52.5 TFLOP/panorama   (baseline)
naive gate: 78% skip                   = 11.6 TFLOP  ->  4.5× cheaper
safe gate:  61% skip (recall 0.9999)   = 20.5 TFLOP  ->  2.55× cheaper

The cascade is worth 2.55×, not 4.5×, and the difference is the price of the first stage’s recall. The gate takes the same tile downscaled 16×, emits one probability, gets its labels free (a tile is positive when a hand-drawn box falls in it), and is judged by its own held-out recall, measured separately, because a gate failure is invisible in the end-to-end number.

Tiling and boundary losses

The stride of a tiling is tile size minus overlap. Overlap must exceed the largest object that can be split, or a face on a seam becomes two partial faces:

tile 1024, overlap 192 → stride 832 → 16 columns × 8 rows = 128 tiles
largest in-scope face (pedestrian at 1 m) = 340 px > overlap 192, so it can split

The fix is not more overlap. Raising it to 340 px gives 200 tiles, 56% more compute. Instead: run NMS in global panorama coordinates so partial detections merge, dilate any edge-touching box to the union of its cross-boundary partners, and rely on large faces being easy (recall on >192 px faces is 0.9997 even when split, because half a large face is still a strong detection).

Box dilation

Because the box term was under-weighted, predicted boxes are systematically imprecise, and blurring exactly what the box says leaves eyes visible when the regressor under-shoots. Measure the under-coverage instead of guessing:

signed box-width error as a fraction of true width:
   p50 +0.02   p90 −0.06   p99 −0.12   p999 −0.19

dilate 15% + 2 px floor: covers p99 fully; area cost 1.15² = 1.32×
   t = 0.035 spends 0.40% raw → 0.53% dilated → over the 0.5% cap,
   so the threshold moves to t = 0.045

Effective recall is the fraction of faces whose blur covers the identity-bearing region (the eyes-nose-mouth triangle), not the fraction whose blur geometrically contains the whole annotated box. Operationally, a face is handled when the blur under-covers its true width by ≤13%. The 13% is anatomical: a face box runs hairline to chin, but the matchable features sit in the middle; lose 13% and what stays exposed is jaw, hairline, ear; lose 30% and you expose an eye. This is a labelling decision, of the kind every number here inherits, and until it is made, effective recall is not a number at all. (Same as the ≥2-character rule for plates: a two-character fragment plus make, model, colour, and street address is routinely enough to identify a vehicle.)

The trap is leaving the definition implicit. Read “handled” as full geometric containment (tolerance zero) and the same data gives effective recall ~0.59, not 0.9903, because the median error is only +0.02, so about four boxes in ten fail containment. When a metric’s headline number and its literal-reading number differ by 40 points, the definition is the deliverable and the number is a footnote.

Dilation buys so much because the residual after it is a heavy-tail phenomenon, not the bulk. Growing every box 15% moves the whole bulk (only a few hundredths of a width short) out of danger at once, and leaves behind a small, badly-localized population: the boxes that scraped through the IoU ≥ 0.35 match at 0.59× width, where 0.59 × 1.15 = 0.68 survives 32% short. Reaching them would need 0.867 / 0.59 = 1.47, a 47% dilation costing 2.2× the area, which the budget will not pay. That residual is bought with association or with lambda_box, not a bigger margin.

On the area side, the constraint that governs is the dilated area, because the dilated blur is what publishes. t = 0.037 is the tightest threshold that fits (1.3225 · area(t) = 0.5%); the lesson ships t = 0.045, deliberately conservative. It gives up about 0.0005 of detection recall to buy 0.09 points of area headroom, which is worth it against dense-urban panoramas whose blurred area runs far above the mean. But it is a choice: a design quoting 0.045 without knowing 0.037 would have fit has not finished the arithmetic.

Redaction must be irreversible

Finding faces was the whole story until now; destroying them is where the standard tool fails, because it merely attenuates, and attenuation is reversible.

A Gaussian blur is a convolution (each output pixel is a weighted average of its neighbours), and any convolution is described by what it does to each spatial frequency, the Fourier transform of its kernel, exp(-2π²σ²f²) for a Gaussian. That expression gets small at high frequencies but is never zero, so a frequency that was only attenuated can be divided back out. Wiener deconvolution does exactly that, succeeding wherever the attenuated signal still sits above sensor noise.

At 40 dB SNR (signal-to-noise ratio; 40 dB is a signal 100× the noise in amplitude) and σ = 2 px, setting the transfer function equal to 1/SNR gives a recoverable cutoff at a 4.1 px period. An eye on a 68 px face is ~14 px across, recoverable, and not marginally: the cutoff sits three times finer than the feature. A Gaussian blur at typical strength is a lossy filter, not a redaction.

The correct primitive destroys information instead of attenuating it:

1. Decimate the region to a grid of block means, many-to-one, no inverse.
2. Add zero-mean noise at the block level, random perturbation that averages to nothing but destroys exact values, and defeats averaging attacks across frames.
3. Re-encode so the DCT coefficients are quantized, the frequency decomposition JPEG is built on, which removes residual structure.

But “8×8” must mean a block size, not a fixed grid, or the primitive fails on exactly the small faces the design is built to catch. A fixed 8×8 grid keeps 64 numbers whatever the box, so its strength depends on width:

face  8 px:  64 means from   64 px →   1.0:1   (the identity function — nothing destroyed)
face 11 px:  64 means from  121 px →   1.9:1
face 16 px:  64 means from  256 px →   4.0:1
face 68 px:  64 means from 4,624 px →  72.3:1

The smallest in-scope face is redacted by doing nothing at all, and the 8–16 px band that holds 62% of misses gets only a 2–4:1 reduction, about a 1.4 px box filter, which deconvolves as well as the Gaussian just rejected. The frequency argument does not rescue it either: on an 8×8 grid the finest surviving detail is w/4, a quarter of the face at every size, so it is scale-invariant in frequency (a 68 px eye and an 11 px eye are both ~1.2× clear) and collapsing in sample count. The redaction is weakest exactly where the privacy risk is highest.

The fix is one line: fix the minimum block size and let the count follow.

n = clamp(1, 8, floor(w / 6))      w = dilated box width
   ratio = (w/n)², and w/n >= 6 by construction, so ratio >= 36:1 at every face size

A 6 px block floor guarantees at least 36:1 sample reduction at every face size, against 1.9:1 for the fixed grid at 11 px. The 8-block cap is the other end: without it a 340 px face becomes one flat rectangle that reads as a rendering bug and draws exactly the attention the blur exists to avoid. And the noise step matters more than it looks. The same face appears in 4–6 published frames, and averaging independent redactions reduces noise by √k, so the block noise must be correlated across frames of the same identity.

Scale and cost

Priced back-of-envelope, compute is not the constraint.

per panorama: 128 tiles · gate 0.077 TFLOP · detector (39% of tiles) 20.5 TFLOP

detector:  2e9 × 20.5 TFLOP / 300 TFLOP/s → 38,000 GPU-hr × $2.50  =  $95,000
decode:    2e9 × 0.74 CPU-s / 3,600 × $0.035                       =  $14,400
re-encode: only $880 — 84% of panoramas pass through untouched;
           re-encoding every pixel of every panorama would be $36,700 (42× more)
                                                                      --------
                                                                      ~$110,000 / year
                                                            = $55 per million panoramas

Three readings, the third being the point:

1. The detector is 86% of cost, decode 13%, re-encode 1%. And re-encode is 1% only because most panoramas pass through untouched.
2. The whole thing is ~$110k of compute for a year of planetary imagery, small enough that cost is not the interesting constraint.
3. The binding constraint is I/O throughput. There are 36 PB to read and 36 PB to write, and over a 30-day window that is 36 PB / (30 × 86,400 s) = 13.9 GB/s sustained, each way, for a month. That sizes the storage tier, the network, and the shard layout.

Failure modes

Demographic recall gap

Covered under the slice grid: a skin-tone × illumination interaction driven by auto-exposure, hidden by marginal tables, fixed by worst-cell gating, exposure augmentation, slice-targeted labeling, and tone mapping.

Reflections and printed faces

Some faces are present in the image but absent from the training distribution, and among them sits a policy question: what to do about faces not attached to a person.

audit of 400 hand-verified misses in commercial districts:
   reflected in a shop window          31%   <- a REAL person, must blur
   on a bus-shelter advertisement       8%   <- not real, blur anyway
   in a photograph in a window display  6%   <- not real, blur anyway
   on a passing vehicle's wrap          3%

recall on reflected faces  0.712      recall on all faces  0.9955

A reflection is a real, identifiable person and the detector is 64× worse at it (28.8% miss vs 0.45%): reflections are low-contrast, distorted, often mirrored, and superimposed on the interior scene, so they match almost none of the training distribution. The fix is targeted data: mine glass surfaces with a material classifier, label the reflections, and oversample. Printed faces go the other way and get blurred too. They cost only blur area, and the alternative (an “is this a real person” classifier in series) is a second model with its own recall that can only lower the system’s.

Motion blur is physics, not modelling

Motion smear is what happens when the subject crosses the sensor while the shutter is open: exposure time × relative speed gives the distance smeared, converted to pixels at 37 px/degree.

vehicle at 40 km/h = 11.1 m/s, a face at 5 m:
   daylight 1/500 s → 9.3 px smear = 14% of a 68 px face   (tolerable)
   dusk     1/125 s →  37 px smear = 55% of a 68 px face   (destroyed)

Part of the dusk recall gap is an optics problem, and no amount of training data fixes a face the sensor did not resolve. The fix is capture-side: cap exposure at 1/500 s and accept sensor noise (which a detector tolerates far better than smear), plus a speed policy for evening captures. The strongest answer to a model failure is sometimes a change to the thing producing the data.

Plate-specific failures

Plates share a detector with faces but fail in their own ways. (“Retroreflective blowout” is the plate’s reflective coating throwing light straight back and saturating the sensor to white, erasing the characters and the plate’s edges together.)

Summary

Human review and the appeal path

Three loops stay in the loop after launch (one measures the miss rate, one repairs individual misses, one turns both into training data), and each is sized from a different number.

1. The audit loop (proactive). A daily sample, stratified by slice cell, checked for missed faces. Size it from statistical power, the sample needed to detect a difference you care about:

goal: detect a 1.5% worst-cell miss rate against a 0.5% target, 80% power
   n ≈ 16 · p̄(1-p̄) / delta² = 16 × 0.01 × 0.99 / 0.01² = 1,584 faces per cell
   at 3.2 faces/panorama, for a cell that is 4% of traffic:
      1,584 / (3.2 × 0.04) = 12,375 panoramas per cell per readout

At 20,000 audited panoramas/day, the two or three riskiest cells read out daily and the rest weekly. The saving is in the sampling, not the scanning unit: draw tiles with a known, cell-dependent probability and reweight, so you stop spending 96% of a whole-panorama sweep on faces outside the cell you are measuring.

2. The appeal loop (reactive). Anyone can report an unblurred face. Size it from the shipped (post-association) miss rate, not the detector-only one:

suburban post-association miss: 1 - (1 - 6.1e-5)^3.2 = 0.0195%
   × 2e9 panoramas = 390,000 with a miss → × 0.008 reported = 3,100/yr
   = 8.6 reports/day × 90 s each ≈ 13 minutes/day

Missed faces do not staff this queue, because association already removed 98.6% of them. Run the same arithmetic on the detector-only 1.43% and you get 627 reports/day and two full-time reviewers: the association work pays for itself a second time, in headcount. What the queue actually holds is policy traffic (standing requests to blur a building or vehicle, over-blur complaints) whose volume comes from product policy, not the miss rate. The SLA is same-day, with the blur applied before verification, because the same cost asymmetry that shaped the threshold applies here: over-blurring wrongly costs a patch of wall, waiting costs another day of exposure.

3. The label loop. Every confirmed appeal is a geolocated, timestamped hard positive. Feed them into the next training set, but never the eval set, because appeals come only from populated, connected, rights-aware regions, and gating on them would optimize for the places that already work.

flowchart LR
    PUB[("Published imagery")] --> REP["User report<br/>~9/day from misses<br/>+ policy requests"]
    PUB --> AUD["Stratified audit<br/>20k panoramas/day<br/>weighted by slice risk"]
    REP --> Q[["Review queue<br/>90 s each"]]
    Q -->|confirmed| FIX["Blur applied same day"]
    Q -->|confirmed| HN[("Hard-positive pool")]
    AUD --> SLICE["Per-cell miss rate<br/>→ the launch gate"]
    AUD --> HN
    HN --> TRAIN["Next training set"]
    SLICE --> TRAIN
    HN -.->|"NEVER"| EVAL[("Eval set")]

    style Q fill:#bc6c25,color:#fff
    style SLICE fill:#2d6a4f,color:#fff
    style EVAL fill:#9d0208,color:#fff

Alternatives considered and rejected

Each rejection is quantitative, not a matter of preference.

Conclusion

The load-bearing ideas, in the order they compound:

• Derive the threshold from costs, then repair the cost model. The textbook ratio gives t = 1.09e-4 and blurs 43% of the image because it treats over-blur cost as linear per box. It is convex in total area, and the real optimum (t = 0.005, 3.2% area) is still unacceptable, so the shipped threshold is a constrained one: max recall subject to a 0.5% blur cap.
• Per-face recall is the wrong unit. Risk compounds within an image, so 99.55% per-face recall is a 16.5% miss on a dense panorama. The needed recall cannot come from the detector.
• Multi-view association is the highest-leverage element, and it is geometry, not modelling. Projecting detections across the 4–6 frames each pedestrian appears in (using pose and depth the rig already computes) takes the dense-panorama miss rate from 16.5% to 0.24% at zero inference cost. It is bought by a single shard key.
• Small faces are hard because of a training-time rule, not a data shortage. Standard anchors assign an 11 px face as background; anchor-free with center sampling removes the failure, and a stride-4 pyramid level (with a cheap head) makes the face resolvable.
• mAP is the wrong headline. Gate on misses per 1,000 at the shipped threshold, per cell of a size × lighting × skin-tone grid, and on effective (coverage) recall, because a detected-but-partially-blurred face is still a published face.
• Redaction must destroy, not attenuate. Gaussian blur is invertible; decimation to a block-size floor with cross-frame-correlated noise is not.
• At planet scale, compute is cheap (~$110k/year) and I/O is the constraint, 14 GB/s sustained each way for a month.

Almost every threshold and metric here is downstream of one policy decision: what counts as an identifiable face. Until that is written down, “recall,” “effective recall,” and “in scope” are all undefined.

Further reading

• Frome et al., Large-scale Privacy Protection in Google Street View, ICCV 2009: the real-world system this lesson mirrors, including face and plate blurring at scale.
• Lin et al., Focal Loss for Dense Object Detection (RetinaNet), ICCV 2017: the imbalance argument behind the classification loss.
• Tian et al., FCOS: Fully Convolutional One-Stage Object Detection, ICCV 2019: the anchor-free detector and center sampling.
• Zhang et al., Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection (ATSS), CVPR 2020: the adaptive-assignment fix.
• Lin et al., Feature Pyramid Networks for Object Detection, CVPR 2017: the pyramid the P2 argument rests on.
• Rezatofighi et al., Generalized Intersection over Union, CVPR 2019: the GIoU box loss.
• Carion et al., End-to-End Object Detection with Transformers (DETR), ECCV 2020: the query-budget model rejected here.

Next: the video-search lesson, where the unit of retrieval stops being an object and becomes an interval, and the index has to represent time.