03 — Street View Blurring

“We capture street-level imagery worldwide. Detect and blur every face and license plate before publication.”

The system finds every human face and every readable vehicle licence plate in a planet’s worth of street-level photography, and destroys that information before the imagery is published.

Four ideas carry the chapter:

• How to derive an operating threshold from costs instead of guessing one.
• Why that derivation produces a degenerate answer the first time, and how to repair it.
• Why small objects are hard for a reason buried in a training-time rule nobody looks at.
• Why the number everyone reports for object detection is the wrong one here.

When you finish, you should be able to draw the pipeline, name each model in it, say what each model eats and produces, say how it is trained and how it is run, name the single number that proves it works, and defend every threshold with arithmetic.

The system in one line. A 13,312 × 6,656 pixel spherical photograph taken from a moving vehicle goes in; the same photograph goes out with every face and plate replaced by a block of destroyed pixels, and nothing else touched.

Three things make this problem different from a standard detection task, and naming them in the first two minutes is most of an interview round.

1. The error asymmetry is extreme and it is legal, not commercial. A missed face is a privacy violation with a regulatory tail; a false positive is a slightly over-blurred pixel region. Throughout this chapter, a false negative (FN) means a real face the system failed to blur, and a false positive (FP) means a region the system blurred that contained no face at all. The operating point is not chosen, it is derived — and the derivation has a twist most people miss.
2. It is offline. Nothing here has a latency budget, because weeks pass between capture and publication. The design is a batch pipeline over petabytes, and the currency is throughput and cost per image rather than milliseconds per request.
3. mAP is the wrong number to report. mAP is mean average precision, the standard summary score for object detectors: for each object class you sweep the detector’s confidence threshold from strict to permissive, record how precision falls as recall rises, average precision across that whole curve, and then average across classes. The right number here is instead a per-slice miss rate whose worst individual cell is the launch gate, and Metrics and why map is the wrong headline number derives why.

Two ideas from elsewhere in this repository get pushed past their usual form below; both are restated in plain words where they are used.

• Class imbalance — what happens when one label is thousands of times rarer than the other (Resampling and the calibration it breaks). Here it shows up as roughly 58,000 background locations for every face, and Objective says what to do about it.
• Choosing a threshold from a cost matrix — writing down what each kind of mistake costs, then solving for the decision boundary (Choosing a threshold from the cost matrix). The cost ratio threshold and its degenerate answer applies that rule and then watches it collapse.

0. The models in this system, and what each one is for

There are only two learned models in this entire system, which is itself the chapter’s first surprise — most of the recall is bought by geometry and by pipeline structure rather than by modelling.

Three further components are often mistaken for models and are not. Multi-view association (Multi view association is worth more than any detector improvement) is pure geometry: it projects a detection from one photograph into the next using the vehicle’s recorded position and a depth map, with nothing learned. Local tone mapping (The slice grid and why marginals hide the problem) is a fixed image-processing pass triggered by a histogram test. Redaction (Redaction must be irreversible) is arithmetic on pixel blocks. Recognising which parts of a system are learned and which are not is most of what makes this design cheap.

Where each runs. Everything here runs offline, in batch, over stored imagery — there is no online serving path at all, and that single fact removes latency, autoscaling, caching and model-warmup from the design and replaces them with throughput and storage bandwidth (Scale and cost).

The assumptions this design rests on. Five carry real weight. Each one, if it turned out false, would break a specific argument later in the chapter — so each is listed with what it breaks.

1. The annotation guideline defines what a face is. Framing argues that if “face” is undefined then “recall” is undefined. Every number in this chapter inherits that definition.
2. The cost of a spurious blur grows faster than linearly in total blurred area, measured as P ≈ 0.017 · (area%)^1.40 in The corrected cost model. If that curve were a straight line instead, the whole threshold argument reverses.
3. The capture rig already produces accurate vehicle pose and a depth map, because it needs them to stitch panoramas anyway. Multi view association is worth more than any detector improvement gets its entire benefit for free from that, and would be unaffordable otherwise.
4. Misses are correlated across views rather than independent, quantified as 1.80 effective independent views out of four. Assuming independence would overstate the benefit enormously.
5. Model-assisted labels are 9% incomplete in exactly the hard cases. This is the most fragile of the five. It is why evaluation uses a separate blind control, and why Annotation costs and the surprise in them calls that the most dangerous number in the chapter.

1. Framing

Six facts anchor every later decision: what the prediction is used for, what each kind of mistake costs, how much time you have, how much data there is, where labels come from, and — the one people skip — what counts as a face in the first place.

That last row is worth pressing on in the interview. If “face” is undefined, “recall” is undefined, and two teams can report 0.99 and 0.96 on the same model because one counts the back of a head as a face. Write the annotation guideline first; it is the specification of the ML objective.

2. Deriving the operating point, and why the naive derivation fails

The confidence threshold the detector actually ships with has to be derived three times before it comes out right: once by the textbook rule, which produces an answer so extreme it destroys the product; once with a repaired cost model, which produces a defensible optimum that the product still refuses; and once as a constrained optimization, which is the number that ships. Even that number cannot reach the real target — which is what forces the geometric trick in Multi view association is worth more than any detector improvement.

2.1 The cost-ratio threshold, and its degenerate answer

The starting point is the standard rule for turning costs into a decision boundary — and the useful part is watching it fail.

A detector emits a confidence score between 0 and 1 for each candidate box, and the threshold t is the score above which you act on the box — here, blur it. If a false positive costs C_fp and a false negative costs C_fn, the expected cost is minimized by acting whenever the probability of a real face exceeds C_fp / (C_fp + C_fn), which is the rule derived in Choosing a threshold from the cost matrix. The intuition is simply that if missing is a thousand times worse than over-blurring, you should act on anything with more than about a thousandth of a chance of being real.

Neither cost is a sticker price. Each is an expectation — an average over everything that might follow — down a chain of conditional events, where each link is a probability of the next thing happening given that the previous one did. Build each chain out explicitly and keep the conditionals visible, because every one of them is a place someone can argue with the estimate.

The two blocks below are multiplication chains — the indented lines are the factors, and the last line of each block multiplies them together. Two of the factors deserve a word before you meet them:

• P(it is a real, identifiable person) = 0.62 is below 1 because plenty of missed “faces” are posters, mannequins, statues and printed advertisements (Reflections and printed faces). Those carry no privacy cost when missed.
• P(noticed and reported | published unblurred) = 0.013 is small because almost nobody browses street imagery of their own street, let alone spots themselves in it.

C_fn -- one missed identifiable face
   P(it is a real, identifiable person)                    0.62
   P(noticed and reported | published unblurred)           0.013
   cost per report: takedown workflow $40
                    + amortized regulatory/legal exposure $960
                                                        = $1,000
   C_fn  =  0.62 × 0.013 × 1,000                        =  $8.06

C_fp -- one spurious blurred box
   P(visible enough to matter)                            0.004
   P(complaint | visible)                                 0.02
   cost per complaint                                     $11
   C_fp  =  0.004 × 0.02 × 11                          =  $0.00088

t*  =  0.00088 / (0.00088 + 8.06)  =  1.09e-4

So one missed face carries $8.06 of expected cost and one spurious blurred box carries $0.00088 — four orders of magnitude apart, exactly as the framing table claimed — and the rule returns a threshold of 1.09e-4, which is one ten-thousandth.

Now look at what a threshold of 0.0001 does to a dense single-stage detector. A dense detector scores tens of thousands of candidate positions per tile, so almost all of those scores sit just above zero rather than at zero; a threshold of 0.0001 therefore accepts essentially every candidate the classification head does not actively suppress. Measured on real panoramas, t = 1e-4 emits about 4,100 boxes per panorama and blurs 43% of the image area. The product is destroyed.

The formula is not wrong; the input is. C_fp was modelled as a constant per box, and it is not. The cost of over-blurring is convex in total blurred area — convex meaning it curves upward, so each additional unit of blur hurts more than the one before — because blur goes from invisible to visible to annoying to disqualifying as it accumulates. Two hundred small boxes scattered across an image are not two hundred times the harm of one.

2.2 The corrected cost model

Rather than assuming that upward curve, measure it — turn “over-blurring gets worse faster than linearly” into a fitted equation with an exponent.

Sample panoramas across a range of blur coverage and measure the rate at which human reviewers call the image unusable:

blurred area      P(panorama rated unusable)
   0.1 %                0.001
   0.5 %                0.004
   1.0 %                0.012
   2.0 %                0.045
   5.0 %                0.19
  10.0 %                0.48

   -> least squares in log space:
      P ≈ 0.017 · (area%)^1.40        superlinear, as expected

Fitting by least squares in log space means taking logarithms of both columns before fitting a straight line, which is the standard way to recover a power law: a straight line in log-log coordinates is exactly y = a · x^b, and the slope of that line is the exponent. Here it gives P ≈ 0.017 · (area%)^1.40, so the probability a panorama is rated unusable rises as blurred area to the power 1.40 — superlinear, which is the curvature the previous subsection assumed and this one now has evidence for.

At an internal value of $0.15 per unusable panorama (the amortized cost of recapturing it plus its product value), the total expected cost per panorama, with n = 3.2 faces in an average one, is

Cost(t)  =  miss_rate(t) · n · C_fn   +   0.15 · 0.017 · area(t)^1.40
           \___________________/         \_________________________/
            expected privacy cost         $0.15 × P(panorama unusable)

The table below evaluates that expression at six thresholds. Recall, false-positive box count and blurred area are all measured on real panoramas at each threshold; the two cost columns are computed from them. Read down the total column and find its minimum.

Check one row by hand so the rest are believable. At t = 0.05 the miss rate is 1 - 0.9940 = 0.0060, so the miss cost is 0.0060 × 3.2 faces × $8.06 = $0.155, and the blur cost is 0.15 × 0.017 × 0.28^1.40 = $0.0004.

The unconstrained cost optimum sits at t = 0.005, which blurs 3.2% of every image. That is the honest answer to the arithmetic, and it is also an answer no imagery product will accept — 3.2% coverage means visible grey patches on most facades.

2.3 So the shipped threshold comes from a constraint, not the optimum

Now the real answer appears, and its lesson generalizes far past blurring: when the cost-minimizing point is unacceptable, the shipped point comes from a constraint, and being able to name which constraint binds is the whole answer.

State the formulation explicitly, because this is what separates a derived answer from a guessed one. “Maximize recall subject to a cap on blurred area” means: among all thresholds that keep the blurred fraction of the image under the cap, take the one that catches the most faces.

maximize    recall(t)
subject to  blurred_area(t)  <=  0.5 %          (product constraint)

-> t = 0.035,  recall 0.9955,  4.5 misses per 1,000 faces,  14 FP boxes
   and that threshold SPENDS 0.40 % of the image, not 0.50 %

The binding constraint is the product’s blur budget, not the legal cost — and naming which constraint binds is the answer.

Keep the cap and the spend separate

Naming the constraint is only half the job, because a constraint has two numbers attached and they are easy to run together.

• The cap is 0.5%. That is a product decision: the most blur the imagery can carry and still be worth publishing.
• The spend is what the shipped threshold actually blurs. That is a measurement, and it is smaller.

Getting the spend requires reading The corrected cost model’s measured area column at t = 0.035, which is not one of its rows — two rows bracket it. Since that column is a power law, interpolate in log space, not with a ruler:

two measured rows bracket t = 0.035:
   t = 0.02   ->   0.71 %
   t = 0.05   ->   0.28 %

fit a power law through them (a straight line in log-log coordinates):
   exponent  b = ln(0.28 / 0.71) / ln(0.05 / 0.02)
               = -0.9304 / 0.9163   =  -1.015
   scale     a = 0.71 · 0.02^1.015  =   0.0134

   area(0.035) = 0.0134 · 0.035^-1.015  =  0.402 %      <- the spend

the WRONG rule -- a straight line in t through the same two rows:
   0.71 + (0.035 - 0.02)/(0.05 - 0.02) · (0.28 - 0.71)  =  0.495 %

So area(0.035) = 0.40%, with a power law fitted to all six rows of §2.2 agreeing at 0.396%.

The 0.50% figure is what the straight-line rule produces, and a straight line is simply the wrong rule for a convex column: a chord drawn between two points on a curve that bends upward always sits above the curve in between. Fitting a power law and then reading it off with a ruler is a specific, common, and entirely silent error — silent because 0.495% and the 0.5% cap agree to two figures, so the mistake looks like the constraint binding exactly.

Where the cap actually binds

So the cap does not bind exactly at the shipped threshold, and it is worth knowing where it does bind:

cap applied to the RAW box area:       area(t) = 0.50 %  ->  t = 0.028
cap applied to the DILATED area:  1.32 · area(t) = 0.50 %  ->  t = 0.037
shipped:                                t = 0.035  ->  0.40 % raw, 0.53 % dilated

The second line is the one that governs, because the blur that gets published is the dilated blur (Box dilation derived from the localization error) — checking a budget against the undilated boxes measures a quantity no user ever sees. Read that way, t = 0.035 is not slack at 0.40%; it is a threshold chosen before dilation was priced, and it is 6% over budget once dilation is applied. That is exactly the discrepancy Box dilation derived from the localization error discovers and repairs by moving the threshold to t = 0.045. The 0.10 points between the spend and the cap are not headroom the design is failing to use — they are the down-payment on a margin §7 has not spent yet.

What the area budget is worth in money

None of the cap-versus-spend bookkeeping moves the money. Every cost figure below is driven by the miss rate, and the miss rate is a function of the threshold, not of the area that threshold happens to spend.

The cost model still earns its place, though: it prices the budget. That is the argument to take to whoever owns the 0.5% number.

under a 0.5 % cap:  t = 0.035, 4.5 misses / 1,000
   miss cost = 0.0045 × 3.2 faces × $8.06  =  $0.116 per panorama

under a 1.0 % cap:  at least the t = 0.02 row (0.71 % area, 2.5 misses)
   miss cost = 0.0025 × 3.2 faces × $8.06  =  $0.064 per panorama

   difference  =  $0.052 per panorama
   × 2e9 panoramas  =  $103M of expected privacy cost per year

That $103M is a floor, not an estimate: the true optimum under a 1.0% cap sits somewhere below t = 0.02, so the real saving is larger. This is the reason the conversation is about the area budget at all — the budget is not a taste question, it has a price tag.

The diagram below is this whole section in six boxes: two measured costs at the top, one dead end in the middle, and one shipped answer at the bottom. Follow the arrows down the left-hand branch.

flowchart TD
    A["C_fn = $8.06 per missed face<br/>C_fp = $0.00088 per spurious box"] --> B["t* = C_fp / (C_fp + C_fn)<br/>= 1.09e-4"]
    B --> C{"What does that<br/>threshold actually do?"}
    C -->|"4,100 boxes/panorama<br/>43% of the image blurred"| D["DEGENERATE<br/>C_fp is not linear in box count"]
    D --> E["Re-model: blur cost is<br/>convex in TOTAL AREA<br/>P_unusable = 0.017 · area^1.4"]
    E --> F["Unconstrained optimum<br/>t = 0.005 · 3.2% area"]
    F --> G{"Acceptable to the product?"}
    G -->|no| H["Constrained optimum<br/>max recall s.t. area <= 0.5% cap<br/>t = 0.035 · recall 0.9955<br/>spends 0.40% raw · 0.53% dilated"]
    G -->|yes| F

    style B fill:#1d3557,color:#fff
    style D fill:#9d0208,color:#fff
    style E fill:#bc6c25,color:#fff
    style H fill:#2d6a4f,color:#fff

Walking the diagram box by box:

1. Start from the two measured costs and apply the ratio rule. Out comes t* = 1.09e-4.
2. Ask the question that saves the design: what does that threshold actually do? 4,100 boxes per panorama, 43% of the image blurred. That branch is labelled DEGENERATE, and the diagnosis beside it names the faulty input — C_fp is not linear in box count.
3. Re-model. Replace the per-box constant with the convex area law P_unusable = 0.017 · area^1.4.
4. The re-modelled cost function has an unconstrained optimum at t = 0.005 and 3.2% area.
5. Ask the plain question: acceptable to the product? It is not.
6. Land on the constrained optimum — maximize recall subject to area at most the 0.5% cap — giving t = 0.035, recall 0.9955, at a measured spend of 0.40%.

No path in that diagram ends at a guessed number. Every terminal box is either derived or rejected for a stated reason, and the final box carries the spend as well as the cap, so the one number nobody checks is checkable.

2.4 The number that reframes the problem: per-face recall is the wrong unit

One line of arithmetic — checkable by anyone — changes what the whole design is optimizing.

A per-face recall of 99.55% sounds excellent — recall being the fraction of the faces that are really there which the system actually finds. But a user does not experience one face; they experience one photograph, which may contain many. So compound the per-face miss rate over an image:

P(a panorama has at least one miss)  =  1 - (1 - m)^n

   m = per-face miss rate,  n = faces in the panorama

suburban, n = 3.2 faces,  m = 0.0045
   (1 - 0.0045)^3.2  =  0.9955^3.2  =  0.98567     P(all 3.2 caught)
   1 - 0.98567                      =   1.43 %
   × 2e9 panoramas                  =  28.6 M published with an unblurred face

dense urban, n = 40 faces, same m
   0.9955^40  =  0.835
   1 - 0.835  =  16.5 %

(n = 3.2 is an average, so the exponent comes out fractional. That is the usual shorthand and it is close enough at these rates.)

One in six dense urban panoramas contains an unblurred face at 99.55% per-face recall. That is the sentence that reorganizes the design.

Now run the same formula backwards. To hold a dense panorama to a 1% chance of any miss:

require  (1 - m)^40  =  0.99

   1 - m  =  0.99^(1/40)  =  0.99975      <- the per-face recall you need
       m  =  2.51e-4                      <- 18x below the shipped 0.0045

That is not reachable by threshold tuning inside the area budget — The corrected cost model’s table runs out of recall long before it gets there, and buying more costs area the cap will not pay for. So the recall you need cannot come from the detector; it has to come from redundancy.

2.5 Multi-view association is worth more than any detector improvement

The highest-leverage idea in the chapter is not a modelling idea at all: the same person is photographed several times, so a face missed in one photograph can be recovered from another using geometry the capture rig already computes.

The capture vehicle triggers a panorama every 8 metres of travel, so a pedestrian it drives past appears in 4 to 6 consecutive frames, 0.72 seconds apart at 40 km/h. A detection in any of those frames can be projected into the others using the vehicle’s recorded position and orientation together with a depth map, and the union of all of them blurred.

What the projection needs, and what it produces

The projection needs exactly three inputs, and the reason this idea is nearly free is that the rig already computes all three in order to stitch the panoramas in the first place.

Work one pedestrian through it. A face is detected with confidence score 0.61 in frame 2, at a bearing of 30 degrees off the direction of travel, 12 metres away. Three steps: turn the 2D detection into a 3D point (back-projection), move that point by however far the vehicle drove, then turn it back into a bearing and a pixel width for the next frame. Watch how far the bearing moves.

frame 2:  bearing 30.0 deg,  depth 12.0 m
   angular width  =  9.17 / 12      =  0.764 deg     (§4.1's face-width rule)
   pixel width    =  0.764 × 36.98  =  28.2 px       (§4.1's px-per-degree)

  back-project into vehicle coordinates at frame 2
     forward  z  =  12 · cos 30  =  10.392 m
     lateral  x  =  12 · sin 30  =   6.000 m

  apply the pose delta to frame 3 (8 m of forward translation)
     z' = 10.392 - 8  =  2.392 m        x' = 6.000 m
     range   =  sqrt(2.392^2 + 6.000^2)  =  6.459 m
     bearing =  atan2(6.000, 2.392)      = 68.26 deg

frame 3:  bearing 68.26 deg,  range 6.459 m
   angular width  =  9.17 / 6.459   =  1.420 deg
   pixel width    =  1.420 × 36.98  =  52.5 px

The bearing went from 30.0 to 68.26 degrees, a swing of 38.26 degrees, which at 36.98 px per degree is 1,415 pixels. The face went from 28.2 px wide to 52.5 px.

The face moved 38.3 degrees — 1,415 pixels — and nearly doubled in size between two consecutive frames. Say that out loud, because it is why this is not object tracking. Tracking algorithms work by assuming things move a little between frames and then matching boxes that overlap; the standard measure of that overlap is IoU, intersection over union: the area two boxes share divided by the total area they cover, which is 1 for identical boxes and 0 for boxes that do not touch. Here the two boxes do not touch at all, so their IoU is exactly zero and no overlap-based matcher could ever pair them. Association happens in world coordinates or it does not happen.

The projection is depth-sensitive, and that is what decides the algorithm

The projection works; the next question is how much to trust it — and the answer decides whether the geometry is used as an answer or merely as a hint.

Perturb only the assumed depth, leaving everything else identical, and re-run the same three lines. The last column is the one to read: it converts the bearing error into pixels at 36.98 px per degree, so at 11.0 m the bearing comes out 74.49° against a true 68.26°, and (74.49 - 68.26) × 36.98 = +230 px.

A 1 m depth error — 8% — displaces the reprojected box by up to 230 px, four times the width of the 52 px face it is meant to cover.

To blur the reprojected box safely you would have to grow it until it covers that displacement in both directions: 52.5 + 2 × 230 = 513 px a side. A 513 px grey square is not a redaction, it is a hole in the image. So the reprojection is used as a gate and not as an answer:

1. Gate. Take every detection in frame 3, all the way down to a confidence of 0.002, back-project each into 3D, and keep the ones lying within R = 1.5 m of the frame-2 detection’s 3D point. That radius is a physical budget rather than a tuned constant, and it is the sum of two things: a pedestrian walking at 1.4 m/s covers 1.4 × 0.72 = 1.01 m in the 0.72 s between triggers, and two standard deviations of depth noise is 2 × 0.25 = 0.50 m. (σ, sigma, is the standard deviation; 2σ covers about 95% of a normally distributed error.) 1.01 + 0.50 = 1.51, rounded to 1.5 m.
2. Promote. If a below-threshold detection falls inside the gate, blur its box rather than the reprojected one. This is the common path and it costs no extra blurred area at all, because the box is a real detection with real localization; the geometry only supplied the permission to accept it at a confidence of 0.03.
3. Fall back. Only when the gate comes back empty does the reprojected region itself get blurred — a quadrilateral, since projecting a rectangle from one viewpoint into another does not leave it rectangular — dilated by the depth uncertainty derived above.

The diagram traces those three steps for a single detection. The decision diamond in the middle is the gate; the two branches out of it are the common cheap path and the rare expensive one, with their measured shares on the arrows.

flowchart LR
    D2["Frame 2 detection<br/>score 0.61 · 30 deg · 12 m"] --> BP["Back-project<br/>bearing + depth -> 3D point"]
    BP --> POSE["Apply pose delta<br/>8 m forward translation"]
    POSE --> G{"Any frame-3 detection<br/>within 1.5 m in 3D?"}
    G -->|"yes · 82%"| PROM["PROMOTE that box<br/>blur it · zero extra area"]
    G -->|"no · 18%"| FB["Blur the reprojected quad<br/>dilated by depth sigma<br/>0.00076% of image area"]
    PROM --> U["Union into the blur set"]
    FB --> U

    style G fill:#bc6c25,color:#fff
    style PROM fill:#2d6a4f,color:#fff
    style FB fill:#1d3557,color:#fff

Read the diagram left to right as the life of one rescue. It begins with the frame 2 detection score 0.61, 30 deg off the direction of travel and 12 m away — the worked example above. That detection is back projected: bearing plus depth gives a 3D point. The pose delta is then applied, which for consecutive triggers is 8 m of forward translation. The decision box asks whether a detection within 1.5 m in 3D of that point exists in frame 3. In 82% of cases one does, and the system takes the promote path, blurring that real box at zero extra area. In the remaining 18% the gate is empty, and the system falls back to blurring the reprojected quadrilateral dilated by the depth uncertainty, which costs 0.00076% of image area. Either way the result is unioned into the blur set for that panorama.

A 513 px grey square sounds ruinous. It is not, purely because of how rarely it fires — and the arithmetic below is the whole argument, so work through it rather than skimming it.

Start from how many faces association rescues at all. The detector alone misses 0.0045 of faces; after association the miss rate is 6.1e-5 (derived two parts below in what it is worth). The difference is what association saved, and only 18% of those saves need the expensive path.

faces rescued per panorama
   3.2 faces × (0.0045 - 6.1e-5)  =  3.2 × 0.004439  =  0.0142

of those, 18 % find an empty gate (measured on the same 5,000 pedestrians)
   0.0142 × 0.18  =  0.0026 fallback boxes per panorama

area they cost, at ~513 px a side
   0.0026 × 513^2 px  /  88.6 MP   =  684 px / 8.86e7 px
                                   =  7.6e-6  =  0.00076 % of image area
                                   =  1/650th of the 0.5 % budget (§2.3)

The expensive path is affordable precisely because it only ever fires on the 0.45% of faces the detector already missed. That is the shape of the whole design: buy recall wherever the volume is small enough that the area cost rounds to nothing.

Two things break it, and both are worth volunteering. Mis-association in a crowd is benign — a 1.5 m gate in a dense scene can grab the person standing next to the intended one, and the consequence is that a face gets blurred that needed blurring anyway, so the gate is loose on purpose. Depth failure on glass is not benign: lidar returns from the interior behind a shop window, so a reflected face (Reflections and printed faces) is placed metres too deep and its gate lands on empty pavement. Association therefore does not rescue reflections at all, which is why Reflections and printed faces’s 0.712 is already the shipped number and not a pre-association one.

Propagation has to be thresholded too

The geometry does not know which of its inputs are real, so one more rule is needed to stop the trick from multiplying the system’s mistakes along with its rescues.

Projection is symmetric: a false positive propagates into neighbouring frames exactly as well as a true detection does.

Union every detection above the shipped threshold into its neighbours, and each panorama now receives its neighbours’ false positives on top of its own. A face is visible across roughly four consecutive frames, so a panorama has about 3 neighbours to receive from, and at t = 0.035 each of them carries 14 false-positive boxes (So the shipped threshold comes from a constraint not the optimum):

propagate everything at t = 0.035   ->  14 + 3 × 14   =  56 FP boxes   (4.0x)
propagate only at score >= 0.50     ->  14 + 3 × 0.4  =  15.2         (1.09x)

At t = 0.50 the detector emits only 0.4 false-positive boxes per panorama (The corrected cost model), so gating propagation at that score adds 8.6% to the false-positive load a panorama already carries instead of quadrupling it. Detections scoring between 0.035 and 0.50 are still blurred where they were found; they are simply not allowed to paint boxes into frames that never saw them. The k_eff below is measured under this rule, which is part of why it comes out at 1.80 rather than nearer 4.

What it is worth

Finally, quantify the benefit — carefully, because the naive quantification is wildly wrong in the optimistic direction.

If per-frame misses were statistically independent across k views, then the chance of missing a face in all of them would be the single-view miss rate raised to the power k, written m_eff = m^k, and the problem would simply vanish. They are not independent: a face that is 9 pixels wide, backlit, and half-occluded tends to be all three of those things in every frame, so the same views fail together. Model that correlation with an effective independent-view count k_eff — the number of genuinely independent looks your k real looks are worth — estimated from the measured joint miss rate rather than assumed:

if the 4 views were independent:
   m^4  =  0.0045^4  =  4.1e-10        <- the fantasy number

measured on 5,000 tracked pedestrians across 4 views:
   P(missed in all 4 views)   =  6.1e-5     <- 150,000x worse than the fantasy
   single-view miss rate      =  0.0045

solve for the exponent that actually fits:
   m^k_eff  =  6.1e-5
   k_eff    =  ln(6.1e-5) / ln(0.0045)
            =  -9.7051 / -5.4037   =   1.80 effective views

so effective per-person miss rate  =  6.1e-5
   0.0045 / 6.1e-5                 =  74x better than one view
dense panorama, 40 people:  1 - (1 - 6.1e-5)^40  =  0.24 %

Two views’ worth of benefit out of four captured views is the honest result. The independence assumption would have promised roughly 150,000 times more than that, which is why it has to be measured rather than assumed.

Association across views takes dense-urban panorama miss probability from 16.5% to 0.24% and costs nothing in inference — the frames were already captured and already run through the detector. It requires pose and depth, which the capture rig already produces for stitching. This is the single highest-leverage design element in the chapter and it is not a modelling idea at all.

Every number in this section is reproduced by the code below, and the asserts at the bottom check them. Four functions, each tied to a figure you have already seen:

• panorama_miss_rate — the 1.43% and the 16.5% of The number that reframes the problem per face recall is the wrong unit.
• effective_views — the k_eff of 1.80.
• required_per_face_recall — the 0.99975 that The number that reframes the problem per face recall is the wrong unit says threshold tuning cannot reach.
• reproject — the worked example. Run it at depth 11 and 13 and you get the +230 px and −181 px rows of the sensitivity table.

The guard inside effective_views is not decoration. k_eff < 1 would mean the measured joint miss rate is worse than a single view — and a union of views cannot be worse than one of its members. So that condition is a signal that the tracking which produced the joint measurement dropped frames, not a signal that the views are useless. Raising there is what stops a broken input from being reported as a broken idea.

def panorama_miss_rate(per_face_miss, faces_per_panorama):
    """P(at least one unblurred face) -- the unit that actually matters.

    Per-face recall is the number people quote and the wrong one to gate on,
    because risk compounds within an image and the compounding is worst
    exactly where the density is highest.
    """
    return 1.0 - (1.0 - per_face_miss) ** faces_per_panorama


def effective_views(joint_miss_rate, single_miss_rate):
    """How many INDEPENDENT views your k captured views are worth.

    Misses correlate across frames -- a 9-pixel backlit face is hard in all
    of them -- so k_eff < k. Estimate it from the measured joint miss rate
    rather than assuming independence, which overstates the benefit wildly.
    """
    from math import log
    if not 0.0 < joint_miss_rate < 1.0 or not 0.0 < single_miss_rate < 1.0:
        raise ValueError("both miss rates must lie strictly inside (0, 1)")
    if joint_miss_rate > single_miss_rate:
        raise ValueError(
            f"joint miss {joint_miss_rate:g} exceeds single-view miss "
            f"{single_miss_rate:g}, so k_eff < 1 -- a union cannot be worse "
            "than one of its members, so the tracking is dropping frames")
    return log(joint_miss_rate) / log(single_miss_rate)


def required_per_face_recall(target_panorama_miss, faces_per_panorama):
    """Invert the compounding to get the per-face recall a target implies."""
    if faces_per_panorama <= 0:
        raise ValueError("faces_per_panorama must be positive")
    return (1.0 - target_panorama_miss) ** (1.0 / faces_per_panorama)


def reproject(bearing_deg, depth_m, baseline_m):
    """Where a detection at (bearing, depth) in frame i lands in frame i+1.

    This is the whole geometry of multi-view association: bearing plus depth
    gives a 3D point, the pose delta moves it, and the new bearing is where
    to look. Sweep `depth_m` to see why the result is a gate and not an
    answer -- the bearing swings faster than the face is wide.
    """
    from math import atan2, cos, degrees, radians, sin, sqrt
    z = depth_m * cos(radians(bearing_deg)) - baseline_m
    x = depth_m * sin(radians(bearing_deg))
    return sqrt(z * z + x * x), degrees(atan2(x, z))


PX_PER_DEG = 13312 / 360                       # §4.1

def close(a, b, tol=0.000001):
    return abs(a - b) < tol


assert close(panorama_miss_rate(0.0045, 40), 0.1650690), "the 16.5% of 2.4"
assert close(effective_views(6.1e-5, 0.0045), 1.795932), "the k_eff of 2.5"
assert close(required_per_face_recall(0.01, 40), 0.9997488), "the 0.99975"

_, true_bearing = reproject(30.0, 12.0, 8.0)
_, off_bearing = reproject(30.0, 11.0, 8.0)
assert close(true_bearing, 68.262, 0.001), "frame-3 bearing of the example"
assert close((off_bearing - true_bearing) * PX_PER_DEG, 230.3, 0.1), "the +230 px row"

try:                                           # joint worse than a single view
    effective_views(0.5, 0.0045)
except ValueError:
    pass
else:
    raise AssertionError("k_eff < 1 must be rejected, not returned")

What interviewers probe: whether you stop at “high recall, low precision.” The strong answer derives the threshold, notices it is degenerate, fixes the cost model, names the binding constraint, and then shows that the constraint cannot deliver the panorama-level target — which is what forces multi-view association into the design.

3. The ML objective and the architecture

With the operating point settled, the one detector in the system can be stated precisely: what it takes in, what it emits, what function training minimizes, which architecture family it belongs to and why the other three are rejected, and — the part most designs leave implicit — which information is deliberately withheld from it.

3.1 Objective

Before choosing an architecture, write down what the model is asked to produce and what training punishes it for, because two of the three choices below are deliberately asymmetric in the same direction the product is.

What goes in is one tile: a square crop of the panorama, 1024 pixels on a side. What comes out is a set of boxes, each with a class drawn from {face, plate} and a confidence score. Training minimizes a loss — a single number measuring how wrong the model currently is, which gradient descent pushes downward — built from two terms, one for “is there an object here” and one for “where exactly is it”:

L  =  L_cls  +  lambda_box · L_box

L_cls  =  focal loss, alpha 0.25, gamma 2.0
L_box  =  GIoU on positive assignments

L_cls is the classification term and L_box the box-regression term, with lambda_box setting how much the second one matters relative to the first.

The classification term uses focal loss rather than plain cross-entropy.

Cross-entropy is the standard classification loss: it charges you the negative logarithm of the probability you assigned to the correct answer, so being confidently wrong is expensive and being confidently right is nearly free. Focal loss multiplies that by a factor that shrinks toward zero as an example becomes easy. Two knobs govern it: gamma = 2.0 sets how aggressively easy examples are discounted, and alpha = 0.25 is a fixed re-weighting between the positive and negative classes. The derivation is in ml 07.

The reason it is needed here is arithmetic. A dense single-stage detector scores every cell of every pyramid level, which at 1024² is 87,296 locations (Why the feature pyramid level matters and what it costs counts them: 65,536 at P2 plus 21,760 across P3–P6). A tile holds roughly 1.5 real objects. That is an imbalance of about 58,000 background locations to 1 object.

Each individual easy negative contributes almost nothing to the loss. But 87,000 of them summed together swamp the handful of positives, and the model learns to say “background” everywhere.

The box term uses GIoU — generalized intersection over union — computed only on the locations assigned as positive. Plain IoU is useless as a loss when two boxes do not overlap, because it is flat at zero and therefore has no gradient pointing toward the right answer; GIoU adds a term based on the smallest box enclosing both, so it keeps pulling even when they are disjoint.

Set lambda_box low — around 0.3 rather than the usual 2.0 — because localization precision is nearly worthless here. A box that is 20% too large blurs slightly more wall; a box that is 5% too small leaves an eye visible. The loss should be asymmetric in the same direction the product is, and the cheapest way to get that is to under-weight the box term and then dilate every box at blur time (Box dilation derived from the localization error).

3.2 Architecture comparison

Four detector families could do this job, and three of them fail on grounds specific to this problem rather than on general reputation — one column of the table below is the one that actually decides.

A word on each family first, because the names are opaque:

• Two-stage (Faster R-CNN). First runs a region proposal network (RPN) that suggests where objects might be, then classifies each proposal. Accurate and expensive.
• Single-stage, anchor-based (RetinaNet). Skips the proposal step and classifies every location directly, comparing each location against a fixed set of template boxes called anchors.
• Single-stage, anchor-free (FCOS). Same as above but drops the templates entirely; each location predicts the distances to the box edges directly.
• DETR family (detection transformer). Emits a fixed number of object “queries” — slots that compete to claim objects — and needs no post-processing to remove duplicates.

Two more terms the table uses. NMS, non-maximum suppression, is the post-processing DETR avoids: when several nearby boxes describe the same object, keep the highest-scoring one and delete the rest. GFLOP means a billion floating-point operations, the unit of compute cost used throughout this chapter.

Anchor-free wins on the column that decides this problem: it removes the anchor-assignment rule that silently labels small faces as background. “Center sampling” is the anchor-free assignment rule that replaces it — a location counts as a positive example if it falls inside the object’s box, or inside a shrunken central region of it, with no overlap test involved at all. DETR is rejected on a specific rather than a general ground: its query budget is fixed at 100 slots, so a scene containing more objects than that cannot be represented no matter how good the model is, and crowded scenes are exactly where the panorama-level risk lives (The number that reframes the problem per face recall is the wrong unit).

3.3 What the model eats, and what is deliberately withheld

The feature-engineering question has an unusual answer in this system: there are no engineered features at all, and the interesting design work is in what is deliberately kept out of the model.

The detector’s input is one tile and nothing else: a 3 × 1024 × 1024 array of unsigned 8-bit integers — three colour channels, red, green and blue, each pixel stored as a whole number from 0 to 255 — cropped from the undistorted equirectangular panorama and normalized per channel so each channel has a comparable numeric range. No metadata is concatenated anywhere. There is no feature engineering in this system, and that is a real answer to the feature stage rather than a gap in it: the learned representation is the pyramid of feature maps in Why the feature pyramid level matters and what it costs, so the design work is choosing which resolutions exist, not which columns of a table to compute.

What is available and deliberately not fed in is the part worth defending. Every panorama carries the time of capture, its satellite position, the camera’s exposure and gain settings, and a region code. Any of those could be concatenated onto the model’s input.

Feed one in and the model acquires a prior — a background expectation formed before looking at the pixels. The prior is not even wrong: a dusk capture in a low-density suburb really does contain fewer faces on average, so a model that starts out more sceptical there will score better overall.

The problem is where that scepticism lands. It lowers scores in exactly the cells that The slice grid and why marginals hide the problem shows are already the worst-performing ones — low light, low density. The aggregate metric goes up while the specific population most at risk gets worse. A geolocation-and-time feature buys mAP, and it buys it out of the worst cell of the slice grid.

So the metadata is used, but only outside the model. Exposure and gain select the tone-mapping pre-pass (The slice grid and why marginals hide the problem); pose and depth drive multi-view association (Multi view association is worth more than any detector improvement). Neither ever reaches the part of the network that decides whether a face is present.

4. The small-object problem, derived

Small faces are hard, and the explanation is not “the model needs more data”. It is two specific mechanisms — a training-time assignment rule that labels small faces as background, and a resolution limit in the feature maps — each of which can be derived from first principles, and each of which has a different fix.

4.1 How small is a face, in pixels

Start by converting distance into pixels, because every argument in this section depends on knowing that a person at a normal street distance is about the size of a postage stamp on a screen.

The rig produces an equirectangular panorama of 13312 × 6656 pixels covering a full 360 degrees. Equirectangular means the sphere of directions around the camera is unrolled onto a rectangle with longitude across and latitude down, so that a fixed number of pixels corresponds to a fixed angle everywhere — which is the property the arithmetic below relies on:

angular resolution  =  13,312 px / 360 deg  =  37.0 px per degree

a human face is ~0.16 m wide, so half of it is 0.08 m;
at distance D the full angular width is
   2 · atan(0.08 / D)   radians-ish

for small angles atan(x) ≈ x, so that is 0.16/D radians, and
   0.16/D radians × 57.30 deg/radian  =  9.17 / D   degrees

then multiply degrees by 36.98 px/deg to get pixels:

D =  5 m  ->  9.17/5  = 1.83 deg  ->  1.83 × 36.98 =  68 px
D = 15 m  ->            0.61 deg  ->                  23 px
D = 30 m  ->            0.31 deg  ->                  11 px
D = 50 m  ->            0.18 deg  ->                   7 px   <- below any
                                                                detection floor

That 9.17 / D rule and that 36.98 px/deg constant are used a dozen more times in this chapter — in the anchor-IoU argument of Why anchors label small faces as background, the reprojection of Multi view association is worth more than any detector improvement, and the motion-smear calculation of Motion blur is physics not modelling.

Measured over real captures, the face-size distribution is:

  < 8 px    4.1 %      out of scope by policy (not identifiable)
 8-16 px   19.3 %      the hard band, and 62 % of all misses
16-32 px   34.8 %
32-64 px   27.1 %
  > 64 px  14.7 %

The 8-16 px row is the one to watch. Faces under 8 px are out of scope, so the in-scope denominator is 100% - 4.1% = 95.9%, and the 8-16 px band’s share of in-scope faces is 19.3 / 95.9 = 20.1%.

Twenty percent of in-scope faces sit in the 8-16 px band, and they produce 62% of the misses. Every architectural decision below is about that band.

4.2 Why anchors label small faces as background

The chapter’s sharpest mechanism is a rule applied silently during training that turns small faces into negative examples, so the model is actively taught that they are not there. More data cannot fix a rule that discards the data.

Recall from Architecture comparison that an anchor-based detector compares each candidate location against a fixed set of template boxes. The assignment rule decides which of those templates get labelled as containing an object during training. The standard rule has three bands:

• IoU above 0.5 with a hand-labelled ground-truth box → the anchor is a positive. Train the model to fire here.
• IoU between 0.4 and 0.5 → ignore. Contributes no gradient at all.
• IoU below 0.4 → negative. Train the model to say background here.

The standard anchor sizes are {32, 64, 128, 256, 512} pixels, spread across the levels of the feature pyramid (FPN, feature pyramid network — the stack of feature maps at decreasing resolutions described in Why the feature pyramid level matters and what it costs). Note the smallest: 32 px.

Now take a perfectly centered 11-pixel face against the smallest anchor available, 32 pixels. Since the small box sits entirely inside the large one, their intersection is the small box and their union is the large one, so the overlap is just the ratio of areas:

IoU  =  (11 × 11) / (32 × 32)  =  121 / 1,024  =  0.118

0.118 < 0.4, so the location is not even ignored — it is assigned as a negative, and the classification loss actively trains the model to output “background” at the exact location where a face is. The gradient does not merely fail to teach; it teaches the wrong thing.

Sweep the face size against that same 32 px anchor and the floor becomes visible. Each row is face² / 32², and the verdict column applies the three-band rule above:

IoU of a centered face against its best 32 px anchor
   face  8 px   ->   64/1024 = 0.063     negative
   face 11 px   ->  121/1024 = 0.118     negative
   face 16 px   ->  256/1024 = 0.250     negative
   face 23 px   ->  529/1024 = 0.517     POSITIVE
   face 32 px   -> 1024/1024 = 1.000     POSITIVE

Nothing between 8 px and 22 px ever becomes a positive example. Not one.

The assignment rule installs a hard detection floor at ~23 px, which is a face at 15 metres. Three fixes, and only the third is clean:

4.3 Why the feature-pyramid level matters, and what it costs

The second mechanism is independent of the first and survives fixing it: even with a perfect assignment rule, a face has to be visible in the internal representation that predicts it — and a small face is not, for a reason whose fix carries a price worth deriving.

A convolutional network does not look at pixels directly at prediction time. It looks at a feature map: a grid of learned descriptors, each summarizing a patch of the image. The stride is how many input pixels separate two neighbouring cells of that grid, so at stride 16 each feature cell stands for a 16 × 16 pixel patch. A feature pyramid is a set of such maps at several strides — here labelled P2 through P6, at strides 4, 8, 16, 32 and 64 — so that large objects can be found on the coarse maps and small ones on the fine maps. A face must be resolvable in whichever map is asked to predict it. At stride s, a face f pixels wide occupies f/s cells:

Below one cell, the face is a sub-cell signal: no location’s receptive field is dominated by it, so its evidence is averaged into whatever else is in that cell. A receptive field is the region of the original image that a single feature cell can see; if the face occupies less than one cell, then every cell that sees it also sees a great deal of pavement and wall, and the face’s contribution to that cell’s descriptor is diluted. An 11-pixel face at P4 is 0.69 cells across. It therefore needs P2, the stride-4 level — and P2 is where the cost is:

Price the head one cell at a time, since the head runs once per cell. Each factor in the product below is a separate design choice, so they are spelled out:

detection head: 4 convs of 256 channels, 3x3 kernels, two heads (cls + reg)

   per cell  =  2      heads (one classification, one box regression)
              × 4      conv layers in each head
              × 2      FLOPs per multiply-accumulate
              × 256    input channels
              × 256    output channels
              × 9      positions in a 3x3 kernel
             =  9.44 MFLOP per cell

P3-P6:  21,760 cells × 9.44 MFLOP   =  205 GFLOP
P2:     65,536 cells × 9.44 MFLOP   =  619 GFLOP        <- 3x everything else

P2 costs three times the other four levels combined for one reason only: it has three times as many cells, because halving the stride quadruples the grid.

Adding a stride-4 level makes the head, not the backbone, the dominant cost. The backbone is the main network that turns pixels into features and is shared by every level; the head is the small network run at every cell of every level to produce the classification score and the box coordinates, so its cost scales with the number of cells rather than with the image. The mitigation is to run a different, cheaper head at P2 — depthwise-separable convolutions, which factor an ordinary convolution into a per-channel spatial filter followed by a 1 × 1 mixing step and cost a fraction as much, with 128 channels instead of 256 and 2 layers instead of 4:

The separable head splits each conv into two cheap pieces, and the two terms in the bracket below are those two pieces:

per cell  =  2 heads × 2 convs × (  2·128·9      depthwise: one 3x3 filter
                                                 per channel, 128 channels
                                  + 2·128·128 )  pointwise: a 1x1 mixing
                                                 step across 128 channels
          =  0.14 MFLOP per cell

P2:  65,536 × 0.14 MFLOP  =  9.2 GFLOP        67x cheaper than the full head

That is defensible because P2 only has to find small, low-detail objects — the discriminative task there is genuinely easier than at P5, where a “face” can be a mannequin, a poster, a statue, or a reflection.

Adding the pieces gives the 410 GFLOP/tile figure quoted in Architecture comparison and used in every cost estimate in Scale and cost. The backbone number is scaled by area: at 224² a ResNet-50 costs 8.2 GFLOP, and (1024/224)² = 20.9, so 8.2 × 20.9 = 171 GFLOP.

backbone (ResNet-50 at 1024^2, scaled from 8.2 GFLOP at 224^2)  = 171 GFLOP
FPN laterals and output convs                                   =  25 GFLOP
heads P3-P6 (full)                                              = 205 GFLOP
head P2 (lightweight)                                           =   9 GFLOP
                                                                  ---------
                                                                  410 GFLOP / tile

recall on the 8-16 px band:   without P2  0.42    with P2  0.89

That last line is what the 9 GFLOP buys: more than doubling recall on the band that carries 62% of the misses.

5. Data and labels

Where the training data comes from hides the chapter’s most under-appreciated result: the labels themselves have a recall of their own, and if you do not measure it, the model’s reported recall is fiction.

5.1 Annotation costs, and the surprise in them

Price the labelling before designing it, because the dominant cost is not the one everybody assumes and the difference reorganizes the whole strategy.

Two activities cost money: drawing a box around a face you have found, and scanning the image to find the faces in the first place. Price both at the same $22/hour fully loaded annotator rate.

DRAWING
   one box, including a verification pass             7.0 s
   at $22/hour:  7.0 / 3600 × 22                      $0.043
   + 40 % for a 20 % QA sample plus adjudication      $0.060 per box

SCANNING
   exhaustively sweeping an 88.6 MP panorama at
   sufficient zoom to find an 11 px face              ~4.0 min
   at $22/hour:  4.0 / 60 × 22                        $1.47 per panorama

PER PANORAMA
   mean 3.2 faces + 1.8 plates = 5.0 boxes
   5.0 × $0.060                                       $0.30 of boxes
   + the scan                                         $1.47
                                                      -------------
                                          total       $1.77 per panorama
                            boxes are 0.30 / 1.77  =  17 % of it

The + 40% line is the overhead of quality assurance: a fifth of the boxes are re-checked by a second annotator, and disagreements go to a third for adjudication. The scan dominates the drawing by 5 to 1, which changes the whole labelling strategy — almost all of the money is spent looking for faces rather than outlining them. Three consequences follow.

1. Annotate tiles, not panoramas. A 1024-pixel-square tile can be scanned in 6 seconds rather than 4 minutes. You lose the guarantee that a whole panorama was covered exhaustively, so you buy that guarantee back statistically: sample tiles with a known probability and reweight the results by the inverse of that probability, which recovers an unbiased estimate of the panorama-level rate.
2. Pre-annotate with the current model. Verifying and correcting a box the model has already proposed takes 1.2 seconds against 7.0 seconds to draw one from scratch, a 5.8-fold speedup on the drawing component.
3. Pre-annotation has a recall of its own, and if you do not measure it, your model’s recall is fiction. Annotators anchored on model proposals systematically miss whatever the model missed, because nothing on screen draws their attention to it. The way to measure that is to annotate a random 5% of tiles blind, with no proposals shown, and compare the two counts.

faces found by blind annotation on the 5 % control       1,412
faces found by model-assisted annotation on the same       1,285
                                                          -----
label-process recall  =  1,285 / 1,412             =       0.910

-> a model measured at 0.9955 recall against assisted labels
   is measured against a ground truth that is itself 9 % incomplete,
   and the missing 9 % is exactly the hard band the model also misses

This is the most dangerous number in the chapter, because it makes the shipped metric optimistic precisely where the risk is. The control has to be permanent, not a one-time audit.

5.2 Active learning: what to label next

A fixed labelling budget forces a choice about which images to spend it on — and one of the strategies below is essentially free, because the geometry labels the examples for you.

Active learning means letting the current model choose what gets labelled next, rather than sampling at random. Random sampling is the wrong default here for two reasons already established: 78% of tiles contain nothing at all (The content gate and why it saves less than you think), so most of the budget buys empty pictures, and 62% of the risk sits in a single size band (How small is a face in pixels).

Multi-view disagreement is free, self-labelling, and 18 times more efficient than random sampling, because the geometry supplies a near-certain positive: a face detected with confidence 0.94 in frame 2 and only 0.03 in frame 3, at a position the projection says should be visible, is almost certainly a face in frame 3 too. Mine those cases, send them to be labelled, and they land squarely in the hard size band.

The mix that works is 50% multi-view disagreement, 30% slice-targeted, and 20% random. That last share is not laziness: a purely targeted label set cannot tell you your overall miss rate, because it no longer resembles the traffic, so the random fifth exists to keep the label distribution estimable.

5.3 Synthetic augmentation

Labelled hard examples are scarce, so manufacture them — knowing that each technique comes with a specific way it can quietly teach the model the wrong thing.

Augmentation means transforming the images you already have to create new training examples. Two terms below need unpacking: MTF falloff is modulation transfer function falloff, the way a real lens loses fine detail toward the edges of the frame and at long distances, and Poisson blending is a compositing method that matches a pasted patch to its surroundings by matching gradients rather than pixel values, so the join is not visible as an edge.

The pasting failure is worth stating unprompted: if the model can detect the seam, it will, because the seam is a much easier feature than a face. Diagnose it by evaluating on real hard examples only; if synthetic-trained recall is 0.97 on synthetic and 0.71 on real 11 px faces, the model learned compositing.

5.4 Training, and the split that decides whether any of §6 is true

One training decision determines whether every metric in Metrics and why map is the wrong headline number is a measurement or a fantasy: how the data is divided between training and test.

The split is the most consequential line in this chapter and it cannot be random.

Multi view association is worth more than any detector improvement established that the same pedestrian appears in 4 to 6 consecutive frames of one capture run. The pipeline shards by capture run precisely to keep those frames together — sharding meaning splitting the work across many machines by some key, so that everything sharing a key lands on the same machine.

A random tile-level split therefore puts frame 2 of a pedestrian in the training set and frame 3 in the test set. Take five views of one face and an 80/10/10 split by tile — 80% of tiles for training, 10% for validation, 10% for test. Pick one of those five views and call it the test face. Its four siblings were each assigned independently:

P(one particular sibling is NOT in train)     =  1 - 0.80  =  0.20
P(none of the four siblings is in train)      =  0.20^4    =  0.0016
P(at least one sibling IS in train)           =  1 - 0.0016
                                              =  99.84 %

Essentially every face in the test set was memorized during training, at a slightly different size, pose and exposure. Measured, the damage runs in the direction that hurts:

random tile split      recall 0.9987      1.3 misses / 1,000 faces
run-level split        recall 0.9955      4.5 misses / 1,000 faces
                                          ---
                       the leak understates the headline metric by 3.5x

The chapter’s headline number would be wrong by three and a half times, every metric in Metrics and why map is the wrong headline number would inherit it, and nothing in the training loop would report an error. So the split is by capture_run_id — the same key the pipeline shards on (The pipeline) — with whole runs never spanning two sides. Two more cuts sit on top of it:

• By geography. Two runs down the same street a month apart share shopfronts, parked cars, and some of the same commuters, so run-level splitting alone still leaks at the street level. Hold out whole cities as a fourth, unseen-region set and report it separately: it is the only number that estimates performance in next year’s expansion markets.
• By time. Freeze the most recent six weeks as a temporal holdout. Rigs, firmware and auto-exposure policy change; a random-in-time split averages a regression across the change instead of surfacing it.

The rest of the training recipe follows from Objective and Active learning what to label next, and each row of the table below exists to defend a number stated elsewhere in the chapter. The sampler is the rule that decides how often each kind of tile appears in a training batch, which is how you fight the natural distribution without discarding data. A warm-start fine-tune means continuing to train the existing model on new data rather than starting over from random weights — much cheaper, and appropriate when the data has been added to rather than replaced. The hard-positive pool is the accumulated set of faces the model has been caught missing.

6. Metrics, and why mAP is the wrong headline number

How do you know the system works? Not by the field’s default metric, which fails here on three specific grounds; something has to replace it, a fairness table that reports one variable at a time turns out to hide the actual problem, and the experiment that would normally validate all of it is forbidden.

6.1 Three reasons mAP is actively misleading here

Three independent objections, each of which alone would be enough to disqualify mean average precision as the launch gate for this system.

1. It averages over recall levels you will never operate at. Average precision (AP) integrates precision over recall from 0 to 1 — that is, it sweeps the threshold across its whole range and averages the precision achieved at every recall level. Precision here is the fraction of blurred regions that really contained a face. You ship at recall 0.9955, so the only region of that curve you care about is the last half-percent, and AP weights it as half a percent of the answer.

2. COCO-style mAP averages over IoU thresholds from 0.5 to 0.95, which prices localization precision. COCO is the benchmark dataset whose evaluation convention became the field default; the convention requires a detection to overlap the true box by increasingly strict amounts and averages across those requirements, so a model is rewarded for drawing tight boxes. Here a loose box is free — every box is dilated before blurring anyway (Box dilation derived from the localization error) — while a tight box that clips an eye is a failure. mAP penalizes exactly the behavior the product wants.

3. It is one number over all object sizes, and 62% of the risk lives in one size band that holds 19% of the objects.

Two real candidate models make the point concretely. Compare the first column against the fourth: mAP says A is the better model, misses per 1,000 says B is four and a half times safer.

Model B misses 4.5x fewer faces and loses on mAP by 7 points. A team gating on mAP ships A. This is not a contrived example — it is what happens when one model is trained with a low box-loss weight (Objective) and a small-object-heavy sampler: it localizes worse and finds more.

The precision nobody quotes

None of the rows above states it, so state it here: the shipped operating point runs at precision 0.26.

at t = 0.035, per panorama:
   true objects   =  (3.2 faces + 1.8 plates) × 0.9955 recall  =  4.98
   false positives                                             = 14

   precision  =  4.98 / (4.98 + 14)  =  0.262

Roughly three of every four regions this system blurs contain nothing at all.

That is not a defect deferred to a later release. It is the accepted cost, and it is already priced. Those 14 boxes are the bulk of the 0.40% of image area that So the shipped threshold comes from a constraint not the optimum spends against its 0.5% cap — four-fifths of the budget — and The corrected cost model’s cost model puts a figure on that: 0.15 × 0.017 × 0.402^1.4 = $0.0007 of complaint cost per panorama.

State the 0.26 unprompted. An interviewer who hears “recall 0.9955” with no precision beside it concludes you have not looked at the other side of the confusion matrix.

6.2 The metrics to actually report

With the default rejected, here is the replacement set — and then the matching rule underneath every recall number, because a recall reported without a matching rule is not a number at all.

The matching rule underneath every recall number

Every recall number in this chapter is measured at IoU >= 0.35, matched greedily by descending score, one-to-one, in global panorama coordinates rather than per tile (Tiling and boundary losses).

Unpack that phrase. “Matched greedily by descending score, one-to-one” means: take the highest-scoring detection first, pair it with the ground-truth box it overlaps most, remove both from consideration, then repeat. No detection can claim two faces, and no face can be credited to two detections.

Say the criterion before anyone asks. A recall without a matching rule is not a number — and this is a chapter whose whole argument is that localization does not matter, which is precisely the condition under which the choice of matching rule moves the answer.

Here is how far it moves it. Take a prediction with the right centre and the wrong size, s times the true width:

• Too small (s < 1): the prediction sits inside the truth, so the intersection is the prediction and the union is the truth. IoU = s².
• Too large (s > 1): the truth sits inside the prediction, so IoU = 1/s².

Invert those and each threshold turns into a range of box widths it will accept:

IoU >= 0.50   ->   sqrt(0.50) = 0.71x   to   1/sqrt(0.50) = 1.41x  true width
IoU >= 0.35   ->   sqrt(0.35) = 0.59x   to   1/sqrt(0.35) = 1.69x  true width

The loose side is the side the product does not care about. Every box is dilated 15% before blurring (Box dilation derived from the localization error), so a box 1.69x too large costs blur area and nothing else. Failing it as a miss would report a privacy incident that did not occur.

The tight side is a real failure — and it is deliberately not IoU’s job here. A box at 0.59x covers about a third of the face by area and still passes at IoU 0.35.

Catching that is what effective recall (Box dilation derived from the localization error) exists for. It credits a face only when the blur covers the identity-bearing region, defined as under-covering by no more than 13% of face width. A 0.59x box fails that even after the 15% dilation, since 0.59 × 1.15 = 0.68, still 32% short. Those boxes are exactly the residual that survives dilation in Box dilation derived from the localization error.

IoU 0.35 asks “did you find it”; coverage asks “did you cover it”; only the first one is allowed to be generous.

Report the same recall at IoU 0.5 as a sanity column, because the gap between the two prices what lambda_box = 0.3 cost in localization:

                    IoU >= 0.35     IoU >= 0.50     gap
   Model A             0.9812          0.9788      0.24 pts
   Model B             0.9958          0.9902      0.56 pts
   shipped             0.9955          0.9908      0.47 pts

Model B’s gap is more than twice Model A’s, which is “B localizes worse and finds more” measured directly instead of inferred from a 7-point mAP drop. A gap that grows release over release is the signal that the box term has been de-weighted past the point where a 15% dilation can still cover it.

One definition the table hides: plate legibility uses >= 2 characters because a two-character fragment plus a make, model, colour and street address is routinely enough to identify a vehicle. That threshold is a policy call about what “identifiable” means, not a property of the blur, and it belongs to whoever signed the annotation guideline in Framing.

6.3 The slice grid, and why marginals hide the problem

Measuring fairness correctly is its own design problem, because the standard presentation — one table per demographic variable — is structurally incapable of showing the problem this system actually has.

A slice is a subset of the data defined by some attribute, and a marginal is what you get when you report performance broken down by one attribute at a time, averaging over all the others. Here is the marginal recall by skin tone at the shipped threshold, using the Monk scale, a published ten-point skin-tone scale designed for exactly this kind of measurement:

Monk 1-2   0.9971      2.9 misses / 1,000
Monk 3-4   0.9968      3.2
Monk 5-6   0.9954      4.6
Monk 7-8   0.9902      9.8
Monk 9-10  0.9847     15.3      <- 5.3x the miss rate of the lightest band

Read off the marginal gap: 0.9971 - 0.9847 = 0.0124, or 1.24 points between the lightest and darkest bands. That is the number a standard fairness table would report.

Now cross the same data with illumination. The rightmost column is the skin-tone gap within each lighting condition — compare the top row to the bottom one:

The gap is an interaction, not a main effect. A main effect is what one variable does on its own; an interaction is what two variables do only in combination. In daylight the skin-tone gap is 0.12 points and would round to nothing; at dusk it is twelve times that. A slice table reporting only the skin-tone marginal averages the 0.12 and the 2.47 together and reports 1.24 — understating the worst cell by half and pointing at the wrong fix, since it suggests the model has a general problem with darker faces when in fact it has a specific problem with darker faces in low light.

The mechanism is physical, and it is worth stating because it names the fix. The camera auto-exposes for the whole scene, which is dominated by sky and road surface. In low light the exposure therefore settles where a darker face lands in the bottom two stops of the sensor’s range — a stop is a doubling or halving of light, so the bottom two stops are the darkest quarter of what the sensor can record. Down there the face’s contrast against its background, which is precisely the signal a detector uses, is compressed toward the sensor’s noise floor. This is a sensor-and-pipeline problem that shows up as a model fairness metric.

Four fixes address it, listed here in the order of how much they return for what they cost:

1. Gate on the worst cell, not the worst marginal. Free, and it is what makes the rest happen.
2. Exposure and gamma augmentation matched to the real capture exposure distribution. Cheap, and it is the largest single model-side gain.
3. Slice-targeted labeling of dusk and night captures — the Active learning what to label next strategy pointed at the four worst cells. 40k boxes, about $2,400.
4. Local tone mapping before the detector on tiles whose histogram indicates crushed shadows. 3% of tiles, negligible cost.

after fixes 2-4:
   dusk,  Monk 8-10   0.9793  ->  0.9951      (gap 1.49 -> 0.31 pts)
   night, Monk 8-10   0.9661  ->  0.9904      (gap 2.47 -> 0.44 pts)
   overall recall     0.9955  ->  0.9968

What interviewers probe: whether “we would check for bias” comes with a measurement design. The strong answer names the two-way grid, explains why marginals hide interactions, gives the physical mechanism, and gates on the worst cell.

6.4 There is no A/B test, and what stands in for one

Almost every system in this repository is validated by a controlled online experiment. This one cannot be — for a structural reason — so three partial substitutes have to stand in, and even together they leave one gap open.

An A/B test splits live traffic into two arms, gives one the new system and the other the old one, and compares outcomes; the arm left on the old system is the control or holdback. Skipping that stage here is defensible, but say why rather than letting it pass unmentioned: the A/B test is structurally forbidden, not merely inconvenient. The control arm of “does blurring work” is published unblurred imagery, which is the harm itself. There is no holdback and there never will be one. Three things stand in, and they cover different parts of the gap between an offline number and field behaviour.

1. The offline number is not the field number, and the reweighting is the difference. The eval set is enriched by Active learning what to label next’s targeting — half of it is multi-view-disagreement mines — so the unweighted mean over it reads 0.9931, or 6.9 misses per 1,000, which is pessimistic. Reweighting each cell of the The slice grid and why marginals hide the problem grid by its frequency in real capture traffic gives 0.9955 and 4.5. Publish both and the weights. An unweighted mean over a deliberately hard eval set is a number nobody can act on; a reweighted one whose weights are not published is a number nobody can check.

2. Shadow scoring, which is the closest thing to a counterfactual you get. A counterfactual is what would have happened under the other choice. That is exactly what an A/B test buys and exactly what you cannot have here.

Shadow scoring approximates it. Run the candidate model over the last 30 days of already-published captures — 164 million panoramas — and compare its box set against the incumbent’s, one panorama at a time.

recall 0.9955 -> 0.9968 (the §6.3 fix set)
   new faces per panorama  =  3.2 × 0.0013  =  0.0042
   × 164M panoramas                         =  ~684,000 boxes of diff

You do not verify 684,000 boxes; you verify a stratified sample of them. The point is that the diff is pre-filtered to disagreements, so the yield per verified box is two orders of magnitude above a fresh random audit. That is the same argument Active learning what to label next makes about multi-view disagreement, applied to model comparison instead of labeling. The readout is a per-cell win/loss on the slice grid.

3. Staged rollout by capture run, with the audit loop as the online metric. Route 5% of runs to the candidate, keeping whole runs on one model so that multi-view association never has to reconcile two detectors inside a run. Human review and the appeal path’s daily per-cell miss rate is the readout; the appeal rate (Human review and the appeal path loop 2) is the lagging confirmation, and it lags by months, which is exactly why it cannot be the gate.

The gap none of the three closes is measurement position. The audit taps the pipeline before association (The pipeline), so it measures the detector, not the shipped system, and those differ by 74x. That is the right tap point — the detector is what changes between releases — but it means the post-association number is derived through k_eff, and k_eff is re-measured on the tracked-pedestrian set only quarterly. A pose or depth regression would therefore reach the appeal queue months before it reached any offline metric. Alarm on the Multi view association is worth more than any detector improvement fallback rate instead: the share of rescues that find an empty gate is a direct, daily, free signal that the geometry still works.

7. The pipeline

Everything above now assembles into the batch job that actually runs. Four of its stages deserve a closer look — a cheap classifier that skips most of the work, the tiling scheme, the margin added to every box before blurring, and the redaction step itself, which turns out to be the easiest thing in the chapter to get catastrophically wrong.

Here is the whole job as one diagram. Read it top to bottom; the two branches out of the content gate (keep 39%, skip 61%) and the audit tap on the right are the parts worth pausing on.

flowchart TD
    CAP[("Capture store<br/>2e9 panoramas · 36 PB")] --> SH["Shard by capture run<br/>keeps adjacent frames together"]
    SH --> DEC["Decode + undistort<br/>0.74 CPU-s each"]
    DEC --> TIL["Tile 1024^2, overlap 192<br/>stride 832 · 16 x 8 = 128 tiles"]
    TIL --> GATE{"Content gate<br/>0.6 GFLOP/tile<br/>recall 0.9999"}
    GATE -->|"39% keep"| DET["Detector<br/>410 GFLOP/tile<br/>anchor-free · P2-P6"]
    GATE -->|"61% skip"| NOBOX["No detections"]
    DET --> NMS["NMS in global panorama<br/>coordinates, not per tile"]
    NMS --> MV["Multi-view association<br/>project across 4-6 frames<br/>union the detections"]
    MV --> DIL["Dilate boxes<br/>+15% of width, +2 px floor"]
    DIL --> BLUR["Irreversible redaction<br/>decimate to 8x8 + noise"]
    NOBOX --> PASS["Pass original bytes through<br/>NO re-encode"]
    BLUR --> ENC["Re-encode affected<br/>2048^2 regions only"]
    ENC --> OUT[("Published tiles")]
    PASS --> OUT
    NMS --> AUD["Audit sampler<br/>stratified by slice cell"]
    AUD --> HR["Human recall audit"]

    style GATE fill:#bc6c25,color:#fff
    style DET fill:#1d3557,color:#fff
    style MV fill:#2d6a4f,color:#fff
    style BLUR fill:#9d0208,color:#fff

Follow one panorama through that diagram before dissecting it.

1. Capture store. 2e9 panoramas, 36 PB of bytes.
2. Shard by capture run, which keeps adjacent frames together on the same worker.
3. Decode and undistort, 0.74 CPU-seconds each — a CPU-second being one second of one processor core.
4. Tile. Tile size 1024 px, overlap 192 px, so stride 832, and the panorama yields 16 columns × 8 rows = 128 tiles.
5. Content gate. A classifier costing 0.6 GFLOP per tile, run at recall 0.9999. The 61% of tiles it skips produce no detections at all, and their original bytes pass through with no re-encode.
6. Detector, on the 39% of tiles the gate keeps: 410 GFLOP per tile, anchor-free, pyramid levels P2 through P6.
7. NMS, performed in global panorama coordinates rather than per tile, so partial detections from adjacent tiles merge instead of double-counting.
8. Multi-view association across 4 to 6 frames: project detections between frames and union them.
9. Dilate every surviving box by 15% of its width, with a 2-pixel floor.
10. Redact irreversibly: decimate to blocks and add noise.
11. Re-encode affected 2048² regions only, then write the published tiles.

One branch runs off to the side: an audit sampler taps the detections before association and feeds a human recall audit. It is stratified by slice cell, meaning it deliberately draws more from the risky cells of The slice grid and why marginals hide the problem’s grid than their traffic share would give.

Four of those nodes carry far more weight than their labels suggest, and three of them are the kind an interviewer will point at precisely because they look like plumbing.

SH — shard by capture run. The shard key is capture_run_id, and it is a hard dependency of Multi view association is worth more than any detector improvement rather than a throughput tweak. Multi-view association is a join across the 4-6 consecutive frames of one pedestrian; shard by panorama id or by a hash of the tile and those frames land on different workers, so the join stops being local and becomes a global shuffle over 36 PB. The chapter’s highest-leverage design element is bought by a shard key, and it would be silently destroyed by the most natural key anyone would otherwise reach for. The same key does two more jobs: a run is the unit of retry when a worker dies, and it is the unit of the train/test split (Training and the split that decides whether any of 6 is true).

DEC — decode and undistort. Each sensor image is resampled onto the equirectangular sphere before tiling. Without that step angular resolution varies across the frame, and How small is a face in pixels’s uniform 36.98 px per degree — the constant every face-size, anchor-IoU and motion-smear derivation in this chapter rests on — holds only near the optical axis.

AUD taps before MV, deliberately. The human recall audit therefore measures detector recall, not the shipped post-association recall, and the two differ by 74x. That is the right tap because the detector is what changes between releases, but it makes the per-cell launch gate a gate on the detector; the panorama-level number is derived and re-measured on its own cadence (There is no ab test and what stands in for one).

ENC — whole 2048² regions. An affected panorama re-encodes block-aligned 2048² regions rather than tight crops around each box, because JPEG quantization is block-aligned and a ragged re-encoded region leaves a visible rectangle. One such region is 4.7% of an 88.6 MP panorama and an affected panorama averages 3.2 of them, which is where Scale and cost’s “16% re-encode ~15% of pixels” comes from.

7.1 The content gate, and why it saves less than you think

The second model in the system is a throwaway classifier that decides which tiles are worth running the detector on — and the saving it delivers is roughly half what the naive calculation promises, for a reason that generalizes to every cascade.

Seventy-eight percent of tiles contain nothing but sky, road surface, foliage, or upper-storey facade, and hold no face or plate at all. A tiny classifier running at one-sixteenth resolution can skip those for 0.6 GFLOP per tile against the detector’s 410, a factor of nearly 700.

But that gate is now the first stage of a cascade — a chain of models in which each stage only passes survivors to the next — and a face the gate drops is an unrecoverable miss, so the gate’s recall must be higher than the detector’s, not lower. The recall of a cascade can never exceed the recall of its first stage. At its natural operating point the gate skips 78% of tiles at recall 0.994, which would add 6 misses per 1,000 on top of the detector’s 4.5. Push it to recall 0.9999 and the skip rate falls. The costs below are given first in GFLOP and then in TFLOP, a TFLOP being a trillion floating-point operations, or a thousand GFLOP:

gate recall   0.9940  ->  skip 78 %   ->  keep 22 %
gate recall   0.9990  ->  skip 71 %   ->  keep 29 %
gate recall   0.9999  ->  skip 61 %   ->  keep 39 %

cost per panorama (128 tiles, §7.2), in GFLOP and then in TFLOP:

   no gate     128 × 410 GFLOP                =  52,480 GFLOP  =  52.5 TFLOP
                                                                  (the baseline)

   naive gate  128 × 0.6 GFLOP     (gate on every tile)
                 + 0.22 × 128 × 410 GFLOP     =  11,622 GFLOP  =  11.6 TFLOP
               52,480 / 11,622  =  4.5x cheaper

   safe gate   128 × 0.6 GFLOP
                 + 0.39 × 128 × 410 GFLOP     =  20,544 GFLOP  =  20.5 TFLOP
               52,480 / 20,544  =  2.55x cheaper

The cascade is worth 2.55x, not 4.5x, and the difference is the price of the first stage’s recall. Quoting the 4.5x is the standard error; deriving the 2.55x is the answer.

State the gate as a model, since it is the second and last one in the system.

• It eats the same tile as the detector, downscaled sixteen-fold.
• It emits a single number: the probability that this tile contains anything worth looking at.
• Its labels cost nothing extra. A tile is a positive example exactly when one of the detector’s hand-drawn boxes falls inside it, so every box already paid for produces gate labels for free.
• It is trained on the same tile pool as the detector. That is an assumption this chapter makes rather than derives, and it is worth stating out loud, because the gate’s job is the opposite of the detector’s: a pool sampled to favour hard positives (Training and the split that decides whether any of 6 is true) under-represents exactly the empty sky and road surface the gate must learn to reject.
• It is served as the first stage of the batch pipeline, ahead of the detector.
• You know it works from its recall on a held-out set, measured separately from the detector’s. A cascade’s recall is bounded by its first stage, and a gate failure is invisible in the end-to-end number — the face never reaches the model that would have been blamed for missing it.

7.2 Tiling and boundary losses

A panorama is far too large to feed a detector whole, so it is cut into tiles — which raises the question of what happens to objects unlucky enough to land on a cut.

The stride of a tiling is the distance between the top-left corners of adjacent tiles, so stride equals tile size minus overlap. Overlap must exceed the largest object you expect to be split, or a face straddling a seam becomes two partial faces, each too small and too incomplete to detect:

tile 1024, overlap 192  ->  stride 832
   13,312 / 832 = 16 columns (the panorama wraps at 360 deg)
    6,656 / 832 =  8 rows                    =  128 tiles

largest in-scope face: a pedestrian at 1 m
   9.17 / 1 = 9.17 deg  ->  9.17 × 36.98  =  340 px
overlap 192 px  ->  faces above 192 px can be split across tiles

mitigation is NOT more overlap. Raising 192 -> 340 gives stride 684:
   13,312 / 684 = 19.5 -> 20 columns
    6,656 / 684 =  9.7 -> 10 rows        =  200 tiles
   200 / 128 = 1.56, so 56 % more compute -- not a rounding error.

instead:
   - NMS in global panorama coordinates so partial detections merge
   - any box touching a tile edge is dilated to the union of its
     cross-boundary partners before blurring
   - large faces are easy: recall on >192 px faces is 0.9997 even
     when split, because half a large face is still a strong detection

7.3 Box dilation, derived from the localization error

Because Objective deliberately under-weighted the box term, the predicted boxes are systematically imprecise; the safety margin that decision spent has to be bought back here, along with the metric that would catch it if the margin were wrong.

Blur exactly what the box says and you leave eyes visible whenever the box regressor under-shoots. So measure the under-coverage distribution rather than guessing a margin. The percentiles below read as follows: p50 is the median error, p90 is the error exceeded by one box in ten, p99 by one in a hundred, and p999 by one in a thousand. A negative number means the predicted box was narrower than the true face.

signed box-width error as a fraction of true face width, on the eval set:
   p50   +0.02      (slightly large)
   p90   -0.06
   p99   -0.12
   p999  -0.19

dilate by 15 % of width with a 2 px floor:
   covers p99 fully; p999 residual is 4 % of face width -- a sliver of jaw
   area cost: (1.15)^2 = 1.32, so 32 % more blurred area per box
   t = 0.035 spends 0.40 % raw (§2.2), so 0.40 · 1.32 = 0.53 %
   -- over the 0.5 % cap, which is why the threshold has to move

-> dilate 15 % and lower the detection threshold less (t 0.035 -> 0.045)
   net: recall 0.9955 -> 0.9948 on detection, but coverage-adjusted
   effective recall (defined immediately below -- it is NOT
   full geometric containment) 0.9903 -> 0.9944

What “handled” means, and why the metric does not exist until you say

Effective recall is the fraction of in-scope faces whose blur covers the identity-bearing region — the eyes-nose-mouth triangle — not the fraction whose blur geometrically contains the whole annotated box. Operationally, a face counts as handled when the blurred region under-covers its true width by no more than about 13%. That is the definition the reported numbers encode, and it has to be stated because two different definitions give answers a factor of two apart on the same model.

The threshold sits near 13% for an anatomical reason, not a statistical one. An annotated face box runs from hairline to chin and ear to ear; the features that make a face matchable occupy the middle. Lose 13% of the width and what stays exposed is a jaw edge, a hairline, an ear — the periphery. Lose 30% and you are exposing an eye. Somebody has to look at redactions and draw the line between those two cases, and that is a labelling decision of exactly the kind Framing says every number in this chapter inherits from. It is the same move as the >= 2 characters rule for plate legibility in The metrics to actually report: a policy call about what “still identifiable” means, made by whoever signed the annotation guideline, and until it is made effective recall is not a number at all — the same way “recall” is not a number until “face” is defined.

Checking the 13% against the error distribution

Here is the arithmetic that takes detection recall to effective recall, so it can be checked rather than believed.

Start by fitting a normal distribution to the measured percentiles. Two rows are enough to pin down its two parameters, mu (the mean) and sigma (the standard deviation). A standard normal’s 10th percentile sits at z = −1.2816 and its 1st percentile at z = −2.3263, so:

   mu - 1.2816 · sigma  =  -0.06        (the p90 row)
   mu - 2.3263 · sigma  =  -0.12        (the p99 row)

subtract:   1.0448 · sigma  =  0.06   ->  sigma = 0.0574
back-sub:   mu = -0.06 + 1.2816 × 0.0574          =  +0.0136

That mu = +0.0136 reproduces the measured p50 = +0.02 row, which is the sanity check that the fit is not nonsense.

Now use the fit to recover the tolerance the reported numbers imply. Phi^-1 below is the inverse standard normal — feed it a probability, get back the z-score below which that fraction of the distribution falls:

pre-dilation, t = 0.035:
   effective / detection  =  0.9903 / 0.9955  =  0.99478
   so 0.522 % of DETECTED faces are under-covered past tolerance
   z = Phi^-1(0.00522) = -2.56
   tolerance c = -(mu + z·sigma) = -(0.0136 - 2.56·0.0574) = 0.133
                                        <- 13 % of face width

post-dilation, t = 0.045, boxes grown 15 %:
   a face now fails only if the raw error is below -(0.133 + 0.15) = -0.283
   z = (-0.283 - 0.0136) / 0.0574 = -5.17  ->  P = 1.2e-7 under the fit
   effective recall = 0.9948 · (1 - 1.2e-7) = 0.9948
   reported: 0.9944.  The 0.0004 gap is the tail, not the bulk.

Notice the mismatch in that block: the fit predicts a post-dilation failure rate of 1.2e-7, and the eval set reports a number consistent with 4.0e-4. Three thousand times more. The fit is not broken — the distribution has a tail the fit does not model, and understanding that tail is what justifies the whole dilation strategy.

The residual after dilation is entirely a heavy-tail phenomenon, and that is precisely why dilation buys so much.

The evidence that the tail is heavy is already in the percentile table. The normal fit predicts p999 = -0.164; the eval set measures -0.19. So the real distribution is already fatter than Gaussian one box in a thousand out, and the gap widens further beyond that. The reported 0.9944 implies 4.0e-4 of detected faces still under-covered past tolerance — about 3,500x what the normal fit predicts at -0.283, and still 8.7x what a log-linear extrapolation of the p99 -> p999 segment predicts.

That is not the same population as the bulk. It is the boxes that scraped through the IoU >= 0.35 match at the loose end. The metrics to actually report showed that rule admits boxes down to 0.59x the true width, and 0.59 × 1.15 = 0.68, so a 32% under-coverage survives dilation intact.

So growing every box by 15% moves the whole bulk out of danger at once — the bulk was only ever a few hundredths of a width short — and leaves behind a separate, small, badly-localized population. Reaching that population would take a much bigger margin:

a 0.59x box must reach 0.867x  (= 1 - the 0.133 tolerance)
   required dilation  =  0.867 / 0.59   =  1.47   ->  47 %
   area cost          =  1.47^2         =  2.2x the blurred area

The So the shipped threshold comes from a constraint not the optimum budget will not pay 2.2x. That residual gets bought with multi-view association (Multi view association is worth more than any detector improvement) or with lambda_box, not with a bigger margin.

The trap, and it is worth walking into deliberately

Read “handled” as full geometric containment — the blurred box must contain the true box, so the tolerance is c = 0 — and run the same fit:

P(signed error >= 0)  with mu = 0.0136, sigma = 0.0574
   z  =  (0 - 0.0136) / 0.0574  =  -0.237
   P  =  0.59

So under the containment reading, effective recall is about 0.59, not 0.9903. That is not a rounding disagreement, it is a different metric.

It follows directly from the percentile table above: the median signed error is +0.02, barely above zero, so on the containment reading roughly four boxes in ten fail. A reader who takes the literal reading, checks it against the percentiles, and gets ~0.6 has not made an error — the chapter had, by leaving the definition implicit.

When a metric’s headline number and its literal-reading number differ by 40 points, the definition is the deliverable and the number is a footnote.

Does the dilation fit the area budget?

One more consistency check the numbers force, this time on the area side of the same trade. It uses the same anchor as So the shipped threshold comes from a constraint not the optimum, which is the point: every area figure below is read off The corrected cost model’s measured table, log-log interpolated on the 0.02..0.05 bracket as area(t) = 0.0134 · t^-1.02. The 0.5% is the product cap and never a measurement.

area(0.035) = 0.402 %   ->  dilated  0.402 · 1.3225 = 0.532 %   OVER the cap
area(0.045) = 0.312 %   ->  dilated  0.312 · 1.3225 = 0.412 %   under, by 0.088 pts

tightest threshold that still fits:  1.3225 · area(t) = 0.5 %  ->  t = 0.037

So the dilation is affordable, not marginal, and the arithmetic no longer depends on which section you anchor to. Two things in that block are worth saying out loud.

The constraint that matters is on the dilated area, because the dilated blur is what publishes. That is why t = 0.035 fails at 0.53% even though its raw 0.40% sits comfortably inside the cap. Checking a budget against the undilated boxes measures a quantity no user ever sees.

t = 0.037 is the tight solution, and the chapter ships t = 0.045. That is deliberately conservative: it gives up about 0.0005 of detection recall to buy 0.09 points of area headroom. The trade is defensible against dense-urban panoramas, whose blurred area runs far above the mean and whose p99 is the number The metrics to actually report actually gates on. But it is a choice, and a design that quotes 0.045 without knowing that 0.037 would have done has not finished the arithmetic.

Effective recall is what matters, and it is strictly below detection recall: a detected-but-partially-blurred face is still a published face. Almost nobody measures it. Define it — in the annotation guideline, with a number — report it, and gate on it.

7.4 Redaction must be irreversible

Everything up to this point was about finding faces. Now they have to be destroyed — and the standard tool for the job does not destroy anything at all. It merely attenuates, and attenuation is reversible.

A Gaussian blur is a convolution: each output pixel is a weighted average of its neighbours.

Any convolution can be described by what it does to each spatial frequency in the image — the Fourier transform of its kernel. For a Gaussian that transfer function is exp(-2·pi^2·sigma^2·f^2), where sigma sets the blur strength and f is frequency measured in cycles per pixel.

That expression gets small at high frequencies, but it is never zero. That is the whole problem: a frequency that was only attenuated rather than removed can be divided back out. Wiener deconvolution is the standard method for doing exactly that, and it succeeds wherever the attenuated signal still sits above the sensor noise.

So the question is where the blur pushes the signal below the noise. SNR is the signal-to-noise ratio; 40 dB (decibels, a logarithmic ratio scale) corresponds to a signal 100 times larger than the noise in amplitude. Set the transfer function equal to 1/SNR and solve for the frequency:

recoverable up to the frequency where the blur kernel exceeds the noise:
   exp(-2·pi^2·sigma^2·f^2)  =  1/SNR

at 40 dB SNR (amplitude ratio 100) and sigma = 2 px:
   2·pi^2·4·f^2 = ln(100) = 4.605   ->   f = sqrt(4.605/78.96)
                                          = 0.2415 cycles/px
                                     ->   period 4.1 px

an eye on a 68 px face is ~14 px across -> RECOVERABLE, and not
marginally: the cutoff sits three times finer than the feature

A Gaussian blur at typical strength is a lossy filter, not a redaction. The correct primitive destroys information rather than attenuating it:

1. decimate the region to a grid of block means         (many-to-one, no inverse;
                                                         block size derived below)
2. add zero-mean noise at the block level               (defeats averaging attacks
                                                         across the multi-view frames)
3. re-encode so the DCT coefficients are quantized      (removes residual structure)

To decimate here means to replace every block of pixels by a single number, its mean — a many-to-one operation, so there is no inverse to invert. “Zero-mean noise” is random perturbation that averages to nothing, so it does not shift the picture’s brightness but does destroy the exact values. The DCT, discrete cosine transform, is the frequency decomposition that JPEG compression is built on; quantizing its coefficients — rounding them to a coarse grid — is how JPEG discards detail, and running the redacted region through it once more removes whatever faint structure survived the first two steps.

But “8x8” has to mean a block size, not a grid, or the primitive evaporates on exactly the faces The small object problem derived spends itself on. Subject the replacement to the same analysis the Gaussian just failed. A fixed 8×8 grid keeps 64 numbers whatever the box is, so its strength is a function of box width:

The ratio below is pixels in / numbers out, so it says how much information the step throws away. Read the top row first:

face   8 px:   64 means from      64 px  ->      1.0 : 1     <- the identity function
face  11 px:   64 means from     121 px  ->      1.9 : 1
face  16 px:   64 means from     256 px  ->      4.0 : 1
face  32 px:   64 means from   1,024 px  ->     16.0 : 1
face  68 px:   64 means from   4,624 px  ->     72.3 : 1

The smallest in-scope face is redacted by doing nothing at all — 64 pixels in, 64 numbers out. And the 8-16 px band that holds 62% of the misses (How small is a face in pixels) gets only a 2-4:1 reduction, which is a 1.4 px box filter and deconvolves about as well as the Gaussian this section has just rejected.

The frequency argument does not rescue it either, because on an 8×8 grid the cutoff scales with the face. Each block is w/8 wide, and destroying a repeating pattern takes at least two blocks per cycle, so the finest surviving detail is w/4 — a quarter of the face, whatever the face’s size:

68 px face:  cutoff 68/4 = 17 px,  eye 14 px   ->  17/14 = 1.21x clear
11 px face:  cutoff 11/4 = 2.75 px, eye 2.3 px ->  2.75/2.3 = 1.20x clear

Scale-invariant in frequency, and collapsing in sample count — the redaction is weakest exactly where the privacy risk is highest.

The fix is one line, and it makes the guarantee scale-free instead of scale-invariant. Instead of fixing the number of blocks, fix the minimum block size at 6 px and let the block count follow:

blocks per side   n = clamp(1, 8, floor(w / 6))      w = dilated box width in px
                    (i.e. floor(w/6), then held to at least 1 and at most 8)

   w =  11 px  ->  floor(1.8)  = 1  ->  n = 1  ->    121 : 1   (one mean; gone)
   w =  16 px  ->  floor(2.7)  = 2  ->  n = 2  ->     64 : 1
   w =  32 px  ->  floor(5.3)  = 5  ->  n = 5  ->     41 : 1
   w =  68 px  ->  floor(11.3) = 11 ->  n = 8  ->     72 : 1   (capped)
   w = 340 px  ->  floor(56.7) = 56 ->  n = 8  ->  1,806 : 1   (capped)

ratio is w^2 / n^2, so it is (w/n)^2, and w/n >= 6 by construction

minimum over every width from 8 to 400 px:  36 : 1,  at w = 12, 18, 24, 48

A 6 px block floor guarantees at least a 36:1 sample reduction at every face size — because w/n >= 6 holds by construction, and 6² = 36 — against 1.9:1 for the fixed grid at 11 px.

The 8-block cap is the other end of the same rule. Without it, a 340 px face becomes one flat rectangle, which reads as a rendering bug rather than a redaction and draws precisely the attention the blur exists to avoid.

Step 2 matters more than it looks: the same face appears in 4-6 published frames, and averaging independent redactions of the same content reduces noise by sqrt(k). Block-level noise that is correlated across frames of the same identity removes that attack.

8. Scale and cost

Price a year of the pipeline and the interesting result is not the total — it is that compute turns out not to be the constraint at all.

In the tally below, 300 TFLOP/s “effective” is the throughput a graphics processor actually sustains on this workload, well below its datasheet peak.

per panorama
   decode 88.6 MP at ~120 MP/s per core            0.74 CPU-s
   tiles                                           128
   content gate  128 × 0.6 GFLOP                   0.077 TFLOP
   detector      0.39 × 128 × 410 GFLOP           20.5  TFLOP
   association + NMS                               negligible
   re-encode: 84 % of panoramas have zero
      detections -> pass bytes through
      16 % re-encode ~15 % of pixels               0.045 CPU-s amortized

Multiply each per-panorama figure by 2e9 panoramas, convert to machine-hours by dividing by 3,600, then price it:

detector    2e9 × 20.5 TFLOP                =  4.11e22 FLOP
            / 300e12 FLOP/s effective       =  1.37e8 GPU-s
            / 3,600                         =  38,000 GPU-hours
            × $2.50/GPU-hour                =  $95,000

decode      2e9 × 0.74 CPU-s                =  1.48e9 CPU-s
            / 3,600                         =  411,000 CPU-hours
            × $0.035/CPU-hour               =  $14,400

re-encode   2e9 × 0.045 CPU-s = 9.0e7 CPU-s =  25,000 CPU-hours
            × $0.035/CPU-hour               =  $   880

                                               --------
                                               $110,300 / year
                       $110,300 / 2,000     =  $55.14 per million panoramas
                                            =  $5.51e-5 per panorama

Three readings of that table, and the third is the one that matters:

1. The detector is 86% of cost; decode is 13%; re-encode is 1% — and re-encode is 1% only because 84% of panoramas are passed through untouched and the rest re-encode a 15% crop. Invert that: $880 / (0.16 × 0.15) = $36,700 is what the naive pipeline that re-encodes every pixel of every panorama would cost. The pass-through is a 42x saving on that line.
2. The whole thing is $110k of compute for a year of planetary imagery, which is small enough that cost is not the interesting constraint.
3. The binding constraint is I/O throughput — input and output, meaning how fast bytes can be moved on and off storage. There are 36 PB to read and 36 PB to write. Over a 30-day processing window that is

36 PB / (30 × 86,400 s)  =  13.9 GB/s sustained, each way

sustained for a month, which sizes the storage tier, the network, and the shard layout. Say this out loud: at planet scale the answer to “how much does it cost” is usually “less than you expect, and the real question is whether your storage system can stream 14 GB/s for thirty days.”

9. Failure modes

The system breaks in the field in a handful of characteristic ways, each with a mechanism, a number that detects it, and a control — and the summary table at the end is the one to be able to reproduce from memory.

9.1 Demographic recall gap

Covered in The slice grid and why marginals hide the problem: the gap is a skin-tone × illumination interaction driven by auto-exposure, marginal slice tables hide it, and the fixes are worst-cell gating, exposure augmentation, slice-targeted labeling, and local tone mapping. This is the failure to volunteer before being asked, because it is the one with a headline attached.

9.2 Reflections and printed faces

Some faces are present in the image but absent from the training distribution — and among them hides a policy question: what to do about faces that are not attached to a person at all.

audit of 400 hand-verified misses in commercial districts:

   face reflected in a shop window          31 %   <- a REAL person, must blur
   face on a bus-shelter advertisement       8 %   <- not a real person, blur anyway
   face in a photograph in a window display  6 %   <- not a real person, blur anyway
   face on a passing vehicle's wrap          3 %

recall on reflected faces      0.712
recall on all faces            0.9955

A reflection is a real, identifiable person and the detector is 64x worse at it — a 28.8% miss rate against 0.45% overall. The mechanism: reflections are low-contrast, geometrically distorted, often mirrored, and superimposed on the interior scene behind the glass, so they match almost none of the training distribution. The fix is targeted data — mine glass surfaces with a material classifier, label the reflections in them, and oversample.

The printed-face rows go the other way and the policy is simple: blur them too. They cost blur area and nothing else, and the alternative — a “is this a real person” classifier — is a second model with its own recall, in series, which can only reduce the system’s recall.

9.3 Motion blur is physics, not modelling

Part of the dusk recall gap is caused before the image is even recorded, which means no amount of training data can address it.

Motion smear is what happens when the subject moves across the sensor while the shutter is open: the exposure time multiplied by the relative speed gives the distance smeared, which converts to an angle and then to pixels using the same 37 pixels per degree from How small is a face in pixels.

vehicle at 40 km/h  =  11.1 m/s

daylight exposure 1/500 s
   distance smeared  =  11.1 / 500          =  0.022 m  =  22 mm
   angle at 5 m      =  atan(0.022 / 5)     =  0.252 deg
   pixels            =  0.252 × 36.98       =  9.3 px of smear
   on a 68 px face   =  9.3 / 68            =  14 % smear, tolerable

dusk exposure 1/125 s
   distance smeared  =  11.1 / 125          =  0.089 m  =  89 mm
   angle at 5 m      =  atan(0.089 / 5)     =  1.01 deg
   pixels            =  1.01 × 36.98        =  37 px of smear
   on a 68 px face   =  37 / 68             =  55 % smear -> destroyed

The dusk recall cell in The slice grid and why marginals hide the problem is partly an optics problem, and no amount of training data fixes a face that was not resolved by the sensor. The fixes are a capture-side exposure policy (cap exposure at 1/500 s and accept sensor noise, which a detector tolerates far better than smear) and a speed policy for evening captures. Say this: the strongest answer to a model failure is sometimes a change to the thing producing the data.

9.4 Plate-specific failures

Licence plates share a detector with faces but fail in their own ways, and each of the four below needs a different fix. “Retroreflective blowout” names the effect of a plate’s reflective coating throwing light straight back at the camera and saturating the sensor to pure white, which erases the characters and the plate’s own edges together.

9.5 Summary

Every failure above, gathered into one table: the mechanism that causes it, the measurement that would reveal it, and the control that holds it down.

10. Human review and the appeal path

Humans stay in the loop after launch, and one point recurs everywhere they appear: each loop must be sized from a different number, and using the wrong one over-staffs a queue by two orders of magnitude.

There are three loops. One measures the miss rate, one repairs individual misses, and one turns the output of both into training data. Because they fail in different ways they are sized from different quantities: the audit loop from statistical power — the sample size needed to detect a difference you care about — the appeal queue from the post-association miss rate rather than the detector’s own, and the label loop from nothing at all, because it only consumes what the other two produce.

1. The audit loop (proactive, measures). This loop draws a daily sample — stratified by slice cell, with a chosen rate per cell so that rare risky cells are still represented in usable numbers — and has humans check it for missed faces. Size it from what you need to be able to detect:

goal: detect a worst-cell miss rate of 1.5 % against a 0.5 % target,
      with 80 % power

   p_bar  =  (0.005 + 0.015) / 2  =  0.01     the average of the two rates
   delta  =   0.015 - 0.005       =  0.01     the difference to detect

   n ≈ 16 · p_bar(1 - p_bar) / delta^2
     = 16 × 0.01 × 0.99 / (0.01)^2   =  1,584 faces per cell

   (the 16 is the usual constant for 80 % power at 5 % significance:
    2 × (1.96 + 0.84)^2 = 15.7, rounded up)

at 3.2 faces per panorama, for a cell that is 4 % of traffic:
   1,584 / (3.2 × 0.04)  =  12,375 panoramas sampled per cell per readout

At 20,000 audited panoramas per day, the two or three riskiest cells read out daily and the rest weekly. Cost: 20,000 × $1.47 = $29,400/day if you scan whole panoramas. Note that scanning every tile instead is more expensive, not less — 128 × 6 s = 768 s against 240 s for the panorama. The saving is in the sampling, not the unit: draw tiles with a known, cell-dependent probability and reweight, so you stop spending 96% of a whole-panorama sweep on faces outside the cell you are trying to measure.

2. The appeal loop (reactive, fixes and labels). Anyone can report an unblurred face, their own or someone else’s, plus standing requests to blur a whole building or vehicle.

Size it from the shipped miss rate, which is the post-association one (The metrics to actually report), not the detector-only 1.43%:

suburban panorama miss, post-association
   1 - (1 - 6.1e-5)^3.2                    =  0.0195 %
   × 2e9 panoramas                         =  390,000 containing a miss
   × P(reported) = 0.008                   =    3,100 reports / year
   / 365                                   =      8.6 reports / day
   × 90 s of review each  =  774 s         =  13 minutes/day

Missed faces do not staff this queue — multi-view association already removed 98.6% of them. Run the same arithmetic on the detector-only 1.43% and you get 28.6M affected panoramas, 627 reports/day, 15.7 review-hours/day and two full-time reviewers: the association work in Multi view association is worth more than any detector improvement pays for itself a second time, in headcount. What the queue actually holds is the other traffic — standing requests to blur a building or a vehicle, over-blur complaints, and reports about content the detector was never asked to find — and those volumes come from product policy, not from the miss rate. Size the staffing from them.

The SLA — service level agreement, the response time you commit to publicly — is the design decision either way: same-day blur application, with the blur applied optimistically before verification rather than after, because the same cost asymmetry that shaped the threshold in Deriving the operating point and why the naive derivation fails applies to the review queue too. Blurring something that turned out not to need it costs a patch of wall; waiting costs another day of exposure.

3. The label loop. Every confirmed appeal is a hard positive that the model missed, geolocated and timestamped. Feed them into the next training set — but do not let them become the eval set, because appeals only come from populated, connected, rights-aware regions, and gating on them would optimize for exactly the places that already work.

All three loops in one picture. The arrow to watch is the dotted one at the bottom right, marked NEVER.

flowchart LR
    PUB[("Published imagery")] --> REP["User report<br/>~9/day from misses<br/>+ policy requests"]
    PUB --> AUD["Stratified audit<br/>20k panoramas/day<br/>weighted by slice risk"]
    REP --> Q[["Review queue<br/>90 s each"]]
    Q -->|confirmed| FIX["Blur applied<br/>same day"]
    Q -->|confirmed| HN[("Hard-positive pool")]
    AUD --> SLICE["Per-cell miss rate<br/>-> the launch gate"]
    AUD --> HN
    HN --> TRAIN["Next training set"]
    SLICE --> TRAIN
    HN -.->|"NEVER"| EVAL[("Eval set")]

    style Q fill:#bc6c25,color:#fff
    style SLICE fill:#2d6a4f,color:#fff
    style EVAL fill:#9d0208,color:#fff

The diagram shows both loops feeding one pool and one deliberate dead end. Published imagery is the source of both: on one side users report roughly 9 misses a day plus policy requests, and on the other a stratified audit of 20k panoramas/day weighted by slice risk samples it proactively. Reports enter a review queue at 90 s each; confirmed ones both trigger a same-day blur and drop into the hard-positive pool. The audit produces the per-cell miss rate that serves as the launch gate, and also contributes to that pool. The pool and the gate both feed the next training set. The dotted arrow marked NEVER is the important one: the hard-positive pool must not reach the eval set, for the reason loop 3 gives.

11. Alternatives considered and rejected

Every choice above had a plausible competitor, and a design is only defensible if you can name what you did not build and why. Each rejection below is quantitative rather than a matter of preference.

12. Interviewer pushback

What remains is the pushback: the hardest challenges to the design, each answered the way you would say it out loud. Use them as a self-test — any answer that surprises you points at the section worth rereading.

“How do you pick the blur threshold?” Testing: whether you derive an operating point or assert one. I start from the cost ratio and then notice it fails. A missed face is about $8 of expected cost — 62% chance it is an identifiable person, 1.3% chance it is noticed and reported, $1,000 per report including amortized regulatory exposure. A spurious box is about $0.0009. That gives t* = 1.1e-4, and at that threshold the detector emits 4,100 boxes per panorama and blurs 43% of the image.

The formula is right and the input is wrong. C_fp is not linear in box count, it is convex in total blurred area, because blur goes from invisible to annoying to disqualifying as it accumulates. I measure that curve — the unusable rate goes as roughly area to the 1.4 — and the true cost minimum lands at t = 0.005 and 3.2% coverage.

Product will not accept 3.2%, so the shipped point is a constrained optimum: maximize recall subject to area at most a 0.5% cap, which is t = 0.035 and 99.55% recall, spending 0.40% of the image.

I keep the cap and the spend separate on purpose. The gap between them is what pays for the 15% box dilation, which is 32% more area and would put 0.40% over the cap at 0.53% — and that is why §7 moves the threshold to 0.045.

The important sentence is that the binding constraint is the area budget, not the legal cost. The cost model still prices that budget: at 0.5% the miss cost is $0.116 a panorama and at 1% it is at most $0.064, so relaxing the cap is worth at least $103M across 2 billion panoramas.

“99.55% recall sounds great. Are you done?” Testing: whether you know per-face recall is the wrong unit. No, and the compounding is the reason. A suburban panorama has 3.2 faces, so 1 - 0.9955^3.2 is 1.43% — 28.6 million published panoramas a year with an unblurred face. A dense urban panorama has 40, so it is 16.5%: one in six. To hold a dense panorama at 1% I would need per-face recall of 99.975%, which is 18x better and is not reachable by threshold tuning inside the area budget.

So the recall has to come from redundancy, not the detector. The vehicle already captures each pedestrian from four to six positions, and the rig already produces pose and depth for stitching, so I project detections across frames and blur the union.

The projection is a gate, though, not an answer. A one-metre depth error at twelve metres swings the reprojected bearing by two hundred and thirty pixels, four times the width of the face it is supposed to cover. So I back-project both frames’ detections to 3D, match anything within a metre and a half — a pedestrian walks about a metre between triggers and two sigma of depth noise is half a metre — and blur the matched frame’s own box, which costs no extra area because it is a real detection. Only when the gate is empty do I blur the reprojected quad, and that path fires on 0.0026 boxes per panorama, a six-hundredth of the area budget.

I also cap propagation at score 0.5, because a false positive projects just as well as a true one, and unioning everything would take fourteen spurious boxes per panorama to fifty-six.

Misses correlate across views — a nine-pixel backlit face is hard in all of them — so I measure the joint miss rate rather than assuming independence: 6.1e-5 across four views against 0.0045 single-view, which is about 1.8 effective independent views. That takes the dense panorama from 16.5% to 0.24%, and it costs nothing, because those frames were already captured and already scored.

“Your mAP dropped 7 points. Explain.” Testing: whether you take a standard metric at face value. mAP is the wrong headline for this problem in three specific ways. It integrates precision across all recall levels and I operate at 0.9955, so the region I care about is half a percent of the curve. COCO mAP averages IoU 0.5 to 0.95, which prices tight localization — and here a loose box is free, because I dilate every box 15% before blurring anyway, while a tight box that clips an eye is a failure. And it is one number across all object sizes when 62% of misses live in the 8-16 pixel band. I have the pairing: model A is 0.612 mAP with 18.8 misses per thousand, model B is 0.541 mAP with 4.2. B is 4.5x safer and loses on mAP. I gate on misses per thousand at the shipped threshold, per cell of the slice grid, plus panorama-level miss rate after association, and I keep [email protected] as a regression tripwire.

“Why are small faces so hard? Just train more.” Testing: whether you can find the mechanism. More data does not fix it, because the assignment rule is throwing the examples away. A face at 30 metres is 11 pixels — 37 px per degree of angular resolution and a 16 cm face. With standard anchors starting at 32 px, a centered 11 px box has IoU 121/1024 = 0.118 with its best anchor, which is below the 0.4 ignore band, so it is assigned as a negative and the classification loss actively trains the model to say background right where the face is. The effective detection floor lands at 23 px, a face at 15 metres. Anchor-free assignment with center sampling removes the failure entirely, and removes the tuning surface with it.

Separately, the face has to be resolvable in the feature map. At stride 16 an 11 px face is 0.69 cells across, so it is a sub-cell signal averaged into whatever else is in that cell. That forces a stride-4 level. A stride-4 level at 1024 squared is 65,536 cells against 21,760 for everything else, so a full-width head there costs 619 GFLOP and triples the model. I use a depthwise-separable 128-channel head at P2 for 9 GFLOP instead, which is defensible because the discrimination task at P2 is genuinely easier.

“There is a recall gap by skin tone. What do you do?” Testing: whether fairness is a measurement design or a sentiment. First, measure it correctly, because the marginal table lies. Marginally the gap between the lightest and darkest Monk bands is 1.24 points. Crossed with illumination it is 0.12 points in daylight and 2.47 points at night. It is an interaction, and averaging over lighting understates the worst cell by half and points at the wrong fix.

The mechanism is physical. The camera auto-exposes for a scene dominated by sky and road, so in low light a darker face lands in the bottom two stops of the sensor range and its contrast against the background compresses toward the noise floor.

So the fixes are exposure and gamma augmentation matched to the real capture exposure distribution, slice-targeted labeling of the four worst cells — about 40,000 boxes and $2,400 — and local tone mapping on tiles with crushed shadows. That takes the night gap from 2.47 to 0.44 points. And structurally, the launch gate is the worst cell of the grid, not the mean and not the worst marginal.

Part of it is not a model problem at all. At a 1/125 s dusk exposure and 40 km/h, a face at five metres smears 37 pixels, which destroys a 68 px face. That is fixed by capping exposure at capture, not by training.

“Walk me through your train/test split.” Testing: whether you can find the leak your own architecture creates. It cannot be random at the tile level, and the reason is the same fact multi-view association is built on: the same pedestrian appears in four to six consecutive frames of one capture run. With five views and an eighty-ten-ten tile split, 1 - 0.2^4 is 99.84%, so essentially every face in the test set has a sibling view in training — the model has already seen that face, at a slightly different size and exposure.

Measured, that reads 1.3 misses per thousand against 4.5 on a run-level split. The headline metric would be optimistic by three and a half times, and nothing in the training loop would report an error.

So I split by capture run, which is also the key the pipeline shards on. On top of that I hold out whole cities, because two runs down the same street a month apart share shopfronts and parked cars, and I freeze the last six weeks as a temporal holdout so a rig or firmware change shows up as a regression instead of being averaged across the split.

And the eval labels come only from the blind five-percent control, never from the model-assisted pool, because assisted labels are nine percent incomplete in exactly the hard band.

“What does this cost to run?” Testing: whether you can size a batch pipeline and whether you know what actually binds. Per panorama: a 13,312 by 6,656 equirect at tile 1024 and overlap 192 is stride 832, so 16 columns by 8 rows — 128 tiles. A content gate at 0.6 GFLOP each, and the detector at 410 GFLOP on the 39% of tiles the gate keeps, which is 20.5 TFLOP.

Across 2 billion panoramas that is 4.1e22 FLOP, about 38,000 GPU-hours at 300 TFLOP/s effective, and at $2.50 a GPU-hour that is $95,000 a year. Decode is 0.74 CPU-seconds each — 411,000 CPU-hours at $0.035 a CPU-hour, another $14,400. Re-encode is 25,000 CPU-hours, only $880, and only because 84% of panoramas have zero detections and pass their original bytes through untouched while the rest re-encode a 15% crop. Inverting that, re-encoding every pixel of every panorama would be $36,700.

Total is about $110,000 a year, or $55 per million panoramas. Which means cost is not the interesting constraint. The binding constraint is I/O: 36 petabytes in and 36 out, and over a 30-day window that is 13.9 GB/s sustained each way for a month. That sizes the storage tier and the shard layout, and it is the number I would plan against.

“Just blur everything that might be a face. Why not?” Testing: whether you understand the constraint you are optimizing against. That is what the naive cost ratio recommends, and I ran it: t = 1.1e-4 blurs 43% of the image. Even the true cost optimum at t = 0.005 blurs 3.2%, which puts visible grey patches on most facades and makes the imagery product substantially less useful. The unusable rate goes as roughly area to the 1.4, so it is 0.4% at half a percent coverage and 19% at five percent.

There is also a second-order cost people miss: over-blurring destroys the storefront signage and address numbers that make the product navigable, so the complaints are concentrated exactly on the tiles with the most commercial value.

The right framing is a constrained optimization with the area budget as the constraint. Then the real work is buying recall from somewhere other than the threshold — multi-view association, a P2 level, exposure augmentation, and box dilation all buy recall at zero or near-zero area cost.

“Your blur was reversed by a researcher. How?” Testing: whether you know redaction is not the same as attenuation. Because Gaussian blur is a convolution and its transfer function is a Gaussian, which is nonzero at every frequency — so Wiener deconvolution recovers everything above the noise floor. At 40 dB SNR and sigma 2 pixels, that is content down to a 4-pixel period, and an eye on a 68-pixel face is about 14 pixels across — recoverable with a factor of three to spare.

The fix is a primitive that is many-to-one rather than merely lossy: decimate the region to block means, which has no inverse, then add block-level noise and requantize through the JPEG encoder.

The trap is specifying that as a fixed eight-by-eight grid. Sixty-four means from an eleven-pixel face is a 1.9-to-1 reduction, and from an eight-pixel face it is the identity function — so the redaction would be weakest on the size band that carries sixty-two percent of the misses. The rule instead is a block size floor with a block-count cap: six pixels minimum, eight blocks maximum, which guarantees at least a thirty-six-to-one sample reduction at every face size.

The noise step matters more than it looks, because the same face appears in four to six published frames and averaging independent redactions reduces noise by root-k. So the noise has to be correlated across frames of the same identity, not sampled fresh per frame.

Next: 04 — Video Search — where the unit of retrieval stops being an object and becomes an interval, and the index has to represent time.