How to design a face generator

Generate one 512x512 photorealistic face of a person who does not exist; the constraints (consent, misuse, provenance) settle the design before the network does, and the pick is latent diffusion.

Interface and constraints

• Input: optional attribute condition (age band, pose, lighting, expression), or nothing (sample freely). Output: one 512x512 face.
• Biometric data, deepfake raw material, and memorization (an output reproducing a training photo, i.e. that person’s photo laundered) drive the design first.
• Six mechanisms: provenance rows + model releases (consent), retrain-on-erasure, high-recall age classifier at ingest (hard reject), ArcFace cosine block vs public-figure + training index, C2PA signed manifest, invisible watermark.
• Erasure = drop rows/shards + retrain; you cannot subtract an image from trained weights.

Watermark: what it proves

• Bit accuracy = fraction of ~48 hidden bits recovered; 1.0 perfect, 0.5 = mark gone. Not a detection rate.
• Attribution decision is governed by FPR (binomial over 48 fair coins): FPR(k) = (sum_{i=k}^{48} C(48,i)) / 2^48.
• Corpus scan needs FPR <= 1e-9 => 44/48 bits => bit accuracy >= 0.917. Below that is not attribution.
• Survives trained distortions (JPEG, resize, crop); dies on img2img (0.61), autoencoder round trip (0.55), adversarial (0.50).
• Absence proves nothing. Watermark attributes your output; it is not a deepfake defence. Real control is upstream: refuse identity-targeted requests, rate-limit + verify the API, log every generation.

Objective = a choice of divergence

• Learn a sampler making p_model close to p_data; “close” = a divergence (zero when identical, asymmetric).
• Forward KL (average over DATA): mode-covering, infinite penalty for missing mass => covers gaps between modes => blur.
• Reverse KL (average over MODEL): mode-seeking, missing a mode is free => sharp but drops modes => mode collapse.
• For faces a dropped mode = demographic dropout = fairness/legal incident FID cannot see. Points to the maximum-likelihood family before any layer is drawn.

Data

• Scale: 2-5M aligned faces for 512x512; beyond that, curate not add.
• Alignment (5 landmarks + similarity transform) buys detail but shrinks support: off-center requests decode to warped centered faces.
• Dedup is a safety control: pHash + embedding cluster at cosine 0.95, cap 1. Duplicate count enters the loss linearly => memorization.
• Audit demographic coverage against a target marginal set by policy, not the corpus. Measure with TV distance = half the sum of absolute per-bin differences. FID rewards reproducing your reference set’s skew, so it cannot see the bias.

Diffusion mechanics

• Forward: x_t = sqrt(abar_t)*x_0 + sqrt(1-abar_t)*eps; train L = ||eps - eps_hat||^2. Target is fixed (RNG noise), so no game: one number goes down.
• Predict noise, not x_0: constant loss on eps = loss on x_0 reweighted by SNR_t = abar_t/(1-abar_t). Spends capacity on low-noise (high-SNR) steps where the mean is nearly one point => no blur. High-noise steps decide layout/pose/identity; low-noise steps decide pore texture (cut steps from the low-noise end).
• CFG: eps_guided = eps_u + w*(eps_c - eps_u). w=1 = plain conditional; w>1 overshoots (exponent on implied classifier p(c|x)), trading diversity for fidelity. FID optimum ~1.5, humans prefer ~3 (proof FID is not the objective). One global w couples quality to demographic coverage.

Training and cost

• Backbone DiT-XL/2 in f=8 latent: 512/8 = 64x64x4 -> 2x2 patches -> 1,024 tokens. ~675M params (445.9M token-scaled + 223M adaLN-zero).
• Recipe: EMA weights (0.9999, worth 3-5 FID), zero terminal SNR + v-prediction (else luminance collapse), horizontal-flip-only augmentation, 10% condition dropout (guidance + memorization control), bf16 not fp16.
• ~$13k GPU (~3.4 days on 64 H100s at 300 TFLOP/s effective, ~30% of peak); ~$50k end to end. Attention is quadratic but only the bill at 4x resolution (crossover at 6d = 6,912 tokens).

Metrics: never FID alone

• FID = Frechet distance between two Gaussians fit to Inception-v3 features. Flaws: first two moments only, blind to per-sample defects, conflates fidelity+coverage, upward-biased on few samples (never compare across N), ImageNet features (no faces), rewards reference-set skew.
• Split into precision (fidelity, generated inside real region) and recall (coverage). Same FID can hide mode collapse.
• Face-specific: identity diversity (ArcFace pairwise cosine distribution), attribute-marginal TV distance vs target, memorization rate (SSCD tail, not mean), symmetry defect rate. Report the tail/distribution, never the mean.
• Human 2AFC: 500 pairs x 5 raters, report fooling rate (target 50%), defect taxonomy, Krippendorff alpha (<0.6 = ambiguous taxonomy). ~$900, gates releases not commits.
• Online A/B: primary accept-on-first-batch; randomize by user; fairness gap > 3 pp is ship-blocking guardrail. n = 16*p(1-p)/delta^2; run 7 days for the weekly cycle.

Serving path

Request -> Auth + rate-limit -> POLICY CHECK (before GPU) -> Queue
  -> Batch cond+uncond in one forward -> DiT 30 DDIM steps, CFG w=3
  -> VAE decode -> Safety cascade (NSFW/minor, public-figure ArcFace,
     training-index memorization) -> Watermark -> C2PA sign -> CDN 24h

• Policy check runs in plain code before the queue: cannot be argued out of running, and never makes an internal copy of the content it blocks.
• Memorization check holds the ~3M-vector (~3.1 GB) HNSW training index in-process; affordable only at this scale.
• On a hit, resample with a new seed (memorization is a seed property); 3 strikes => refuse + alert.

Cost levers (rank)

• Latent vs pixel space is 8,600x (pixel-space at 512 = ~30 min/image); everything else is under 15x: distillation 13.7x (60 -> 4 steps), step count 8.3x, guidance 2x.
• Full design ~$0.00024/image, ~$489/day at 2M/day.

Memorization, the one legal failure

• Denoiser converges to the posterior mean = softmax-weighted average of training images; low noise + high duplicate count k_i sharpen the softmax onto one image => that photograph, not a face like it.
• Controls in order: dedup at ingest (kills the k_i multiplier), 10% condition dropout, output-side NN check on the distribution tail. Nothing drives it to zero, so the training index is a permanent serving dependency.