How to design personalized headshots

What this system is about

In this lesson, we’ll design a system that trains a small piece of a neural network on one specific person’s face, then generates new pictures of them. By the end you’ll be able to pick a personalization method from a latency and storage budget, price one customer end to end, and defend the deletion path for a per-user model. Image generation itself is covered in the text-to-image chapter; this one restates only what it needs. The distinctive constraint is that there is now one model artifact per customer, which ties quality, cost, storage, and privacy into a single problem.

• In: 10-20 selfies uploaded from a phone, plus consent to train on them.
• Out, about twenty minutes later: roughly 40 images of that person in professional-headshot settings (studio lighting, a neutral backdrop, business attire, several poses) recognizable to their colleagues.
• In between: a small per-user file of learned weights that did not exist before they uploaded, and must not exist after they ask you to delete it.

Background from image generation

One line each; each term reappears in context below.

• Latent diffusion model: the image generator. It works in a compressed space instead of on pixels directly.
• Variational autoencoder (VAE): compresses an image into a small grid of numbers called a latent, and expands a latent back into pixels.
• Denoiser: a large network that looks at a noisy latent and predicts the noise in it.
• Sampler: the loop that runs the denoiser about 30 times, subtracting a little predicted noise each round, until a clean latent falls out.
• Text encoder: turns the prompt into a sequence of vectors.
• Cross-attention: how the prompt reaches the denoiser: at every layer, each spatial position of the image asks the text what should be drawn there. This is the mechanism every adapter here attaches to.
• Classifier-free guidance (CFG): runs the denoiser twice per step, once with the prompt and once with a blank one, then extrapolates away from the blank to make the image follow the prompt harder. Its strength is a dial w. It doubles the compute per step, and too high degrades the image.

Two results from that chapter are reused: generation costs about 26.8 trillion floating-point operations per denoiser pass, and guidance above about 4.5 starts dragging the face back toward the base model’s generic one.

The one question everything collapses onto

When every user needs their own model, everything reduces to: how many parameters does a user get, and who pays to store them?

The design is a fine-tuning ladder, a ranked set of ways to specialize an already-trained model, from “retrain everything” down to “change nothing and pass in an extra vector.” Two words get used constantly:

• Fine-tuning means continuing to train an existing model’s own weights on new data. You end with a whole new model.
• An adapter means leaving every original weight frozen and adding a small number of new parameters alongside them. You end with the original model plus a small file.

That distinction is the entire economics: a fine-tune is 5.2 GB per customer and cannot be shared between customers; an adapter is 42 MB and can.

Two conventions for the numbers

Storage is quoted in fp16, 16-bit floats, two bytes per parameter. That is the only conversion between a parameter count and a file size here: a 21.0M-parameter adapter is 21.0e6 · 2 = 42 MB.

Speed is quoted as MFU, model FLOPs utilization, the fraction of a card’s peak arithmetic you actually achieve. An H100 peaks around 990 TFLOP/s on the 16-bit matrix math these models use, so 22% MFU is ~218 TFLOP/s, 40% is ~396, and 48% is ~475. (The text-to-image chapter quotes the same card as “300 TFLOP/s effective,” which is about 30% MFU. The whole argument here turns on moving the MFU, so it is split out.)

Problem framing

• Input: 10-20 selfies of wildly variable quality.
• Output: ~40 images in professional-headshot settings.
• Constraints: identity recognizable to the subject’s colleagues; turnaround inside a session; per-user marginal cost (what one more customer costs, ignoring fixed costs) a small fraction of a ~$29 price.
• Why it is hard: the model must learn a new concept, this person’s face, from ~15 examples without learning the 15 photos.

That last point is the whole difficulty. Gradient descent, the procedure that repeatedly nudges weights in whichever direction lowers the loss, treats “learn what this person looks like” and “memorize these 15 pictures” as the same operation. The loss has no term that prefers one over the other, as the next section shows.

The two quality axes

• Identity preservation (“identity”): the generated face is recognizably the same person as the uploads.
• Prompt following (prompt adherence): the image contains what the prompt asked for: the suit, the studio backdrop, the mountain trail.

These pull against each other along every training knob. Train harder and identity goes up while prompt following goes down. Choosing where to sit on that trade is the product.

Three reframes

Three terms in that table:

• Monotonically means “in one direction, always.” More training steps never lower identity and never raise prompt following.
• The knee of a curve is the bend past which you pay a lot for a little.
• Biometric-derived data is data computed from someone’s body, here from their face. In most jurisdictions it carries stricter consent, retention, and deletion rules than ordinary personal data. Under-classify it and an artifact you kept is an unlawful retention; over-classify it and you cannot ship in a region you thought you could.

ML objective

Personalization gets no special objective. It is the ordinary denoising loss (from the text-to-image chapter), restricted to a tiny dataset and a subset of the parameters:

D_user = { (x_i, "a photo of <tok> person") },   i = 1..15
theta_trainable  subset of  theta_base

loss = E_{i, t, eps} || eps - eps_theta(z_t^i, t, c_i) ||^2

• The dataset D_user is 15 pairs of a photo x_i and a caption. Every caption uses one template, "a photo of <tok> person". <tok> is a trigger token, a rare, otherwise-meaningless string (often something like sks) chosen because the base model has no prior associations with it, so it is free to become a handle for this person.
• What may move is theta_trainable, the subset of base weights you allow to change. Which subset that is is the entire ladder.
• The loss corrupts the user’s image latent with noise eps at a random timestep t, and trains the denoiser eps_theta to predict that noise given the caption. || ... ||^2 is squared error; E averages over images, timesteps, and noise draws.

Nothing here is personalization-specific. The personalization is entirely in what data you feed and which weights you unfreeze. So two things are absent, and both become failure modes:

• No term says “learn the face, not the room.” Any weight change that lowers reconstruction error on those 15 images is rewarded equally, whether it encodes bone structure or wallpaper.
• No term preserves the base model’s behavior. The optimizer is free to overwrite general knowledge, catastrophic forgetting, where training hard on a narrow task destroys competence on everything else. Everything that stops it (prior preservation, low rank, early stopping) is a regularizer: an extra constraint bolted on to steer training away from a behavior you do not want. Choosing among them is the design.

The personalization ladder

There are five ways to give a model a new face. The savings do not come from where people assume, and the two cheapest have hard limits.

The backbone every number is computed against

The reference backbone throughout, the same as in the text-to-image chapter, is a 2.6B-parameter DiT, a Diffusion Transformer, meaning the denoiser is built from transformer blocks. Its size numbers are hidden width d = 2048 and 40 blocks. Each block holds 8 square d × d matrices used by attention (4 for self-attention: query, key, value, output; and the same 4 for cross-attention), so:

8 matrices · 40 blocks = 320 matrices
320 · 2048^2  =  1.34B parameters

Attention alone is 1.34B of the model’s 2.6B parameters, spread over 320 matrices. Those 320 matrices are what every adapter here attaches to. Nothing else gets touched.

The five rungs

• Full fine-tune: update all 2.6B weights. No constraint, no protection.
• DreamBooth: a full fine-tune plus a prior-preservation loss: an extra term that trains simultaneously on generic images the base model produced itself, so the general meaning of “person” is not overwritten by this one person.
• LoRA (Low-Rank Adaptation): freeze every original weight, and learn a small additive correction to each matrix. The rank r controls how small.
• Textual inversion: change no network weights at all; learn a few brand-new word vectors that stand for this person.
• Encoder-based identity adapter: train nothing per user. Take a face-recognition vector computed from the selfies and inject it into cross-attention through layers trained once, offline, across millions of identities.

Three columns need a note. Storage / user is the trainable-parameter count times 2 bytes (fp16), the only conversion in that column. Train time is solo, one user’s job alone on a card, for a like-for-like comparison; measured that way LoRA is 374/287 = 1.3x faster than a full fine-tune, not 3x. ArcFace cos runs the ArcFace face-recognition model over the generated face and the reference selfies and takes the cosine similarity between the resulting vectors (1 for identical directions, 0 for unrelated); its anchors come later.

The last row is the exception, and the interesting one: the encoder-based rung trains zero parameters per user, so it has no adapter to store. Its 1 KB is a 512-dimensional face embedding computed at request time, and if you never write it to disk, the per-user storage is genuinely nothing.

Lining the storage column up against the identity column exposes the gap between them:

storage:   5.2 GB  ->  42 MB  ->  32 KB          spans ~160,000x
identity:  0.73    ->  0.68   ->  0.44           spans 1.7x

Storage spans five orders of magnitude; identity spans a factor of 1.7. That asymmetry is the entire argument, and it is why nobody ships full fine-tunes.

Choosing a rung

We ask three questions in priority order. Latency first, because it is a hard product constraint: if the product promises ten seconds, no quality argument matters. Then whether identity is what users actually complain about. Then what the storage budget will bear.

flowchart TD
    Q1{"Is per-user latency<br/>under 10 s required?"} -->|yes| ENC["Encoder-based adapter<br/>0 params/user · 1 KB<br/>identity 0.52"]
    Q1 -->|no| Q2{"Is identity the<br/>top complaint?"}
    Q2 -->|no| TI["Textual inversion<br/>32 KB/user<br/>identity 0.44"]
    Q2 -->|yes| Q3{"Fleet storage budget<br/>at 50k users/day?"}
    Q3 -->|"tight"| LR4["LoRA r=4 cross-attn<br/>5.2 MB/user<br/>identity 0.58"]
    Q3 -->|"normal"| LR16["LoRA r=16 all-attn<br/>42 MB/user<br/>identity 0.68"]
    Q3 -->|"unbounded"| DB["DreamBooth full weights<br/>5.2 GB/user<br/>identity 0.73<br/>NOT BATCHABLE"]

    style ENC fill:#2d6a4f,color:#fff
    style LR16 fill:#1d3557,color:#fff
    style DB fill:#9d0208,color:#fff

The two LoRA leaves differ only in where the adapters attach: r=16 all-attn puts rank-16 adapters on all 320 attention matrices (42 MB, identity 0.68), while r=4 cross-attn puts rank-4 adapters on the 160 cross-attention matrices only (5.2 MB, identity 0.58). The red box carries a warning the table does not: full weights are not batchable, two users cannot share a forward pass, which costs about 1.9x per image, shown later.

LoRA’s saving: 2r/d

A weight matrix W (a d × d grid) is not changed directly. You add a low-rank correction dW = B · A, with B at d × r and A at r × d. Rank is the number of independent directions a matrix can express; forcing rank r means dW factors into a tall skinny matrix times a wide skinny one, and you store only those two factors, never the big dW.

parameters in dW (full)   =  d^2    =  4,194,304
parameters in B and A     =  2·d·r  =     65,536
ratio  =  2·d·r / d^2  =  2r / d  =  32 / 2048  =  1.5625%

2r/d is the whole formula. The saving depends only on rank relative to model width, not on how many layers you touch, because the ratio applies to each matrix independently. Rank 64 would be 2·64/2048 = 6.25%, with no further work.

Across all 320 matrices, 1.34B trainable params become 21.0M at r=16, 42 MB in fp16, against 5.2 GB for a full fine-tune (124x). Variants: r=4 all matrices is 10.5 MB, r=4 cross-attention only is 5.2 MB, r=32 all is 83.9 MB.

The saving is memory, not FLOPs

The common misconception is that training 0.81% of the parameters makes training ~100x cheaper in compute. It does not.

You still push every image through the whole 2.6B model, every frozen matrix still multiplies, and you still backpropagate through every layer, because that is the only way the training signal reaches an adapter down in layer 3. Freezing a layer does not let you skip it. The only thing you skip is computing the weight gradients for the frozen matrices. A backward pass computes gradients with respect to inputs (needed to keep going backwards) and with respect to weights (needed to update them); LoRA drops the second for 99.19% of the weights.

As an order of magnitude: a full fine-tune costs about 3x the forward pass (forward, input gradients, weight gradients); LoRA costs about 2.3x (it keeps the forward and input gradients and pays only a sliver for the few weight gradients it needs). That is roughly 13.0 vs 17.0 TFLOP per image, 23% cheaper, not 100x.

What LoRA actually buys is memory, and memory buys batching. Training needs four things resident in the card’s memory:

1. The weights, 2 bytes per parameter (fp16).
2. The gradients, 2 bytes per trainable weight.
3. The optimizer state. Adam keeps two fp32 running averages per trainable weight, 8 bytes each.
4. A master copy of the trainable weights in fp32 (4 bytes), because accumulating tiny updates in 16-bit loses them to rounding.

Items 2-4 are sized by the trainable count; item 1, the only one sized by the total, is also the only one that can be shared between two users’ jobs on the same card, because for LoRA it is frozen and identical for everyone.

                    weights   grads    Adam (fp32)   master (fp32)   total
full fine-tune      5.2 GB    5.2 GB     20.8 GB        10.4 GB      41.6 GB
LoRA r=16           5.2 GB   0.04 GB     0.17 GB        0.08 GB       5.5 GB
                    (frozen, SHARED across co-resident jobs)

On an 80 GB card, the full fine-tune’s 41.6 GB of state plus ~12 GB of activations leaves room for 1 job. LoRA pays the 5.2 GB base once, then each job costs only ~2.8 GB (0.29 GB adapter state + ~2.5 GB activations), so (80 − 5.2) / 2.8 ≈ 26 concurrent jobs. (Activations are the intermediate forward values the backward pass reads back; they scale with batch size, not parameter count.)

More jobs per card matters because of MFU. At small batch the card spends most of its time waiting for weights to arrive from memory instead of computing; raising the batch spreads each weight read over more work, so the achieved rate more than doubles:

batch 4                          MFU ~22%   ->  218 TFLOP/s
batch 32 (= 8 users co-batched)  MFU ~48%   ->  475 TFLOP/s

per user: 1,200 steps · 4 images · 13.0 TFLOP  =  62.5 PFLOP
alone   62.5e15 / 218e12  =  287 GPU-s  =  $0.199   (4.8 min wall clock)
8-way   62.5e15 / 475e12  =  132 GPU-s  =  $0.092   (17.5 min wall clock)

Batch 32 is eight users co-batched at batch 4 each: you are not raising one user’s batch, you are stacking eight users’ jobs into one forward pass through the shared frozen base. Alone, one job owns the card for 287 seconds; co-batched, each job’s share is 132 GPU-seconds but eight jobs interleave, so the user waits ~17.5 minutes.

Co-batching eight users cuts per-user cost 2.2x and multiplies latency 3.7x. That is a tiering decision with the arithmetic done: free tier co-batched at 20 minutes, paid tier dedicated at 5 minutes, instant tier on the encoder-based adapter at 10 seconds. Full fine-tuning cannot participate: there is no frozen base to stack jobs on.

Textual inversion and encoder-based, honestly

Two rungs look nearly free, and each hits a structural ceiling.

Textual inversion changes no network weights. It optimizes only a handful of new embedding vectors, the lookup vectors the text encoder assigns to tokens, in effect inventing four new words that mean “this person.” With a text encoder of width 4096, that is 4 · 4096 · 2 bytes = 32 KB. It gets prompt following perfect by construction (the base is untouched, so there is no forgetting to defend against) and its file is a rounding error. But it converges slowly, the gradient still travels back through the whole network yet arrives at only 16k parameters, and it tops out at identity 0.44, because a single point in text-embedding space cannot express everything a face is. Use it for a style or an object; it is under-powered for faces.

Encoder-based identity adapters move personalization out of training entirely. Train once, offline: a projection, a learned linear map, from a face-recognition embedding into the space cross-attention reads, plus a few adapter layers, call it ~90M shared parameters trained across millions of identities before any customer exists. At serve time you run a face encoder over the selfies, average the embeddings, and inject that vector. Zero training, zero per-user storage, generation-only latency.

Identity lands around 0.52 against LoRA’s 0.68, and the ceiling is structural. A face-recognition model is built for invariance: it must return the same answer for the same person under different lighting, with and without glasses, with a new haircut, ten years apart. So glasses, hairstyle, lighting-dependent skin tone, and facial asymmetries, a large fraction of what a person looks like, are exactly what it was trained to throw away, and therefore what it cannot hand to the generator.

Ship both: encoder-based for the instant preview, LoRA for the delivered set. The preview converts the user; the LoRA satisfies them.

Per-user economics

With a rung picked, we price one customer from upload to delivery, then put that price next to revenue, where it turns out not to decide whether the business works. Prices: H100 at $2.50/GPU-hour, object storage (bulk cloud file storage such as S3) at $0.023/GB-month, and egress (the charge for data leaving the provider’s network) at $0.09/GB.

Generation is the larger GPU line item. Same backbone, but at 1024 pixels instead of the 512 used for training, so 4x the tokens and 26.8 TFLOP per forward pass. The sampler runs 30 steps and CFG runs the denoiser twice per step, so 30 steps is 60 passes. CFG is literally half the generation bill:

per image        60 · 26.8                =  1,608 TFLOP
batch 48, MFU ~40%                         =    396 TFLOP/s
                 1,608 / 396               =   4.06 s per image
48 · 4.06 = 195 GPU-s = $0.135

You generate 48 and deliver the best 40; the extra eight cover images the identity gate rejects (below).

The bill

Every line after “GPU at 100% utilization” is a multiplier on the line above it. QC is quality control, the automated checks a finished job must pass. A re-roll is a user asking for another set at no charge.

LoRA training      132 GPU-s  (8-way co-batched)           $0.092
generation, 48     195 GPU-s                               $0.135
restore + upscale to 2048px, 40 images                     $0.008
                                                           ------
GPU at 100% utilization                                    $0.235
fleet utilization 55%          ($0.235 / 0.55)             $0.428
4% of jobs fail QC and rerun   (· 1.04)                    $0.445
12% request a regeneration     (+ 0.12 · 0.246)            $0.474

adapter storage    42 MB · 90 days                         $0.0029
outputs            80 MB · 90 days                         $0.0055
source selfies     60 MB · 30 days                         $0.0014
egress             80 MB at $0.09/GB                       $0.0072
                                                           ------
                                          MARGINAL COST    $0.491

Fleet utilization is the second-largest multiplier: you rent the card by the hour, not by the second of work, so if the fleet is busy 55% of the time every real second carries the idle around it. A re-roll regenerates images but does not retrain the adapter, so it costs only the generation line at fleet utilization (~$0.246), times the 12% who ask.

Call it $0.49 per user against a $29 price, 98% gross margin (revenue minus delivery cost, over revenue). That sounds like the end of the conversation. It is not.

The honest part

Put $0.49 in a P&L (profit-and-loss statement) next to everything else that scales per sale.

price                                     $29.00
refunds and chargebacks, 9%               -$2.61
marginal compute and storage              -$0.49
paid acquisition (category CAC)          -$15.00
support, 6% of users at $4                -$0.24
                                          ------
contribution per user                     $10.66
compute as a share of contribution:  0.49 / 10.66  =  4.6%

Chargebacks are payments reversed by the customer’s bank; CAC is customer acquisition cost, the ad spend per buyer; contribution per user is what is left after every cost that scales with one more sale.

Now compare two quarters of work. Halving the GPU bill saves $0.245, 2.3% more contribution. Cutting refunds from 9% to 5% saves $1.16, 10.9% more contribution, roughly five times as much. And refunds here are almost entirely “it doesn’t look like me,” so the identity gate (below) is worth about five times any compute optimization. This is the same shape as the customer-support agent chapter, where the human cost dwarfed the model cost.

Why the ladder choice still matters

If compute is 4.6% of contribution, the ladder cannot be justified on per-user dollars. It is justified on three other things.

Fleet storage. At 50,000 new users a day with 90-day retention, steady-state stored volume is per-user size · 50,000 · 90:

full fine-tune   5.2 GB · 50,000 · 90  =  23.4 PB   ->  $538,000 / month
LoRA r=16         42 MB · 50,000 · 90  =   189 TB   ->    $4,347 / month
LoRA r=4 cross   5.2 MB · 50,000 · 90  =  23.4 TB   ->      $538 / month
textual inv.      32 KB · 50,000 · 90  =   144 GB   ->        $3 / month
encoder-based                      0   =       0    ->        $0

Full fine-tuning is $538,000 a month in storage while the per-user compute for those same users is a few cents each. 23 petabytes is not a line item. It is a data-center project, which is why full fine-tuning is off the table before any quality argument.

Serving topology, how the fleet is arranged, what lives on which card, what is shared. A shared base with unmerged adapters (kept as separate small matrices instead of folded into the base weights) lets one GPU batch a thousand different users through one matmul at a 1.56% overhead. Per-user full weights pin batch size at 1, which costs about 1.9x per image. (Derived below.)

Legal lifecycle. A per-user adapter is a model derived from biometric data: deletion requests must reach it, retention applies to it, and data residency rules (data about a region’s residents must be stored inside that region) apply too. Size sets how operable that is: a 42 MB artifact you can enumerate and drop is workable, a 5.2 GB one is 124x heavier, and a 1 KB embedding you never write to disk is the problem largely not existing. The compliance surface (the sum of places a regulated artifact can be found and must be tracked, secured, and deleted) is smallest for the encoder-based rung, a real argument for it independent of cost.

Multi-tenancy: thousands of adapters, one base model

One card serves many users at once. Multi-tenancy is one shared piece of infrastructure serving many independent customers without mixing them up. Three mechanisms make it work: hot-swapping adapters is nearly free, keeping adapters unmerged costs 1.6% and buys 1.9x, and a three-tier cache means the swap almost never happens.

The diagram has two independent entry points. The left branch runs once per user (upload to adapter in storage); the right runs many times per user (request to delivered images). They meet only at the adapter store.

• Face QC is the upload gate: is there a face, is it a live person and not a photo of a screen, are all 15 uploads the same person, and is that person not a public figure. Anything failing comes back with a stated reason.
• Training queue holds jobs until eight can be co-batched.
• Adapter store is object storage, one 42 MB file per user.
• Generation queue is drained by tier priority, so paid users are not stuck behind free ones.
• Adapter cache looks in three places in ascending cost: VRAM, then local NVMe, then S3.
• Sampler pool is the fleet of cards that generate images.
• Restore + upscale repairs faces and enlarges to 2048 pixels.
• Identity gate compares each output against the references; below threshold it drops and resamples, which is why you generate 48 to deliver 40.
• Safety + watermark is a final content check plus an invisible watermark, a pattern a detector can read and a viewer cannot see, so a generated image can later be identified as one.

flowchart TD
    U([Upload · 15 selfies]) --> QC{"Face QC<br/>detect · liveness ·<br/>all-same-person ·<br/>public-figure block"}
    QC -->|fail| REJ([Reject with reason])
    QC -->|pass| TQ[[Training queue<br/>co-batch 8 users]]
    TQ --> TR["LoRA trainer pool<br/>base frozen · shared<br/>26 jobs per 80 GB GPU"]
    TR --> AS[("Adapter store<br/>42 MB · user<br/>object storage")]

    P([Generation request]) --> GQ[[Generation queue<br/>priority by tier]]
    GQ --> CACHE{"Adapter cache<br/>VRAM -> NVMe -> S3"}
    AS -.-> CACHE
    CACHE --> SAMP["Sampler pool<br/>base resident 5.2 GB<br/>UNMERGED adapters<br/>batched LoRA kernel"]
    SAMP --> UP[Restore + upscale]
    UP --> IDG{"Identity gate<br/>ArcFace cos ≥ 0.45"}
    IDG -->|below| DROP[Drop · resample]
    DROP --> GQ
    IDG -->|pass| SAFE{Safety + watermark}
    SAFE --> D([Deliver 40])

    style QC fill:#bc6c25,color:#fff
    style SAMP fill:#1d3557,color:#fff
    style IDG fill:#2d6a4f,color:#fff
    style REJ fill:#9d0208,color:#fff

Hot-swapping is free

Hot-swapping means changing which user’s adapter is loaded onto the card without restarting anything. Three layers by ascending speed: object storage over the network (~1 GB/s), NVMe (the local SSD, ~5 GB/s), and VRAM reached over PCIe (the bus to the card, ~50 GB/s). A 42 MB adapter is 42 MB / bandwidth:

from object storage  =  42 ms
from local NVMe      =   8.4 ms
host RAM -> VRAM     =   0.84 ms
generation of one image  =  4,060 ms

Even a cold pull from object storage is 42/4060 = 1.0% of a single image’s generation, and a request generates 48, so the swap is 0.02% of the request. Swap cost is a rounding error relative to generation, which is what makes per-user models viable.

Capacity tells the same story. An 80 GB card with the base model (5.2 GB) and ~9 GB of activations leaves ~66 GB, which holds ~1,570 r=16 adapters, versus 13 full fine-tunes, because each is its own 5.2 GB model with no shared base to subtract. That is a 120x difference in how many users one card can hold ready.

Keep the adapters unmerged

Inference is running a trained model to produce output. At inference you can fold B·A into W, a merged model with zero arithmetic overhead. Do not. Unmerged costs an extra 2r/d = 1.56% (the down-project by A and up-project by B, on top of the shared W x, the same 2r/d as the parameter ratio, counting the same two skinny matrices). But merging bakes one user’s identity into W', so W' differs per user, so no two users share a matmul, so batch size is pinned at 1, where MFU is worst:

merged      batch 1 across users   ->  MFU ~25%  ->  247 TFLOP/s
unmerged    batch 64 across users  ->  MFU ~48%  ->  475 TFLOP/s
            475 / (247 · 1.0156)   =   1.89x in favour of unmerged

Paying 1.6% to keep the base matmul shared buys a 1.9x throughput win. This is what makes heterogeneous batching, requests from different users in one batch, the cheaper option, not just a possible one.

The code below writes that out: one big shared matmul for the whole batch, then a small per-user correction gathered onto only the rows belonging to each adapter. W appears once, outside the loop. That placement is the entire optimization.

def lora_forward(x, W, adapters, adapter_index, scale=1.0):
    """Batched heterogeneous LoRA. One shared base GEMM, per-adapter residual.

    x             (B, T, d)   activations for B requests from B different users
    W             (d, d)      frozen base weight, shared by every request
    adapters      list of (A, B_) pairs, A is (r, d), B_ is (d, r)
    adapter_index (B,)        which adapter each request in the batch uses
    """
    y = x @ W.T                                  # shared: 2*d*d FLOPs per token
    for slot, (A, B_) in enumerate(adapters):    # residual: 4*d*r FLOPs per token
        rows = [i for i, a in enumerate(adapter_index) if a == slot]
        if not rows:
            continue
        h = x[rows] @ A.T                        # down-project
        y[rows] = y[rows] + scale * (h @ B_.T)   # up-project
    return y

Production kernels do the loop as a grouped GEMM (many small matrix multiplies in one launch) instead of a Python loop, but the FLOP accounting is exactly the above. A GEMM is a general matrix-matrix multiply, the card’s core operation. The per-request adapter_index must never be wrong: a mis-routed row generates one user’s face on another user’s request (a biometric leak, not a quality bug), so it deserves an assertion and a test, not a comment.

Cache policy

User behavior is bursty: an initial batch of 40, a handful of re-rolls over the next twenty minutes, then nothing for months. Almost every access to a given adapter happens inside one twenty-minute window, which suggests three tiers:

• VRAM: LRU eviction (drop the least recently used) with a 30-minute TTL (time-to-live: dropped 30 minutes after arrival regardless of space).
• NVMe: the last 7 days (~15 TB at 50k users/day). Catches the “came back the next morning” case at 8 ms.
• Object store: everything else, with a lifecycle rule that deletes at 90 days and emits an audit record, because the artifact is biometric-derived.

The VRAM tier has room for ~1,500 but only ~17 are ever resident: 11 concurrent requests per GPU over 20-minute sessions is 33 arrivals/hour, and the 30-minute TTL holds half an hour of them. The TTL binds, not the capacity, which is the point. Every re-roll inside a session finds its adapter already in VRAM, so the hit rate sits above 0.9 by construction, with no cache size to sweep.

Two things must hold, and both are load-bearing beyond performance. Adapters from different users share a card and a batch, so the safety of that sharing rests entirely on the adapter index being correct. And the deletion path (below) must reach the VRAM and NVMe tiers, not just the object store: an adapter surviving in cache after the object copy is deleted is an undeleted biometric artifact that nothing in the system will flag.

Metrics: identity and prompt following as separate axes

Serving is solved; output quality is not. The measurement stack has three jobs: turn “does it look like them” into a number that means something, measure how it trades against prompt following, and run the one gate whose return is 19 to 1.

Measuring identity

1. Push the generated face and each reference selfie through a face-recognition model. ArcFace and AdaFace are the standard open ones. Each returns an embedding: a few-hundred-number vector trained so that two photos of the same person point in nearly the same direction.
2. Average the reference embeddings into one vector representing “this person.”
3. Take the cosine similarity between the generated embedding and that average.

That gives a number between 0 and 1 that, on its own, means nothing. Report it against the two distributions that define the scale, measured on the same encoder and pipeline:

different people, unconstrained photos       mean cos  ~0.02   (p99 ~0.22)
same person, two different photos            mean cos  ~0.65
verification threshold at FAR = 1e-4         cos        0.36

our LoRA outputs                             mean cos   0.68
our encoder-adapter outputs                  mean cos   0.52

FAR is the false accept rate, the fraction of different-person pairs a verification system wrongly calls a match; FAR = 1e-4 is roughly where phone face-unlock is set. Now 0.68 reads: above the same-person anchor of 0.65 and far above the 0.36 threshold, as similar to the references as two real photos of the same person are to each other. Quoting an identity score without those anchors is the same error as quoting a precision-recall PR-AUC without saying how rare the positive class was.

The tradeoff, measured

Identity and prompt following move in opposite directions along every knob. The measurements below come from a fixed panel of 30 users, rerun on every recipe so changes are comparable, not confounded. Columns: ArcFace cos (identity, higher better, anchored at 0.65 same-person and 0.02 different-people); VQA prompt adherence (decompose the prompt into yes/no questions, “is there a suit?”, and ask a vision-language model each, scoring the fraction correct); background-leak rate (fraction of outputs showing the user’s own training-photo background despite the prompt, lower better).

The step sweep goes one way only: from 400 to 2,000 steps identity climbs 0.41 → 0.72, adherence falls 0.79 → 0.54, background leak explodes 3% → 58%. Past 1,200 steps you buy 0.04 of identity and pay 0.17 of adherence, the knee.

The rank sweep shows something less obvious: rank 4 at 1,200 steps (0.61 / 0.75 / 11%) sits close to rank 16 at 800 steps (0.58 / 0.76 / 9%). Lowering rank behaves like training fewer steps. Rank is a regularizer, not just a capacity dial: the low-rank constraint limits how much of the training set the adapter can memorize no matter how long you run it. That is why cross-attention-only rank-4 stays a legitimate rung, and why rank 64 is worse on both columns that matter.

A checkpoint is a saved copy of the weights mid-training. Selecting on identity alone always picks the last checkpoint (identity is monotone in steps) and the last is the most overfit (learning the examples instead of the concept). Pick the knee, not the maximum.

The identity gate

Score every generated image against the references with the same cosine, and drop the failures before the user sees them.

at threshold cos ≥ 0.45   ->  14% of outputs dropped
                          ->  generate 48 to deliver 40   (+20% generation cost)

8 extra images · 4.06 s   =  32.5 GPU-s  =  $0.0226 at 100% util  =  $0.041 at 55%

measured effect on refunds: 9.1% -> 6.4%
                            0.027 · 29 = $0.78 of refund saved per user
                                          vs $0.041 spent   ->  19:1

Each avoided refund returns the full $29, so a 2.7-point drop is $0.78 per user against $0.041 spent. Price the gate at the same 55% utilization as the rest of the bill; at 100% it looks like 35:1, but that is a fleet you do not have. This is the highest-return component in the system, because the refund reason and the automated metric are the same thing: “it doesn’t look like me.”

Online metrics

A cohort is a group of users sharing a property, here a demographic one. When comparing recipes, hold the 30-user panel fixed: a hand-labeled panel rerun on every change is worth more than an online test you wait a week for. If you do run one, randomize by user, which is easy here because there is only one training run per user, so no user sees both arms.

Failure modes

Every failure traces back to the missing term in the loss.

Background and clothing overfit

Start from a realistic training set and read what comes out:

TRAINING SET   15 selfies, all in the same apartment over one weekend
               13/15 have the same kitchen backsplash
               11/15 wear the same grey hoodie
               all 15 shot on the same phone's front camera

PROMPT         "<tok> person, professional headshot, studio lighting,
                plain grey backdrop, navy suit"

OUTPUT at 2000 steps
  8/8   kitchen backsplash behind a studio-lit subject
  5/8   grey hoodie collar under the suit jacket
  8/8   wide-angle selfie lens distortion
  ArcFace cos 0.72     VQA adherence 0.54

The mechanism: the loss rewards any weight change that lowers reconstruction error on those 15 images, and cannot distinguish the face from what co-occurs with it. <tok> appears in 15/15 captions; the face appears in 15/15 images and the backsplash in 13/15, from the loss’s view, nearly the same signal. Gradient descent makes <tok> predict whatever it correlates with, and nothing ranks “bone structure” above “wallpaper,” so <tok> learns the joint distribution of the training set, everything in the photos together, not the person. Not a failure of the method; exactly what it was asked to learn.

flowchart TD
    S["15 selfies<br/>same room · same hoodie<br/>same camera"] --> L["Loss: reduce error<br/>on THESE 15 images"]
    L --> A["encode bone structure<br/>rewarded"]
    L --> B["encode the backsplash<br/>rewarded EQUALLY"]
    A --> T["trigger token tok<br/>appears in every caption"]
    B --> T
    T --> O["tok = the joint distribution<br/>of the training set,<br/>not the person"]
    O --> F1["identity UP<br/>with steps"]
    O --> F2["prompt following DOWN<br/>with steps"]

    style L fill:#1d3557,color:#fff
    style B fill:#9d0208,color:#fff
    style O fill:#bc6c25,color:#fff
    style F1 fill:#2d6a4f,color:#fff
    style F2 fill:#9d0208,color:#fff

Five fixes, ranked by effect. The ones that change what the objective rewards beat the ones that only change the data.

1. Caption the nuisance variables. A nuisance variable is something in the photos you do not want attached to the person. Instead of "a photo of <tok> person", write "<tok> person wearing a grey hoodie in a kitchen". Now “kitchen” claims that cross-attention mass, so the pressure on <tok> to encode the backsplash drops: the model no longer needs <tok> to explain the wallpaper. Largest single lever; costs one VLM (vision-language model) captioning pass over 15 images.
2. Face-masked loss. A segmentation mask marks which pixels belong to what; weight the per-pixel loss toward the face. With the face at ~18% of the frame and background weighted 0.2, the background’s gradient contribution falls ~5x.
3. Prior preservation. Generate ~200 images of “a person” from the base model itself and train on them alongside the user’s photos, captioned with the generic class word “person.” This holds “person” in place while <tok> moves; without it, “person” collapses onto this one user (which is why a group-shot prompt returns eight copies of the subject).
4. Lower rank, or fewer steps. Rank 4 cuts background leak from 21% to 11%.
5. Augmentation (random crop, flip, background replacement across the 15). Helps least. It changes the data but not what the objective rewards.

The rest, with traces

Identity drift across a batch: a few bad images in a good batch, so a serving-time problem. Per-image ArcFace shows two outliers at ~0.30 among 0.68-0.72; 0.30 is barely above the 0.02 different-people anchor, so those are a different person, not “slightly off.” Cause: high guidance pushes latents toward the base model’s prototypical face, overriding the adapter. Guard: the identity gate catches them per image; cap guidance at 4.5 (from the text-to-image chapter).

Prompt collapse: every image wrong the same way. Prompt asks for “hiking on a mountain trail”; output is an indoor head-and-shoulders portrait, 8/8. Cause: all 15 training images are head-and-shoulders selfies, so <tok> absorbed the framing. Guard: prior preservation; caption framing explicitly; report an off-distribution prompt eval slice (a named subset, here prompts asking for something no training photo showed, reported on its own, because averaging mountain-trail prompts into 40 studio portraits hides the failure).

Class bleed: damages a word, not an image. Prompt ” person and two colleagues” returns three people who all look like the subject. Cause: fine-tuning dragged the class token “person” toward this identity. Guard: prior preservation is the direct fix: that is exactly what the ~200 regularization images are for.

The table collects these plus the failures that live at the system edges.

Selfie-lens distortion is the exaggerated nose and narrow ears a wide-angle front camera produces at arm’s length; a focal-length classifier estimates the lens from an image, which makes it auto-detectable, not only human-visible. All of these are consequences of the missing term in the loss, not of undertraining. If they were undertraining, the fix would be more steps, and the tradeoff table shows more steps makes every one of them strictly worse.

Demographic bias, and why one metric cannot measure it

The identity metric is itself less accurate for some users than others, so it cannot tell you how much of a measured gap is the generator’s fault. Slice the identity metric and the download rate by skin-tone bucket (the Monk scale, a ten-point skin-tone scale built for this) and by gender presentation (how a person appears, not how they identify, the only thing an image metric can see).

The table is an illustrative shape, not a measurement from a real system.

Two distinct causes sit behind the bottom row, and conflating them is the mistake this section prevents:

• Cause 1: the metric is biased. ArcFace is trained on a corpus skewed toward light-skinned faces and has measurably higher error on darker-skinned faces. So some of that 0.59 is the encoder being worse at the measurement, not the generator being worse at the job. You cannot separate model bias from metric bias using the metric: every slice is measured with the same suspect instrument. The way out is a second measurement built differently: a human same-person/different-person study on a demographically balanced panel, against the same outputs. If humans say identity is fine and ArcFace says 0.59, the gate threshold is the bug, not the generator.

• Cause 2: the base model’s prior is biased. A prior is what a model produces by default before conditioning pushes it anywhere. “Professional headshot” in the pretraining corpus skews toward particular lighting and styling: three-point lighting calibrated on light skin under-exposes dark skin, and rare hair textures render as mush. This shows up in the download rate, a human judgment that never routes through ArcFace, which is why the download column falling 7.1 → 4.2 is the honest signal in that table.

What to do: report the download rate and refund rate by bucket, not the ArcFace score by bucket, those two are the metrics whose instrument is a person. Then fix the causes separately: a balanced fine-tuning set for the biased prior; per-bucket gate thresholds calibrated against the human study (not against ArcFace) for the biased metric; and a launch gate requiring that no bucket’s download rate fall more than 15% below the best.

Consent and likeness

The product ships a model of a specific person’s face, so consent, retention, and deletion are design constraints. Four controls; the first three prevent an unlawful artifact from ever existing, and the fourth destroys one that did.

1. All-same-person check at upload. Compute pairwise face-embedding cosines across the 15 uploads and require they agree. A scraped celebrity set fails (different eras and photographers); a mixed set, the user plus their partner, fails too, which is the honest mistake. Reject with a stated reason. The alternative is silently training a chimera: a face averaged from two people, belonging to neither.

2. Public-figure gallery match, at upload and on outputs. A gallery is a stored set of face embeddings for known public figures. At upload it stops someone training on a celebrity before it costs anything; on outputs it catches the opposite case, the base model’s prior dragging an ordinary user’s face toward a celebrity’s.

3. Liveness or attestation. The user has to assert they are the subject. Liveness (a check that the face is a live person and not a photo, screen, or mask) is the strong, costly version; attestation, a checkbox, is the weak, free one. Which you pick is a policy decision with a price tag, not a technical question.

4. Deletion that actually deletes. Because the adapter is biometric-derived, a deletion request must reach the source images, the adapter, every cached copy of the adapter (VRAM and NVMe, not just the object store), and the generated outputs, and it must emit an audit record, a durable log entry that can later prove deletion happened. Write the audit query first and run it as a scheduled test: the failure is silent, nothing errors when an adapter survives a deletion request, and you find out from a regulator.

Three legal facts are load-bearing, and each invalidates the design instead of degrading it. Consent must be specific, informed, and withdrawable instead of bundled into terms of service: training the adapter, generating the images, retaining the adapter, retaining the source photos, and improving the base model are five distinct purposes, and consent to one is not consent to another. User uploads must never contribute a gradient to the shared base model, or deletion becomes unachievable and the privacy claims become false. And you need a lawful basis for the public-figure gallery itself. It is a biometric database about people who never uploaded anything to you, so it needs its own basis, retention rule, and audit.

Alternatives considered and rejected

Conclusion

The design collapses onto one question: how many parameters does each user get, and who stores them.

• LoRA r=16 (42 MB, identity 0.68) is the default. It sits at the knee of the identity/adherence trade. Nobody ships full fine-tunes, because 5.2 GB per user is 23 PB and $538k/month of storage at 50k users/day, for 0.05 more identity.
• LoRA’s real saving is memory, not FLOPs (~23% cheaper training, not 100x). A frozen shared base fits ~26 training jobs and ~1,570 unmerged adapters on one 80 GB card, which is what makes co-batching (2.2x cheaper training) and heterogeneous serving (1.9x cheaper inference) possible.
• Compute is under 5% of contribution, so optimize for refunds, latency, and compliance. The identity gate that cuts refunds is worth ~5x any GPU optimization.
• The per-user adapter is biometric-derived data. It is in scope for deletion, the deletion must reach the VRAM and NVMe caches and not just the object store, uploads must never touch the base model, and consent must be specific and withdrawable. Get the first three points wrong and the product is expensive; get this one wrong and there is no product.

The full lifecycle, including the deletion path the serving diagram omits:

flowchart TD
    subgraph TRAIN["Train · once per user"]
      U([15 selfies + consent]) --> QC{Face QC:<br/>same-person · public-figure}
      QC -->|pass| TR["Co-batched LoRA trainer<br/>frozen base shared<br/>knee: 1200 steps · r=16"]
      TR --> AS[("Adapter store<br/>42 MB / user")]
    end
    subgraph SERVE["Serve · many times per user"]
      R([Request]) --> CACHE{"Adapter cache<br/>VRAM -> NVMe -> S3"}
      AS --> CACHE
      CACHE --> SAMP["Sampler pool<br/>base resident · unmerged adapters<br/>batched across users"]
      SAMP --> IDG{"Identity gate<br/>cos ≥ 0.45"}
      IDG -->|below| SAMP
      IDG -->|pass| DL([Deliver 40 of 48])
    end
    subgraph COMPLY["Delete + comply · on request"]
      DR([Deletion request]) --> WIPE["Erase source photos · adapter ·<br/>VRAM + NVMe caches · outputs"]
      WIPE --> AUD[("Audit record")]
    end
    AS -.biometric-derived.-> WIPE

One line to remember: personalization is a storage-and-deletion problem wearing a quality problem’s clothes, so pick the smallest artifact that clears the identity bar.

Further reading

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021).
• Ruiz et al., DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022).
• Gal et al., An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022).
• Ye et al., IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023), the encoder-based approach.
• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (2022), Stable Diffusion.
• Ho & Salimans, Classifier-Free Diffusion Guidance (2022).
• Deng et al., ArcFace: Additive Angular Margin Loss for Deep Face Recognition (2019).
• Google, Monk Skin Tone Scale, documentation of the ten-point scale used for the fairness slices.

Next: Text-to-video, the same problem with a temporal axis, where the compute blow-up decides the product shape before any modelling choice does.