10 — Personalized Headshot Generation

“A user uploads 15 selfies. Twenty minutes later they get 40 professional headshots that look like them. Design it.”

What this chapter is about

This is a system that trains a small piece of a neural network on one specific person’s face, then generates new pictures of them.

The interesting engineering is not the picture-making. That is chapter 09’s subject, and this chapter restates only what it needs. The interesting part is that there is now one model artifact per customer, which collapses quality, cost, storage and privacy into a single problem.

Four things get derived here:

• A five-rung ladder of personalization methods, spanning five orders of magnitude in per-user file size.
• The price of each rung, carried all the way out to a profit-and-loss statement.
• How one graphics card serves a thousand different users at once.
• Consent, retention and deletion as design constraints, not as a compliance appendix bolted on at the end.

When you finish, you should be able to pick a rung and defend it with arithmetic, explain why training longer makes the product worse, and say precisely what has to be deleted when a user asks you to forget them.

The input and the output, concretely

In: 10-20 selfies that a person uploads from their phone, plus their consent to train on them.

Out, about twenty minutes later: roughly 40 images of that same person in professional-headshot settings — studio lighting, a neutral backdrop, business attire, several poses — recognizable to their colleagues.

In between, and this is what makes the system unusual: a small per-user file of learned weights that did not exist before they uploaded, and must not exist after they ask you to delete it.

The machinery this chapter stands on

You do not need chapter 09 to follow this one. Here is everything from it that gets used, one term per line. Skim it now; each term reappears in context later.

• Latent diffusion model — the image generator. It works in a compressed space rather than on pixels directly.
• Variational autoencoder (VAE) — compresses an image into a small grid of numbers called a latent, and expands a latent back into pixels.
• Denoiser — a large network trained to look at a noisy latent and predict the noise in it.
• Sampler — the loop that runs the denoiser about 30 times, subtracting a little predicted noise each round, until a clean latent falls out.
• Text encoder — turns the prompt into a sequence of vectors.
• Cross-attention — how the prompt reaches the denoiser. At every layer, each spatial position of the image asks the text what should be drawn there and gets back a weighted blend of the answers. This is the mechanism every adapter in this chapter attaches to.
• Classifier-free guidance (CFG) — a sampling trick that runs the denoiser twice per step, once with the prompt and once with a blank prompt, then extrapolates away from the blank one. That is what makes an image follow its prompt harder. Its strength is a dial called w. It doubles the compute per step, and turning it too high degrades the image.

Chapter 09 derives all of that. Only two of its results are reused here:

1. Generation costs about 26.8 trillion floating-point operations per denoiser pass.
2. Guidance above about 4.5 starts overriding personalization — it drags the face back toward the base model’s generic one.

The one question the design collapses onto

This chapter is what happens when every user needs their own model. Everything reduces to: how many parameters does a user get, and who pays to store them?

The interview is not really about diffusion. It is about a fine-tuning ladder — a ranked set of ways to specialize an already-trained model, from “retrain everything” at the top down to “change nothing and pass in an extra vector” at the bottom. Five rungs, differing by one to five orders of magnitude in how many parameters each user gets, plus the arithmetic that decides which rung the product can afford.

Separate two words now, because they get used a hundred times:

• Fine-tuning means continuing to train an existing model’s own weights on new data. When you are done, those weights are different and you have a whole new model.
• An adapter means leaving every original weight frozen and adding a small number of new parameters alongside them. When you are done, you have the original model plus a small file.

The distinction sounds academic. It is the entire economics of this chapter: a fine-tune is 5.2 GB per customer and cannot be shared between customers; an adapter is 42 MB and can.

How to read the compute numbers

Two conventions run through every cost block, so pin them down once.

Storage is quoted in fp16. fp16 is a 16-bit floating-point number format, so two bytes per parameter. That is the only conversion between any parameter count and any file size in this chapter. A 21.0M-parameter adapter is 21.0e6 · 2 = 42 MB, and nothing more clever is happening.

Speed is quoted as MFU against the H100’s peak. MFU is model FLOPs utilization: the fraction of a graphics card’s theoretical arithmetic throughput that you actually achieve. An H100 peaks at roughly 990 TFLOP/s on the 16-bit matrix math these models use, so:

MFU 22%   ->  0.22 · 990  =  218 TFLOP/s
MFU 40%   ->  0.40 · 990  =  396 TFLOP/s
MFU 48%   ->  0.48 · 990  =  475 TFLOP/s

Those three rates are the ones every timing below is divided by. Chapter 09 quotes the same hardware as “300 TFLOP/s effective,” which is the same statement at about 30% MFU; this chapter splits it out because the whole argument turns on moving the MFU, and you cannot move a number that has been folded away.

1. Problem framing

The contract, in four lines:

• Input: 10-20 selfies, uploaded by the subject, of wildly variable quality.
• Output: ~40 images of that person in professional-headshot settings — studio lighting, neutral backdrop, business attire, several poses.
• Constraints: identity has to be recognizable to the subject’s colleagues; turnaround has to fit inside a session or an email; and per-user marginal cost — what one additional customer costs you, ignoring fixed costs — has to be a small fraction of a ~$29 price point.
• Why it is hard: the model must learn a new concept — this person’s face — from 15 examples, without learning the 15 photos.

That last point is the whole difficulty, so spell it out. Gradient descent is the training procedure that repeatedly nudges weights in whichever direction lowers the loss. Under gradient descent, “learn what this person looks like” and “memorize these 15 pictures” are not two different operations. They are the same operation, and the loss has no term that prefers one over the other. Ml objective shows exactly where that term is missing.

The opening statement

This is what you say in the first thirty seconds, before any diagram:

“The design decision is the personalization method, and it is decided by storage and batching at fleet scale, not by per-user GPU dollars. A full fine-tune is 5.2 GB per user; a rank-16 LoRA is 42 MB. At 50,000 users a day with 90-day retention that is 23 petabytes versus 189 terabytes — and only the second one lets one GPU serve a thousand different users out of one batched matmul.”

Three terms in that sentence get their full treatment in The personalization ladder derived and Multi tenancy thousands of adapters one base model. In short:

• LoRA stands for Low-Rank Adaptation. It is the adapter method this chapter selects, and rank (written r) is its one size dial. Rank 16 is the pick.
• A matmul is a matrix multiplication — the operation that dominates all of this arithmetic.
• Batching means pushing many requests through one matmul at once. It is the only way a graphics card runs anywhere near its rated speed, which is why it appears in a sentence about storage.

The two quality axes

Two more terms, and the whole chapter turns on the tension between them.

Identity preservation — usually shortened to “identity” — is the property that the generated face is recognizably the same person as the uploads. Metrics identity and prompt following as separate axes turns it into a number.

Prompt following, also called prompt adherence, is whether the image contains what the prompt asked for: the suit, the studio backdrop, the mountain trail.

These two pull against each other along every training knob you have. Train harder and identity goes up while prompt following goes down. Choosing where to sit on that trade is the product.

Three reframes

The table below contrasts the answer a candidate gives on reflex with the answer that survives the follow-up question. Read the right column as the thesis of the whole chapter — each row is developed in a later section.

Three words in that table need pinning down:

• Monotonically means “in one direction, always.” More training steps never lower identity, and never raise prompt following.
• The knee of a curve is the point past which you pay a lot for a little — the bend where identity has stopped improving quickly and prompt following has started collapsing.
• Biometric-derived data is data computed from someone’s body, here from their face. In most jurisdictions it carries stricter consent, retention and deletion rules than ordinary personal data.

That last phrase is not decoration. Consent and likeness is where it becomes the design.

Assumptions in this stage.

State out loud — you are free to pick these, and being wrong costs a re-derivation:

• 10-20 uploads, ~40 delivered images, a ~$29 price, a turnaround measured in tens of minutes. Each is a product decision. Being wrong re-prices Per user economics end to end without changing any mechanism.

Ask, never assume — the answer changes the architecture:

• What “recognizable” means to this business. Recognizable to a colleague glancing at a company directory is a far lower bar than recognizable to a close friend. The bar sets the identity threshold in Metrics identity and prompt following as separate axes, which sets the refund rate, which is the largest single line in the P&L.
• Whether users may upload photos of anyone other than themselves. This is a policy answer, not a technical one, and it decides whether Consent and likeness’s controls are a hard gate or a warning banner.

Load-bearing — if this is wrong the design is not suboptimal, it is invalid:

• The per-user artifact is biometric-derived data with a legal lifecycle. That single classification is what makes storage a compliance surface rather than a line item, what makes deletion a tested audit query rather than an S3 lifecycle rule, and what makes the zero-storage rung in Textual inversion and encoder based honestly attractive for reasons that have nothing to do with cost.
• Being wrong in either direction hurts. Under-classify and an artifact you kept is an unlawful retention. Over-classify and you have a product you cannot ship in a region you thought you could.

2. ML objective

Personalization does not get its own objective. The loss is the ordinary denoising loss of Ml objective, restricted to a tiny dataset and to a subset of the parameters:

D_user = { (x_i, "a photo of <tok> person", ) }   i = 1..15
theta_trainable  subset of  theta_base

loss = E_{i, t, eps} || eps - eps_theta(z_t^i, t, c_i) ||^2

Read it one line at a time.

Line 1, the dataset. D_user is this user’s training set: 15 pairs of a photo x_i and a caption. Every caption uses the same template, "a photo of <tok> person". <tok> is a trigger token — a rare, otherwise-meaningless string, often something like sks, chosen precisely because the base model has no prior associations with it. That makes it free to become a handle for this person.

Line 2, what may move. theta_base is the full set of the base model’s weights. theta_trainable is the subset you allow to change. Which subset that is is the entire ladder in The personalization ladder derived.

Line 3, the loss. Corrupt the user’s image latent with noise eps at a random timestep t, and train the denoiser eps_theta to predict that noise, given the caption c_i. The || ... ||^2 is squared error between the noise you added and the noise the model guessed. E_{i, t, eps} means “average over” — over images i, timesteps t, and noise draws eps.

That is the ordinary diffusion training loss. Nothing about it is personalization-specific. The personalization is entirely in what data you feed it and which weights you unfreeze.

Which means two things are absent from it. Both become failure modes in Failure modes:

There is no term that says “learn the face, not the room.” Any weight change that lowers reconstruction error on those 15 images is rewarded equally, whether it encodes bone structure or wallpaper.

There is no term that preserves the base model’s behavior. The optimizer is free to overwrite general knowledge — a phenomenon called catastrophic forgetting, where training hard on a narrow new task destroys competence on everything the model could previously do. Everything that stops it — prior preservation, low rank, early stopping — is a regularizer, meaning an extra constraint or penalty bolted on to steer training away from a behaviour you do not want, and choosing among them is the design.

Assumptions in this stage.

State out loud:

• 15 training images with a fixed caption template, and a rare trigger token with no prior meaning. Both are recipe choices. The caption template in particular is revisited in Failure modes, where enriching it turns out to be the single largest quality lever in the chapter.

Ask:

• Whether the user’s uploads may be retained at all after training completes, and for how long. That answer decides whether re-running a failed job is even possible, and it is a legal answer rather than a technical one.

Load-bearing:

• Nothing in the loss distinguishes the face from what co-occurs with it. Every guard in Failure modes — nuisance captioning, face-masked loss, prior preservation, rank as a regularizer, early stopping — exists only because that is true. The identity-versus-adherence trade in Metrics identity and prompt following as separate axes is its direct consequence.
• If the objective could separate subject from background, you would train to convergence and the entire tuning apparatus disappears.

3. The personalization ladder, derived

There are five ways to give a model a new face. The savings between them do not come from where people assume, and the two cheapest have hard limits — all of which is worth deriving rather than memorizing.

3.1 The five rungs

Put all five methods on one table and the asymmetry that decides the design — five orders of magnitude in storage against a factor of 1.7 in quality — becomes visible in one glance.

The backbone every number is computed against

Reference backbone throughout, the same one as chapter 09: a 2.6B-parameter DiT. DiT stands for Diffusion Transformer, meaning the denoiser is built out of transformer blocks. Its two size numbers are hidden width d = 2048 and 40 blocks.

Where the number 320 comes from, because it recurs on every line below. Each of the 40 blocks holds 8 square weight matrices of size d × d, all used by attention:

per block:  4 for self-attention   (query, key, value, output projections)
          + 4 for cross-attention  (the same four again)
          = 8 matrices

fleet:      8 · 40 blocks                =    320 matrices
            320 · 2048^2  =  320 · 4,194,304  =  1.34B parameters

So attention alone is 1.34B of the model’s 2.6B parameters, spread over 320 matrices. Those 320 matrices are what every adapter in this chapter attaches to. Nothing else gets touched.

The five rungs, named

Before the table, here is what each row actually does:

• Full fine-tune — update all 2.6B weights. No constraint, no protection.
• DreamBooth — a full fine-tune plus a prior-preservation loss: an extra loss term that trains simultaneously on generic images the base model produced itself, so the general meaning of the word “person” is not overwritten by this one person.
• LoRA (Low-Rank Adaptation) — freeze every original weight, and learn a small additive correction to each matrix. The rank r controls how small. Loras saving derived derives it.
• Textual inversion — change no network weights at all. Instead learn a few brand-new word vectors that stand for this person.
• Encoder-based identity adapter — train nothing per user. Take a face-recognition vector computed from the selfies and inject it into cross-attention through layers that were trained once, offline, across millions of identities.

Three columns need a note before the table means anything.

Storage / user is the trainable-parameter count times 2 bytes, because everything is stored in fp16. Full fine-tune: 2.6e9 · 2 = 5.2 GB. LoRA r=16: 21.0e6 · 2 = 42 MB. Textual inversion: 16.4e3 · 2 = 32 KB. That is the only conversion between column 2 and column 3.

The last row is the exception, and it is the interesting one. The encoder-based rung trains zero parameters per user, so it has no adapter to store at all. Its 1 KB is a 512-dimensional face embedding you compute at request time — and if you choose not to write it to disk, the per-user storage is genuinely nothing.

Train time is deliberately solo — one user’s job alone on a graphics card — because the interesting comparison is like for like. Measured that way, LoRA is 374 / 287 = 1.3x faster to train than a full fine-tune, not 3x. Co-batching (The saving is not where people think it is) means running several users’ training jobs through the same forward pass; it takes the r=16 rung from 287 GPU-s down to 132. But that lever is available to every rung that shares a frozen base, and to none that does not — so leaving it out of this table keeps the comparison honest.

ArcFace cos is the identity metric Metrics identity and prompt following as separate axes derives. Run a face-recognition model called ArcFace over the generated face and over the reference selfies, and take the cosine similarity between the resulting vectors: 1 for identical directions, 0 for unrelated ones. A raw cosine is meaningless without anchors telling you what “good” is; Metrics identity and prompt following as separate axes supplies them.

Now read the storage column against the identity column, top to bottom:

storage:   5.2 GB  ->  42 MB  ->  32 KB          spans ~160,000x
identity:  0.73    ->  0.68   ->  0.44           spans 1.7x

Storage spans five orders of magnitude; identity spans a factor of 1.7. That asymmetry is the entire argument of the chapter, and it is why nobody ships full fine-tunes.

Turning the table into a pick

The decision tree below asks three questions in priority order. Latency comes first because it is a hard product constraint rather than a preference — if the product promises ten seconds, no amount of quality argument matters. Then whether identity is what users are actually complaining about. Then what the storage budget will bear.

flowchart TD
    Q1{"Is per-user latency<br/>under 10 s required?"} -->|yes| ENC["Encoder-based adapter<br/>0 params/user · 1 KB<br/>identity 0.52"]
    Q1 -->|no| Q2{"Is identity the<br/>top complaint?"}
    Q2 -->|no| TI["Textual inversion<br/>32 KB/user<br/>identity 0.44"]
    Q2 -->|yes| Q3{"Fleet storage budget<br/>at 50k users/day?"}
    Q3 -->|"tight"| LR4["LoRA r=4 cross-attn<br/>5.2 MB/user<br/>identity 0.58"]
    Q3 -->|"normal"| LR16["LoRA r=16 all-attn<br/>42 MB/user<br/>identity 0.68"]
    Q3 -->|"unbounded"| DB["DreamBooth full weights<br/>5.2 GB/user<br/>identity 0.73<br/>NOT BATCHABLE"]

    style ENC fill:#2d6a4f,color:#fff
    style LR16 fill:#1d3557,color:#fff
    style DB fill:#9d0208,color:#fff

The three storage answers — tight, normal and unbounded — are the whole content of the bottom branch. Its two LoRA leaves differ only in where the adapters are attached: LoRA r=16 all-attn puts rank-16 adapters on all 320 attention matrices at 42 MB per user for identity 0.68, while LoRA r=4 cross-attn puts rank-4 adapters on the 160 cross-attention matrices only, at 5.2 MB per user for identity 0.58. The red box carries a warning the table above does not: full weights are not batchable, meaning two users cannot share a forward pass, which Multi tenancy thousands of adapters one base model shows costs about 1.9x per image on its own.

3.2 LoRA’s saving, derived

LoRA’s parameter saving reduces to one formula, 2r/d — worth being able to reproduce on a whiteboard, because it answers every follow-up about rank immediately.

A weight matrix W in R^{d · d} — meaning a d-by-d grid of real numbers — is not changed directly. Instead you add a correction dW alongside it, and you constrain dW to be low rank.

Rank is the number of independent directions a matrix can express. A 2048 × 2048 matrix can express up to 2048 of them. Forcing rank r means dW must factor into a tall skinny matrix times a wide skinny one — d × r times r × d — and that factoring is why it takes so few parameters to store. You never write down the big dW; you store only the two skinny factors.

The block below does the substitution at d = 2048, r = 16. Watch the third line: everything cancels down to 2r/d.

W' = W + dW,      dW = B · A,      B in R^{d · r},   A in R^{r · d}

parameters in dW (full)   =  d^2         =  2048^2      =  4,194,304
parameters in B and A     =  2·d·r       =  2·2048·16   =     65,536

ratio  =  2·d·r / d^2  =  2r / d  =  32 / 2048  =  1.5625%

2r/d is the whole formula. It says the saving depends only on the rank relative to the model width — not on how many layers you touch, because the same ratio applies to each matrix independently.

Have it at your fingertips, because the follow-up is always “what if I use rank 64,” and the answer is 2 · 64 / 2048 = 6.25% — four times as much — with no further work.

Now scale it to all 320 matrices. The block below goes from one matrix to the whole model, then converts to bytes at 2 bytes per parameter.

320 attention matrices  (8 per block · 40 blocks)

full attention weights   320 · 4,194,304   =  1.342e9  params
LoRA r=16                320 ·    65,536   =  2.097e7  params   =  21.0M
                                                                   1.5625% of attention
                                                                   0.81%   of the 2.6B model

storage, fp16            21.0e6 · 2 bytes  =  41.9 MB           -> call it 42 MB
r=4,  all 320 matrices                        10.5 MB
r=4,  cross-attention only (160 matrices)      5.2 MB
r=32, all 320 matrices                        83.9 MB
full fine-tune           2.6e9 · 2 bytes   =   5.2 GB           -> 124x the r=16 adapter

3.3 The saving is not where people think it is

The most common misconception about LoRA is that it makes training dramatically cheaper in compute. It does not. The real saving is memory — which buys batching, which buys the cost reduction.

The misconception: LoRA barely reduces training FLOPs

FLOPs are floating-point operations — the count of arithmetic the graphics card performs. A TFLOP is a trillion of them; a PFLOP is a thousand TFLOP.

The intuition people arrive with is: “LoRA trains 0.81% of the parameters, so it must be roughly 100x cheaper.” It is not, and the reason is worth being able to state.

You still push every image through the whole 2.6B model — every frozen matrix still has to multiply. And you still backpropagate, meaning run the gradient computation backwards through every layer, because that is the only way the training signal reaches an adapter sitting down in layer 3. Freezing a layer does not let you skip it.

The only thing you skip is computing the weight gradients for the frozen matrices. A backward pass computes two things per layer — gradients with respect to the inputs (needed to keep going backwards) and gradients with respect to the weights (needed to update them) — and LoRA drops the second for 99.19% of the weights.

The block below prices that. 2 · params · tokens is the standard count for a forward pass through a transformer’s weight matrices; the attention term is separate because it scales with tokens squared rather than with parameters.

training at 512px:  latent 64 · 64, patch 2  ->  1,024 tokens

forward per image   2 · 2.6e9 · 1,024                   =  5.32 TFLOP
                    attention 4 · 1024^2 · 2048 · 40    =  0.34 TFLOP
                                                           ----
                                                           5.66 TFLOP

full fine-tune      fwd + bwd  ~=  3.0 · fwd            = 17.0 TFLOP / image
LoRA                fwd + bwd  ~=  2.3 · fwd            = 13.0 TFLOP / image
                                                           -> 23% saving. That is all.

Read the two multipliers. A full fine-tune costs 3.0 · fwd: one unit forward, one for input gradients, one for weight gradients. LoRA costs 2.3 · fwd — it keeps the forward and the input gradients, and pays only a sliver for the weight gradients it still needs. 13.0 / 17.0 = 0.765, so 23% cheaper, not 100x cheaper.

Where the saving actually is: memory

What LoRA buys is memory, and memory buys batching.

Training a model needs four separate things resident in the graphics card’s memory, not one:

1. The weights themselves — 2 bytes per parameter in fp16.
2. The gradients — one number per trainable weight, 2 bytes each.
3. The optimizer state. Adam, the standard training algorithm, keeps two running averages per trainable weight (conventionally m and v) in 32-bit precision (fp32, 4 bytes). That is 8 bytes per trainable parameter.
4. A master copy of the trainable weights in fp32, 4 bytes each, because accumulating tiny updates in 16-bit loses them to rounding.

Here is the point of that list: items 2, 3 and 4 are sized by the trainable count, not the total count. And item 1, the only one sized by the total, is the only one that can be shared between two users’ jobs running on the same card — because for LoRA it is frozen and therefore identical for everyone.

                    weights   grads    Adam m,v (fp32)   master (fp32)   total
full fine-tune      5.2 GB    5.2 GB       20.8 GB          10.4 GB      41.6 GB
LoRA r=16           5.2 GB   0.04 GB       0.17 GB          0.08 GB       5.5 GB
                    (frozen, and SHARED across co-resident jobs)

Checking one row so the table is reproducible: LoRA r=16 has 21.0M trainable parameters, so gradients are 21.0e6 · 2 = 42 MB, Adam is 21.0e6 · 8 = 168 MB, and the master copy is 21.0e6 · 4 = 84 MB. Add the 5.2 GB base and you get 5.5 GB. The full fine-tune row is the same arithmetic with 2.6B in place of 21.0M.

Now put that on an 80 GB card. One more term first: activations are the intermediate values a forward pass produces and the backward pass has to read back. They scale with batch size, not with parameter count, which is why they get their own column.

full fine-tune   41.6 GB state + ~12 GB activations   ->  1 job per GPU
LoRA             5.2 GB shared base
                 + per job: 0.29 GB adapter state + ~2.5 GB activations
                 (80 - 5.2) / 2.8                     ->  26 concurrent jobs per GPU

The LoRA line is the one to trace. The 5.2 GB base is paid once for the whole card. Each additional job then costs only 0.29 + 2.5 = 2.8 GB — its own adapter state and its own activations. So the card’s remaining 80 - 5.2 = 74.8 GB divides into 74.8 / 2.8 = 26 concurrent jobs. The full fine-tune gets 1, because there is nothing to share.

Why more jobs per card is the same thing as lower cost

Fitting 26 jobs on a card only matters because of what it does to MFU.

At small batch sizes the card spends most of its time waiting for weights to arrive from memory rather than doing arithmetic — it reads a whole weight matrix to do a tiny amount of work with it. Raising the batch spreads each weight read over more work. That is why the achieved rate more than doubles between the two rows below.

batch 4  (4,096 tokens/step)     MFU ~22%   ->  218 TFLOP/s
batch 32 (32,768 tokens/step)    MFU ~48%   ->  475 TFLOP/s

1,200 steps · 4 images · 13.0 TFLOP  =  62.5 PFLOP per user

alone   62.5e15 / 218e12  =  287 GPU-s  =  $0.199   (4.8 min wall clock)
8-way   62.5e15 / 475e12  =  132 GPU-s  =  $0.092   (17.5 min wall clock)

Three things to trace in that block.

The 62.5 PFLOP. The recipe is 1,200 optimizer steps at batch 4, so 1,200 · 4 = 4,800 images pass through training. Each costs 13.0 TFLOP from the block above, giving 4,800 · 13.0 = 62,400 TFLOP. The block says 62.5 because it carries the unrounded 13.04 TFLOP per image rather than the displayed 13.0; the difference is 0.2% and changes nothing downstream.

Batch 32 is eight users co-batched at batch 4 each. 8 · 4 = 32. That is the mechanism: you are not raising one user’s batch size, you are stacking eight users’ jobs into one forward pass through the shared frozen base.

The dollars. GPU-seconds times the H100 rate: 287 / 3600 · $2.50 = $0.199, and 132 / 3600 · $2.50 = $0.092.

The two wall-clock figures are the same numbers seen from the user’s side. Alone, one job owns the card for its full 287 seconds — 4.8 minutes. Co-batched eight ways, each job’s share of the card is 132 GPU-seconds, but eight jobs are interleaved on that card, so the user waits 132 · 8 = 1,056 seconds, or 17.5 minutes.

That is the trade, in two ratios:

cost    $0.199 / $0.092   =  2.2x cheaper co-batched
latency  17.5 min / 4.8   =  3.7x slower co-batched

Co-batching eight users cuts per-user cost 2.2x and multiplies per-user latency by 3.7x. That is a tiering decision with the arithmetic already done: run the free tier co-batched at 20 minutes, the paid tier dedicated at 5 minutes, and the instant tier on the encoder-based adapter at 10 seconds.

Full fine-tuning cannot participate in any of this, because two users’ weights cannot share a forward pass. There is no frozen base to stack jobs on top of.

3.4 Textual inversion and encoder-based, honestly

Two rungs of the ladder look nearly free. Each hits a structural ceiling, and the second ceiling is the most interesting fact in the chapter.

Textual inversion

Textual inversion changes no network weights at all. It optimizes only a handful of brand-new embedding vectors — the lookup vectors the text encoder assigns to tokens — living in the text encoder’s input space. In effect it invents four new words that mean “this person,” and leaves the model itself alone.

Where the 32 KB comes from, with a text encoder of width 4096:

4 new tokens · 4,096 dims        =  16,384 floats
16,384 · 2 bytes (fp16)          =  32,768 bytes  =  32 KB

What it gets right: the base model is untouched, so prompt following is perfect by construction and there is no catastrophic forgetting to defend against. Its file is a rounding error.

What it gets wrong, in two ways.

It converges slowly — converge meaning the loss stops improving — needing a few thousand steps rather than 1,200. The reason is structural: the gradient still has to travel backwards through the entire network on every step, but it arrives at only 16k parameters, so each step buys very little.

And it tops out at a much lower identity fidelity, 0.44. A single point in text-embedding space simply cannot express everything a face is. Use textual inversion when the concept is a style or an object; it is under-powered for faces.

Encoder-based identity adapters, and the ceiling they hit

Encoder-based identity adapters move the personalization out of training entirely.

Train once, offline: a projection — a learned linear map — from a face-recognition embedding into the space the denoiser’s cross-attention reads from, plus a small set of adapter layers. Call it 90M shared parameters, trained across millions of identities. That training happens before any customer exists.

At serve time you run a face encoder over the user’s selfies, average the resulting embeddings, and inject that vector. Zero training, zero per-user storage, generation-only latency.

Identity lands around 0.52 against LoRA’s 0.68, and — this is the part worth understanding rather than memorizing — the ceiling is structural, not a matter of training the adapter harder.

The adapter can only express identity to the extent that the face-recognition embedding captured it. And a face-recognition model is built for invariance: it must return the same answer for the same person under different lighting, with and without glasses, with a new haircut, ten years apart. That is its whole job.

So glasses, hairstyle, lighting-dependent skin tone and facial asymmetries are exactly what it was trained to throw away — and therefore exactly what it cannot hand to the generator.

A face-recognition embedding is optimized to discard everything that varies between photos of the same person, which is a large fraction of what a person looks like. That sentence is why encoder-based approaches plateau, and it is the answer to “why not just use the embedding?”

The mature answer is to ship both: encoder-based for the instant preview, LoRA for the delivered set. The preview converts the user; the LoRA satisfies them.

Assumptions in this stage.

State out loud:

• A 2.6B DiT at d = 2048 with 40 blocks and 320 attention matrices; rank 16 across all of them; 1,200 training steps at batch 4 and 512 pixels; MFU of 22% solo and 48% co-batched eight ways; identity figures measured on a 30-user panel.
• Every dollar and second above is a product of those. Different model sizes move all six rows of the ladder together and reorder none of them.

Ask:

• The tier structure the product will actually sell — free, paid, instant. Tiering is what decides whether you co-batch, and co-batching is a 2.2x cost swing against a 3.7x latency swing in opposite directions.
• Whether an unpersonalized preview is acceptable to show the user, since the entire encoder-based rung exists to serve that one moment.

Load-bearing:

• A frozen shared base is what makes co-batching possible. The saving comes from memory, not from FLOPs. It is why LoRA’s 23% FLOP saving is a footnote and its 26-jobs-per-card memory footprint is the headline, and it is the same fact that returns in Multi tenancy thousands of adapters one base model as the 1.9x serving win.
• If the base could not be shared — if each user’s job needed its own copy of the 5.2 GB of weights — the ladder collapses to “everything costs what a full fine-tune costs,” and the product does not exist at this price.

4. Per-user economics, end to end

With a rung picked, price one customer from upload to delivery — and then do the more important thing: put that price next to the revenue, where it turns out not to be what decides whether the business works.

Prices: H100 at $2.50/GPU-hour, object storage — bulk cloud file storage such as Amazon S3 — at $0.023/GB-month, and egress, the charge for data leaving the cloud provider’s network, at $0.09/GB.

Generation

Generation is the larger of the two GPU line items. Same backbone arithmetic as Cost per image derived, but at 1024 pixels instead of the 512 used for training, so the token count goes up 4x:

at 1024px:  latent 128 · 128, patch 2  ->  4,096 tokens

parameter work   2 · 2.6e9 · 4,096            =  21.3 TFLOP
attention        4 · 4096^2 · 2048 · 40       =   5.5 TFLOP
                                                 ----
one forward pass                                 26.8 TFLOP

Now count passes. The sampler runs 30 steps, and CFG runs the denoiser twice per step — once with the prompt, once blank — so 30 steps is 60 passes, and CFG is literally half the generation bill.

per image        60 · 26.8                    =  1,608 TFLOP
batch 48, MFU ~40%                            =    396 TFLOP/s
                 1,608 / 396                  =   4.06 s per image

Generate 48, deliver the best 40. The extra eight cover the images the identity gate rejects — an automatic check, derived in The gate that pays for itself, that scores every generated face against the user’s references and drops the ones that do not look enough like them.

48 · 4.06                                     =  195 GPU-s   =  $0.135

That dollar figure is 195 / 3600 · $2.50 = $0.135, and the same GPU-seconds-to-dollars conversion is used on every line below.

The bill

Now assemble every line, including the ones people forget: the failure reruns, the free re-rolls, the storage, and the bandwidth.

Two terms first. QC is quality control, the automated checks a finished job must pass before delivery. A re-roll is a user asking for another set of images at no extra charge.

The block below has two halves: GPU cost on top, then storage and bandwidth underneath. Every line after “GPU at 100% utilization” is a multiplier applied to the line above it, so read it top to bottom.

LoRA training      132 GPU-s  (8-way co-batched)           $0.092
generation, 48     195 GPU-s                               $0.135
restore + upscale to 2048px, 40 images                     $0.008
                                                           ------
GPU at 100% utilization                                    $0.235
fleet utilization 55%                                      $0.428
4% of jobs fail QC and rerun end to end   (· 1.04)         $0.445
12% of users request a regeneration       (+ 0.12 · 0.246) $0.474

adapter storage    42 MB · 90 days                         $0.0029
outputs            80 MB · 90 days                         $0.0055
source selfies     60 MB · 30 days                         $0.0014
egress             80 MB at $0.09/GB                       $0.0072
                                                           ------
                                          MARGINAL COST    $0.491

Four lines in there are not self-evident.

Fleet utilization 55%. You rent the graphics card by the hour, not by the second of work you do on it. If the fleet is busy 55% of the time, every second of real work has to carry 45 seconds of idle for every 55 it does: $0.235 / 0.55 = $0.428. This is the second-largest multiplier in the bill.

The QC rerun, · 1.04. 4% of jobs fail QC and are rerun end to end at your expense, so the whole bill above is multiplied by 1.04: $0.428 · 1.04 = $0.445.

The re-roll, + 0.12 · 0.246. 12% of users ask for another set. A re-roll regenerates images but does not retrain the adapter, so it costs only the generation line at fleet utilization: $0.135 / 0.55 = $0.246. Multiply by the 12% who ask: 0.12 · 0.246 = $0.0295, giving $0.445 + $0.0295 = $0.474.

The storage lines. Object storage is priced at $0.023 per GB-month, so 90 days is 3 months. The adapter line is 0.042 GB · $0.023 · 3 = $0.0029. Outputs are 0.080 GB · $0.023 · 3 = $0.0055. Source selfies are kept only 30 days, so one month: 0.060 GB · $0.023 · 1 = $0.0014. Egress is a one-time charge on the 80 MB the user downloads: 0.080 · $0.09 = $0.0072.

Call it $0.49 per user against a $29 price. 98% gross margin on marginal cost.

Gross margin is revenue minus the cost of delivering it, as a percentage of revenue: (29 - 0.49) / 29 = 98%. That sounds like the end of the conversation. It is not.

Now the honest part

At 98% margin, the GPU bill is not the number that decides whether the business works — and the arithmetic says what is.

The $0.49 is only meaningful next to everything else that scales per sale. Put it in a P&L — a profit-and-loss statement, the list of what one sale earns and what it costs.

Four terms in the block below:

• Chargebacks are payments reversed by the customer’s bank.
• CAC is customer acquisition cost, what you pay in advertising to get one buyer. “Category CAC” means the going rate in this market.
• Support at $4 is the loaded cost of one support contact, paid on the 6% of users who need one.
• Contribution per user is what is left after every cost that scales with one more sale. It is the number that has to cover fixed costs and profit.

price                                     $29.00
refunds and chargebacks, 9%               -$2.61
marginal compute and storage              -$0.49
paid acquisition (category CAC)          -$15.00
support, 6% of users at $4                -$0.24
                                          ------
contribution per user                     $10.66

compute as a share of contribution:  0.49 / 10.66  =  4.6%

Two of those lines are just percentages of the price: refunds are 0.09 · 29 = $2.61, support is 0.06 · $4 = $0.24. Everything sums to 29 - 2.61 - 0.49 - 15.00 - 0.24 = $10.66.

Now compare two optimizations you could spend a quarter on.

halve the GPU bill:      save 0.245  ->  0.245 / 10.66  =  2.3% more contribution
refunds 9% -> 5%:        save 0.04 · 29 = 1.16
                              1.16 / 10.66  = 10.9% more contribution

Halving the GPU bill moves contribution by 2.3%. Cutting refunds from 9% to 5% moves it by 10.9% — roughly five times as much.

And refunds in this product are almost entirely “it doesn’t look like me,” which means the identity gate in The gate that pays for itself is worth about five times what any compute optimization is worth. Same shape of reframe as case study 06, where the human cost dwarfed the model cost.

The three reasons the ladder choice still matters

If compute is 4.6% of contribution, the ladder choice cannot be justified on per-user dollars. It is justified on three other things.

Reason 1: fleet storage. Retention is how long you keep an artifact before deleting it. 90 days is chosen here so that a user who comes back next quarter does not have to retrain — and it is a number Consent and likeness has opinions about.

At 50,000 new users a day with 90-day retention, the steady-state stored volume is per-user size · 50,000 · 90, because at any moment you are holding the last 90 days of arrivals. Multiply by $0.023/GB-month for the bill.

full fine-tune   5.2 GB · 50,000 · 90  =  23.4 PB   ->  $538,000 / month
LoRA r=16         42 MB · 50,000 · 90  =   189 TB   ->    $4,347 / month
LoRA r=4 all    10.5 MB · 50,000 · 90  =    47 TB   ->    $1,087 / month
LoRA r=4 cross   5.2 MB · 50,000 · 90  =  23.4 TB   ->      $538 / month
textual inv.      32 KB · 50,000 · 90  =   144 GB   ->        $3 / month
encoder-based                      0   =       0    ->        $0

Put the top row next to the per-user compute bill. Full fine-tuning costs $538,000 a month in storage while the entire per-user compute bill for those same users is a few cents each. 23 petabytes is not a line item, it is a data-center project. That single row is why full fine-tuning is off the table before any quality argument is made.

Reason 2: serving topology. Topology here just means how the serving fleet is arranged — what lives on which card and what is shared between cards.

Multi tenancy thousands of adapters one base model derives it in full. The short version: a shared base with unmerged adapters — adapters kept as separate small matrices rather than folded into the base weights — lets one GPU batch a thousand different users through one matmul, at a 1.56% arithmetic overhead. Per-user full weights make batch size 1 a hard architectural constraint, and batch 1 costs about 1.9x per image in MFU alone.

Reason 3: legal lifecycle. A per-user adapter is a model derived from biometric data. Deletion requests must reach it, retention policies apply to it, and data residency rules — the requirement that data about a country’s residents be stored inside that country or region — apply to it as well.

Size changes how operable that problem is:

• A 42 MB artifact you can enumerate and drop is a workable version of the problem.
• A 5.2 GB artifact is the same problem with a 124x heavier tail.
• A 1 KB face embedding you never write to disk at all is the problem largely not existing.

Compliance surface is the sum of places a regulated artifact can be found, and therefore has to be tracked, secured and deleted. The encoder-based rung has the smallest compliance surface on the ladder, and that is a real argument for it independent of cost.

Assumptions in this stage.

State out loud:

• A $29 price, 9% refunds, $15 CAC, 6% of users needing $4 of support, 55% fleet utilization, 4% of jobs failing QC, 12% of users re-rolling, 50,000 users a day.
• Every one is a business input you would measure rather than derive. The whole P&L should be rebuilt when any of them moves.

Ask:

• The actual refund rate and its stated reasons. It is the largest controllable line in the P&L and the second-largest line overall — and “it doesn’t look like me” is the reason that makes the identity gate the highest-return component in the system.
• The retention period the business wants, and the one its lawyers will accept. 90 days versus 30 versus “delete on delivery” moves the storage bill by 3x and moves the Consent and likeness exposure by more than that.

Load-bearing:

• Compute is a small share of contribution (4.6%), so the design should be optimized for refunds, latency and compliance rather than for GPU dollars. It is why the chapter rejects DreamBooth on storage and topology rather than on training cost, why the identity gate is worth spending 20% more generation on, and why the encoder-based rung’s smallest-compliance-surface property counts as an argument at all.
• If compute were 40% of contribution instead of 4.6%, the ranking inverts and the cheap-but-worse rungs become the right default.

5. Multi-tenancy: thousands of adapters, one base model

One graphics card can serve many different users at once. Multi-tenancy is the general name for that: one shared piece of infrastructure serving many independent customers without mixing them up. Three mechanisms make it work — hot-swapping adapters is nearly free, keeping adapters unmerged costs 1.6% and buys 1.9x, and a three-tier cache means the swap almost never happens at all.

The diagram below has two independent entry points. The left branch starts when a user uploads 15 selfies and ends with an adapter in storage — it runs once per user. The right branch starts when someone asks for images and runs many times per user. They meet only at the adapter store.

Reading the left branch, top to bottom:

• Face QC is the upload gate from Consent and likeness. Four checks: there is a face at all, it is a live person rather than a photo of a screen, all 15 uploads are the same person, and that person is not a public figure.
• Reject with reason — anything that fails comes back with a stated reason, not a silent failure.
• Training queue holds jobs until eight can be co-batched, per The saving is not where people think it is.
• Adapter store is object storage holding one 42 MB file per user.

Reading the right branch:

• Generation queue is drained in priority by tier order, so paid users are not stuck behind free ones.
• Adapter cache looks in three places in ascending cost: VRAM -> NVMe -> S3, meaning graphics-card memory first, then the machine’s local solid-state disk, then object storage.
• Sampler pool is the fleet of cards that generate images.
• Restore + upscale is the post-processing pass that repairs faces and enlarges the image to 2048 pixels.
• Identity gate compares each output against the references. Anything below threshold takes the drop, resample path — discard that image and send the request back to the queue for another try. That loop back to the queue is why you generate 48 to deliver 40.
• Safety + watermark is the final step: a content check, plus an invisible watermark — a pattern embedded in the pixels that a detector can read and a viewer cannot see, so a generated image can later be identified as one.

flowchart TD
    U([Upload · 15 selfies]) --> QC{"Face QC<br/>detect · liveness ·<br/>all-same-person ·<br/>public-figure block"}
    QC -->|fail| REJ([Reject with reason])
    QC -->|pass| TQ[[Training queue<br/>co-batch 8 users]]
    TQ --> TR["LoRA trainer pool<br/>base frozen · shared<br/>26 jobs per 80 GB GPU"]
    TR --> AS[("Adapter store<br/>42 MB · user<br/>object storage")]

    P([Generation request]) --> GQ[[Generation queue<br/>priority by tier]]
    GQ --> CACHE{"Adapter cache<br/>VRAM -> NVMe -> S3"}
    AS -.-> CACHE
    CACHE --> SAMP["Sampler pool<br/>base resident 5.2 GB<br/>UNMERGED adapters<br/>batched LoRA kernel"]
    SAMP --> UP[Restore + upscale]
    UP --> IDG{"Identity gate<br/>ArcFace cos ≥ 0.45"}
    IDG -->|below| DROP[Drop · resample]
    DROP --> GQ
    IDG -->|pass| SAFE{Safety + watermark}
    SAFE --> D([Deliver 40])

    style QC fill:#bc6c25,color:#fff
    style SAMP fill:#1d3557,color:#fff
    style IDG fill:#2d6a4f,color:#fff
    style REJ fill:#9d0208,color:#fff

Why hot-swapping is free

The operation everyone worries about first is loading a different user’s adapter onto the card. Hot-swapping means changing which user’s adapter is loaded onto the card without restarting anything — and priced out, it is a rounding error.

Three storage layers matter, in ascending speed:

• Object storage over the network, roughly 1 GB/s.
• NVMe, the fast solid-state disk attached to the machine, roughly 5 GB/s.
• VRAM, the graphics card’s own memory, reached from host RAM over PCIe — the bus connecting the card to the rest of the computer — at roughly 50 GB/s on PCIe 5.

Each line below is just 42 MB / bandwidth. The last line is what to compare them against.

adapter size                                        42 MB

from object storage at ~1 GB/s                =  42 ms
from local NVMe at ~5 GB/s                    =   8.4 ms
host RAM -> VRAM over PCIe 5 at ~50 GB/s      =   0.84 ms

generation of one image                       =  4,060 ms

Take the worst case, a cold pull all the way from object storage: 42 / 4060 = 1.0% of a single image’s generation time — and a request generates 48 images, so the swap is 0.02% of the request.

Swap cost is a rounding error. That is the mechanism that makes per-user models viable at all, and it is worth stating as a ratio rather than as “it’s fast.”

Capacity tells the same story. How many adapters fit in VRAM alongside the base model:

80 GB card:   base model 5.2 GB + activations ~9 GB  ->  ~66 GB free

adapters resident, r=16 at 42 MB    ->  ~1,570 users hot simultaneously
full fine-tunes at 5.2 GB           ->  (80 - 9) / 5.2  =  13 users, no shared base

The two rows differ structurally, not just numerically. The LoRA row pays for the base model once and then fits 66,000 MB / 42 MB = 1,570 users in what is left. The full-fine-tune row has no shared base to subtract — each user is a 5.2 GB model — so it fits 13. That is a 120x difference in how many users one card can hold ready.

Why you keep the adapters unmerged

Here is the single most counter-intuitive trade in the design: pay 1.6% of extra arithmetic on purpose, and get 1.9x back.

Inference is the act of running a trained model to produce output, as opposed to training it.

At inference you have a choice. You can fold B·A into W — literally add the correction into the base weights, producing a new matrix W' — and serve a merged model with zero arithmetic overhead. It looks free.

Do not do it. Here is the overhead you would be saving:

per token, per matrix:

merged      y = W' x                       2·d^2       =  8.39 MFLOP
unmerged    y = W x + B(A x)               2·d^2 + 4dr =  8.39 + 0.131 MFLOP

overhead    4dr / 2d^2  =  2r / d          =  1.5625%

The unmerged path does three multiplies instead of one: the shared W x, then down-project with A, then up-project with B. The extra two cost 4dr and the shared one costs 2d^2, so the overhead is 4dr / 2d^2 = 2r/d — the same formula as the parameter ratio in Loras saving derived. Not a coincidence: both are counting the same two skinny matrices.

Now price what merging costs you. This is the part people miss.

merged:    each user needs their own W'   ->  batch size 1
           batch 1 at 4,096 tokens        ->  MFU ~25%  ->  247 TFLOP/s

unmerged:  one W shared by every user     ->  batch 64 across users
           batch 64                       ->  MFU ~48%  ->  475 TFLOP/s

           475 / (247 · 1.0156)  =  1.89x in favour of unmerged

The logic chain, spelled out: merging bakes one user’s identity into W', so W' is different for every user, so no two users can go through the same matmul, so the batch size is pinned at 1 — and batch 1 is where MFU is worst.

The last line divides the unmerged throughput by the merged throughput, after inflating the merged side by the 1.56% overhead you avoided. 475 / (247 · 1.0156) = 1.89.

Heterogeneous batching is the name for what the unmerged path enables: putting requests from different users into one batch, which is normally impossible when each user has different weights.

Paying 1.6% to keep the base matmul shared buys a 1.9x throughput win, so heterogeneous batching is not just possible, it is the cheaper option. That is the single fact that turns per-user models from a research demo into a product.

The function below writes that idea out in code. One big shared matrix multiply for the whole batch, then a small per-user correction gathered onto just the rows belonging to each adapter. Notice that W appears exactly once and outside the loop — that placement is the entire optimization. GEMM in the docstring is a general matrix-matrix multiply, the graphics card’s core operation.

def lora_forward(x, W, adapters, adapter_index, scale=1.0):
    """Batched heterogeneous LoRA. One shared base GEMM, per-adapter residual.

    x             (B, T, d)   activations for B requests from B different users
    W             (d, d)      frozen base weight, shared by every request
    adapters      list of (A, B_) pairs, A is (r, d), B_ is (d, r)
    adapter_index (B,)        which adapter each request in the batch uses

    The base term is one large matmul over the whole batch -- that is where the
    MFU comes from. The residual is 2r/d of the FLOPs, gathered per request.
    """
    y = x @ W.T                                  # shared: 2*d*d FLOPs per token
    for slot, (A, B_) in enumerate(adapters):    # residual: 4*d*r FLOPs per token
        rows = [i for i, a in enumerate(adapter_index) if a == slot]
        if not rows:
            continue
        h = x[rows] @ A.T                        # (n, T, r)   down-project
        y[rows] = y[rows] + scale * (h @ B_.T)   # (n, T, d)   up-project
    return y


def lora_overhead(d=2048, r=16):
    """Fraction of extra FLOPs from keeping adapters unmerged. Equals 2r/d."""
    return (4 * d * r) / (2 * d * d)

A kernel is a single hand-written GPU routine; a grouped GEMM does many small independent matrix multiplies in one launch. Production kernels do the loop that way instead of as a Python loop, but the FLOP accounting is exactly the above and that is what the interview is asking for.

Cache policy

Cache design follows from user behaviour, and this product’s behaviour is extreme. Users generate in bursts: an initial batch of 40, then a handful of re-rolls over the next twenty minutes, then nothing for months. Almost every access to a given adapter happens inside one twenty-minute window.

That shape suggests three tiers:

• VRAM tier: LRU eviction — least recently used, the rule that discards whichever entry has gone longest untouched — with a 30-minute TTL, a time-to-live, meaning an entry is dropped 30 minutes after it arrives regardless of whether there is space. Sized to ~1,500 adapters.
• NVMe tier: the last 7 days of adapters, ~15 TB at 50k users/day. Catches the “came back the next morning” case at 8 ms.
• Object store: everything else, with a lifecycle rule that deletes at 90 days and emits an audit record, because this artifact is biometric-derived.

The VRAM tier is sized at ~1,500 adapters and will never come close to using it. Work out how many it actually holds:

11 concurrent requests per GPU, 20-minute sessions
  ->  11 · 3 sessions per hour        =  33 adapter arrivals / hour
30-minute TTL holds half an hour of arrivals
  ->  33 · 0.5                        =  ~17 adapters resident

Seventeen, against capacity for 1,500. The TTL binds, not the capacity — meaning the TTL, not a shortage of space, is what decides when something gets evicted.

That is the point rather than an oversight. It means every re-roll inside a session finds its adapter already in VRAM, so the hit rate — the fraction of lookups served without going to a slower tier — sits above 0.9 by construction rather than by tuning. Nobody has to sweep a cache size.

Assumptions in this stage.

State out loud:

• An 80 GB card, base model 5.2 GB, ~9 GB of activations, MFU of 25% at batch 1 and 48% at batch 64, 42 MB adapters, 11 concurrent requests per GPU, 20-minute sessions, a 30-minute TTL.
• These size the cache and set the 1.9x. Different hardware moves them together.

Ask:

• Whether adapters may be cached outside the region they were trained in. Data-residency rules can forbid the NVMe tier on a machine in another jurisdiction. That is a constraint on the architecture, not on its parameters — you delete a whole tier, you do not retune one.

Load-bearing, and this one is a privacy assumption as much as a performance one:

• Adapters from different users can safely share a graphics card and a batch. The 1.9x, the 26-jobs-per-card figure, and the entire multi-tenant design rest on it.
• That safety requires the per-request adapter index in the code above to be never wrong. A mis-routed adapter row generates one user’s face for another user’s request — which is a biometric data leak, not a quality bug. That is why the index deserves an assertion and a test rather than a comment.

Also load-bearing:

• The object-store lifecycle rule and the deletion path in Consent and likeness actually cover the cache tiers. An adapter that survives in VRAM or on NVMe after the object copy is deleted is an undeleted biometric artifact, and nothing in the system will tell you.

6. Metrics: identity and prompt following, as separate axes

Serving is solved; whether the outputs are any good is not. The measurement stack has three jobs: make “does it look like them” into a number that means something, measure how that number trades against prompt following, and run the one gate whose return on investment is 19 to 1.

Measuring identity

Making the identity metric interpretable mostly means refusing to report it without the two anchors that give it a scale.

The procedure, in three steps:

1. Push the generated face and each reference selfie through a face-recognition model — ArcFace and AdaFace are the standard open ones. Each comes back as an embedding: a vector of a few hundred numbers, trained so that two photos of the same person point in nearly the same direction.
2. Average the reference embeddings into one vector representing “this person.”
3. Take the cosine similarity between the generated embedding and that average.

That gives you a number between 0 and 1. And on its own, that number tells you nothing.

A raw cosine is meaningless without its anchors. You have to report it against the two distributions that define the scale, measured on the same encoder and the same image pipeline.

One term in the block below: FAR is the false accept rate, the fraction of different-person pairs a verification system wrongly calls a match. FAR = 1e-4 is the operating point where one impostor in ten thousand gets through, roughly where phone face-unlock is set.

The first three lines are the scale. The last two are the measurements.

different people, unconstrained photos       mean cos  ~0.02   (p99 ~0.22)
same person, two different photos            mean cos  ~0.65
verification threshold at FAR = 1e-4         cos        0.36

our LoRA outputs                             mean cos   0.68
our encoder-adapter outputs                  mean cos   0.52

Now 0.68 can be read. It sits above the same-person anchor of 0.65 and far above the 0.36 verification threshold.

0.68 means “as similar to the references as two real photos of the same person are to each other,” and that is the only reading of 0.68 that means anything.

Quoting an identity score without those anchors is the same error as quoting PR-AUC — the area under a precision-recall curve, a standard classifier score — without saying how rare the positive class was. The same PR-AUC number means something entirely different at 50% prevalence and at 0.1% (Pr auc vs roc auc under heavy imbalance and the comparison pr auc is not allowed to make).

The tradeoff, measured

Identity and prompt following move in opposite directions along every training knob — that is the central trade of the chapter, and its measured shape dictates which checkpoint to ship.

The measurements below come from a fixed panel of 30 users — the same 30 people’s uploads and reference sets are rerun on every recipe, so changes are comparable rather than confounded by who happened to be in the sample.

Three columns need defining:

• ArcFace cos is the identity metric just derived. Higher is better. Anchor it at 0.65 (same person) and 0.02 (different people).
• VQA prompt adherence decomposes each prompt into yes/no questions (“is there a suit?”, “is the background plain grey?”) and asks a vision-language model each one, scoring the fraction answered correctly. Higher is better.
• Background-leak rate is the fraction of outputs in which the user’s own training-photo background shows up despite the prompt asking for something else. Lower is better.

Read the first four rows as a sweep of training steps at fixed rank 16, then the last two as a sweep of rank at fixed 1,200 steps.

Two readings of that table.

The step sweep shows the trade going one way only. From 400 to 2,000 steps, identity climbs 0.41 → 0.58 → 0.68 → 0.72, and adherence falls 0.79 → 0.76 → 0.71 → 0.54 while background leak explodes 3% → 9% → 21% → 58%. Past 1,200 steps you buy 0.04 of identity and pay 0.17 of adherence for it. That is the knee.

The rank sweep shows something less obvious. Rank 4 at 1,200 steps (0.61 identity, 0.75 adherence, 11% leak) sits close to rank 16 at 800 steps (0.58, 0.76, 9%). Lowering the rank behaves like training for fewer steps.

Rank is a regularizer, not just a capacity dial. The low-rank constraint limits how much of the training set the adapter is able to memorize, no matter how long you run it. That is why cross-attention-only rank-4 adapters stay a legitimate rung rather than merely a cheap one — and why rank 64 in the last row is worse on both of the columns that matter.

Which checkpoint to ship

Two terms. A checkpoint is a saved copy of the weights at some point during training; choosing which one to ship is its own decision. Overfitting is learning the training examples themselves rather than the concept they illustrate.

Selecting on identity alone selects the overfit checkpoint every time. The argument is two lines: identity is monotone in steps and adherence is not, so the identity-maximizing checkpoint is always the last one, and the last one is always the most overfit.

Pick the knee, not the maximum.

The gate that pays for itself

The knee gives you the checkpoint to ship; the identity gate protects every image generated from it. The gate is simple: score every generated image against the references with the same cosine as above, and drop the failures before the user ever sees them. What makes it interesting is the return.

The block below has a cost half and a benefit half. Compare the last two lines.

at threshold cos ≥ 0.45   ->  14% of outputs dropped
                          ->  generate 48 to deliver 40   (+20% generation cost)

8 extra images · 4.06 s   =  32.5 GPU-s  =  $0.0226 at 100% utilization
at the fleet's 55%                       =  $0.041 per user

measured effect on refunds: 9.1% -> 6.4%
                            0.027 · 29 = $0.78 of refund saved per user
                                          vs $0.041 spent

Trace both halves.

Cost. A 14% drop rate means you need 40 / 0.86 = 47 generated to land 40; round to 48. The 8 extra images cost 8 · 4.06 = 32.5 GPU-s, which is 32.5 / 3600 · $2.50 = $0.0226 of pure GPU time, or $0.0226 / 0.55 = $0.041 at the fleet’s real utilization.

Benefit. Refunds fall 2.7 percentage points, from 9.1% to 6.4%. Each avoided refund returns the full $29, so 0.027 · 29 = $0.78 per user.

$0.78 / $0.041 = 19. Nineteen to one.

One methodological point, because it is the kind of thing an interviewer catches. Price the gate at the same 55% utilization that the rest of The bill’s bill uses. At 100% utilization the cost is $0.0226 and the return looks like 35:1 — but that is a fleet you do not have, and quoting a return against imaginary hardware is how optimizations get funded that never pay.

This is the single highest-return component in the system, and it exists for a specific reason: the refund reason and the automated metric are the same thing — “it doesn’t look like me.”

Online metrics

Production adds its own signals, and each one is telling you something specific. A cohort in the last row is a group of users sharing some property, here a demographic one:

Two notes on experimentation. An A/B test is the live experiment where some users get the current recipe and some get the new one.

Randomize by user, which is unusually easy here because there is only one training run per user — there is no risk of a user seeing both arms.

Hold the panel fixed when comparing recipes. A 30-user evaluation panel with hand-labeled reference sets, rerun on every recipe change, is worth more than an online test you have to wait a week for.

Assumptions in this stage.

State out loud:

• An identity threshold of 0.45, a 14% drop rate, a 30-user panel, and the measured refund move from 9.1% to 6.4%.
• The threshold is the one to sweep first — and Demographic bias and why one metric cannot measure it argues it should not be a single global number at all.

Ask:

• Whether the 30 panel members consented to their photos being used as a permanent internal benchmark, and on what terms. A fixed panel of real faces retained indefinitely for engineering purposes is a biometric dataset with its own consent and retention obligations. It is the one most teams forget they are holding.

Load-bearing:

• ArcFace cosine against the references tracks what a user means by “it looks like me.” It is the release gate, the per-image serving gate, and the bias diagnostic, all at once — so if it is wrong, it is wrong in three places simultaneously.
• Demographic bias and why one metric cannot measure it shows it is wrong for some users. That is why the human study there is not optional.

7. Failure modes

None of what goes wrong here is bad luck: every failure traces back to the missing term in the loss from Ml objective. The first is the one to be able to derive on demand; the rest come with traces you could read off a real run.

Background and clothing overfit — the one to explain mechanistically

It follows in two steps from the absent term in the loss. Start from a realistic training set and a realistic prompt, and read what comes out.

TRAINING SET   15 selfies, all taken in the same apartment over one weekend
               13 of 15 have the same kitchen backsplash
               11 of 15 wear the same grey hoodie
               all 15 shot on the same phone's front camera

PROMPT         "<tok> person, professional headshot, studio lighting,
                plain grey backdrop, navy suit"

OUTPUT at 2000 steps
  8/8   kitchen backsplash visible behind a studio-lit subject
  5/8   grey hoodie collar under the suit jacket
  8/8   wide-angle selfie lens distortion, despite "studio lighting"
  ArcFace cos 0.72     VQA adherence 0.54

The mechanism, stated the way it should be in an interview: the loss rewards any weight change that lowers reconstruction error on those 15 images, and it cannot distinguish the face from anything that co-occurs with the face.

Follow it in two steps.

Step one: what is <tok> correlated with? It appears in 15 of 15 captions. The face appears in 15 of 15 images — and the backsplash appears in 13 of 15. From the loss’s point of view those are nearly the same signal. In information-theory terms, <tok> has high mutual information — a measure of how much knowing one thing tells you about another — with the backsplash almost as much as with the face.

Step two: what does gradient descent do with that? It makes <tok> predict whatever <tok> is correlated with. Nothing in the objective ranks “bone structure” above “wallpaper,” so both get encoded.

The result is that <tok> learns the joint distribution of the training set — the distribution over everything in those photos together: face and kitchen and hoodie and lens. Not because the method failed, but because that is exactly what it was asked to learn.

The diagram below is that argument in five boxes. It starts from 15 selfies shot in the same room, same hoodie, same camera; the loss says only reduce error on these 15 images, so encoding bone structure and encoding the backsplash are rewarded equally; both route through the one trigger token that appears in every caption; and the result is that identity goes up with steps while prompt following goes down with steps.

flowchart TD
    S["15 selfies<br/>same room · same hoodie<br/>same camera"] --> L["Loss: reduce error<br/>on THESE 15 images"]
    L --> A["encode bone structure<br/>rewarded"]
    L --> B["encode the backsplash<br/>rewarded EQUALLY"]
    A --> T["trigger token tok<br/>appears in every caption"]
    B --> T
    T --> O["tok = the joint distribution<br/>of the training set,<br/>not the person"]
    O --> F1["identity UP<br/>with steps"]
    O --> F2["prompt following DOWN<br/>with steps"]

    style L fill:#1d3557,color:#fff
    style B fill:#9d0208,color:#fff
    style O fill:#bc6c25,color:#fff
    style F1 fill:#2d6a4f,color:#fff
    style F2 fill:#9d0208,color:#fff

Five fixes, ranked by how much they move the number. The ranking is not arbitrary — the ones that change what the objective rewards beat the ones that only change the data.

1. Caption the nuisance variables. A nuisance variable is something present in the training photos that you do not want attached to the person.

Instead of "a photo of <tok> person", write "<tok> person wearing a grey hoodie in a kitchen". Now the backsplash has its own handle. Cross-attention lets the token “kitchen” claim that spatial attention mass, so the gradient pressure on <tok> to encode it drops — the model no longer needs <tok> to explain the wallpaper, because “kitchen” explains it better.

Largest single effect in the list, and it costs nothing but a VLM — vision-language model — captioning pass over 15 images.

2. Face-masked loss. A segmentation mask marks which pixels belong to which thing. Weight the per-pixel loss by a face mask, so errors on the face count for more than errors elsewhere. With the face at ~18% of the frame and background weighted 0.2, the background’s contribution to the gradient falls about 5x.

3. Prior preservation. Generate ~200 images of “a person” from the base model itself, and train on them alongside the user’s photos — captioned with the generic class word “person” rather than the trigger token.

This holds the word “person” in place while <tok> moves. Without it, “person” collapses onto this one user, which is the failure that makes a group-shot prompt come back as eight copies of the subject.

4. Lower rank, or fewer steps. Straight from the The tradeoff measured table: rank 4 cuts background leak from 21% to 11%.

5. Augmentation — synthetically varying the training images: random crop, horizontal flip, segmentation-based background replacement across the 15 photos. Helps least, and the reason is the same reason the list is ordered this way: it changes the data but not what the objective rewards.

The rest, with traces

Three more failures, each in the same form: what you observe, what causes it, and what guards against it.

The first one is about a few bad images in an otherwise good batch, which makes it a serving-time problem rather than a training-time one.

IDENTITY DRIFT ACROSS A BATCH
  40 outputs, ArcFace cos per image:
    0.71 0.69 0.72 0.68 0.31 0.70 0.29 0.67 ...
  two clear outliers at 0.30 -- not "slightly off," a different person
  cause: high guidance pushing latents toward the base model's prototypical
         face prior, which overrides the adapter at extreme w
  guard: identity gate (§6); cap guidance at 4.5 ([The tradeoff derived rather than asserted](/learn/genai-system-design/09-text-to-image/#43-the-tradeoff-derived-rather-than-asserted))

Notice what the numbers look like: not a gentle slide from 0.70 down to 0.60, but two images sitting at 0.30 while the rest are fine. That gap is the tell. 0.30 is barely above the different-people anchor of 0.02 — those are not “slightly off,” they are somebody else.

The two guards are the identity gate of The gate that pays for itself, which catches these per image at serve time, and a hard cap on the guidance scale at 4.5. The cap follows from what The tradeoff derived rather than asserted derives: high guidance is mode-seeking, and the mode it seeks is the base model’s generic prototypical face rather than this user’s.

The next failure is the opposite shape — every image is wrong in the same way.

PROMPT COLLAPSE
  prompt  "<tok> person hiking on a mountain trail, backpack, wide shot"
  output  head-and-shoulders portrait, indoors, 8/8
  cause:  all 15 training images are head-and-shoulders selfies, so <tok>
          has absorbed the framing along with the face
  guard:  prior preservation; caption framing explicitly in training
          captions; report a separate "off-distribution prompt" eval slice

Two terms from that guard line. An eval slice is a named subset of the evaluation set, reported on its own rather than averaged into the total. An off-distribution prompt is one that asks for something none of the training photos showed — a mountain trail, a full-body shot.

The two go together for a reason: off-distribution prompts are exactly where this failure hides, and exactly where an aggregate metric will not show it. Average the mountain-trail prompts in with forty studio-portrait prompts and the failure disappears into the mean.

The third failure damages a word rather than an image.

CLASS BLEED
  prompt  "<tok> person and two colleagues in a meeting room"
  output  three people who all look like the subject
  cause:  fine-tuning shifted the class token "person" toward this identity
  guard:  prior preservation is the direct fix -- it is exactly what the
          regularization images are for

Class bleed is the name for it. The class token — the ordinary English word “person” — has been dragged toward this one identity, so every person the model draws becomes them. Notice the damage is not to <tok>; it is to a word the model already knew and that other prompts depend on.

The regularization images are the ~200 generic pictures of “a person” that prior preservation trains on alongside the user’s photos. They exist precisely to hold that word in place while <tok> moves.

The table below collects every failure in this section plus the ones that live at the system edges — upload validation, caching, deletion. Read the middle column as “what alerts you” and the right column as “what you build.”

Two rows use terms worth pinning:

• Selfie-lens distortion is the exaggerated nose and narrow ears a phone’s wide-angle front camera produces at arm’s length. A focal-length classifier estimates from an image which kind of lens it looks like it was shot on, which is what makes this failure automatically detectable rather than only visible to a human.
• The last row’s audit query is the deletion check Consent and likeness insists you write before you need it.

Assumptions in this stage.

State out loud:

• A face occupying ~18% of the frame with background weighted 0.2, ~200 prior-preservation images, a guidance cap of 4.5. All are recipe settings measured on the panel.

Ask:

• Whether the product will accept richer captions on user uploads. Captioning the nuisance variables is the largest single fix, and it means running a captioning model over private photos — which is another processor touching biometric data, and therefore a consent question rather than only an engineering one.

Load-bearing:

• These failures are consequences of the missing term in the loss, not of insufficient training. Every fix in the ranked list follows from that reading.
• If the model were simply undertrained, the fix would be more steps — and the The tradeoff measured table shows more steps makes every one of these failures strictly worse.

8. Demographic bias, and why one metric cannot measure it

A measurement trap sits under this whole topic: the identity metric is itself less accurate for some users than others, so the metric cannot tell you how much of a measured gap is the generator’s fault. Two causes need separating, and only some of the numbers deserve trust.

Slice the identity metric and the download rate by skin-tone bucket and by gender presentation.

Two terms. The Monk scale is a ten-point skin-tone scale designed for exactly this kind of measurement. Gender presentation is how a person appears rather than how they identify — the only thing an image metric can see, and worth naming precisely so nobody confuses the two.

The table below is an illustrative shape of what you find, not a measurement from a real system. Read across the bottom row: every column is worse, and by a lot.

There are two distinct causes behind that bottom row, and conflating them is the mistake this section exists to prevent.

Cause 1: the metric is biased

ArcFace is trained on a corpus skewed toward light-skinned faces and has measurably higher error rates on darker-skinned faces.

So some of that 0.59 is the encoder being worse at the measurement, not the generator being worse at the job. You are reading a ruler that is shorter in some places.

You cannot separate model bias from metric bias using the metric. No amount of slicing the ArcFace numbers more finely tells you which of the two you are looking at, because every slice is measured with the same suspect instrument.

The only way out is a second measurement built differently: a human same-person/different-person study on a demographically balanced panel, run against the same outputs. If humans say the identity is fine and ArcFace says 0.59, then the gate threshold is the bug, not the generator.

Cause 2: the base model’s prior is biased

A model’s prior is what it tends to produce before your conditioning pushes it anywhere — its default.

“Professional headshot” in the pretraining corpus skews toward particular lighting setups, hair rendering and styling. A three-point lighting setup calibrated on light skin under-exposes dark skin. Hair textures that are rare in the corpus render as mush.

This one shows up in the download rate, which is a human judgment and does not route through ArcFace at all. The download column falling from 7.1 to 4.2 is the honest signal in that table, precisely because no biased model was involved in producing it.

What to do about it

Report the download rate and the refund rate by bucket. Do not report the ArcFace score by bucket. Those two are the metrics whose measurement instrument is a person.

Then fix the two causes separately:

• For the biased prior: a balanced fine-tuning set for the base aesthetic model.
• For the biased metric: per-bucket gate thresholds instead of a single global 0.45, calibrated — meaning set by reference to an external ground truth, here the human study rather than the metric being questioned.
• For both: a launch gate requiring that no bucket’s download rate falls more than 15% below the best bucket.

Assumptions in this stage.

State out loud:

• The Monk scale as the bucketing, three buckets rather than ten, a launch gate at 15% below the best bucket.
• The bucket count and the gate threshold are policy choices with real consequences. Set them with someone who owns that policy.

Ask, never assume:

• Whether you are permitted to collect the demographic labels this section requires. In several jurisdictions, inferring skin tone or gender from a user’s photo is itself sensitive-category processing, and self-reported labels need their own consent and their own retention rule.
• You cannot measure this fairness gap without holding that data, and holding it is not free. The answer decides whether the measurement happens on live traffic or only on a consented panel.

Load-bearing:

• You cannot separate model bias from metric bias using the metric. It is why the human same-person study exists, why per-bucket thresholds must be calibrated against that study rather than against ArcFace, and why the honest columns are the ones a person produced.
• If ArcFace were uniformly accurate across skin tones, the whole section reduces to one sliced dashboard and the human study is wasted money.

9. Consent and likeness

The product ships a model of a specific person’s face, which makes consent, retention and deletion design constraints rather than paperwork. Four controls, and the ordering matters — the first three prevent an unlawful artifact from ever being created, and the fourth destroys one that was.

Control 1: all-same-person check at upload

Compute pairwise face-embedding cosines across the 15 uploads and require that they all agree with each other.

Two things fail this check, and the second is the more common one. A scraped celebrity set fails often, because the photos span different eras and photographers. And a mixed set — the user plus their partner — fails it too, which is the honest mistake rather than the malicious one.

Reject with a stated reason. The alternative is silently training a chimera: a model that has averaged two people into one face belonging to neither.

Control 2: public-figure gallery match, at upload and again on outputs

A gallery is a stored set of face embeddings for known public figures.

Run the check in both places, because they catch different things:

• At upload, it stops the run before it costs anything — someone trying to train on a celebrity.
• On outputs, it catches the opposite case: the base model’s prior dragged an ordinary user’s generated face toward a celebrity’s.

Control 3: liveness or attestation

Somewhere the user has to assert they are the subject. There are two strengths of that assertion:

• Liveness is a check that the face in front of the camera is a live person rather than a photograph, a screen or a mask. A selfie-video liveness check is the strong version, and it costs real money per user.
• Attestation is the weaker act of the user simply declaring it. A checkbox costs nothing.

State which one you chose and why. This is a policy decision with a price tag attached, and the interview wants to see that you know it is not a technical question.

Control 4: deletion that actually deletes

The adapter is derived from biometric data, so a deletion request has to reach four things:

1. The source images.
2. The adapter.
3. Every cached copy of the adapter — in VRAM and on NVMe, not just in the object store.
4. The generated outputs.

And it must emit an audit record: a durable log entry that can later prove the deletion happened.

Write the audit query first and run it as a scheduled test. The failure mode here is silent — nothing errors when an adapter survives a deletion request. You find out from a regulator.

Assumptions in this stage, and this is the block to get right. These are the assumptions where being wrong is a legal problem rather than a tuning problem, so they are marked accordingly.

State out loud: a 90-day adapter and output retention, 30-day source-image retention, an all-same-person threshold on pairwise cosine, and a public-figure gallery you have the right to hold. These are the numbers on the table; they are chosen, not derived, and every one of them should be shown to counsel before it is shipped.

Ask, never assume, and none of these has a technical answer:

• What exactly the user consented to, and whether that consent covers each use separately. Training an adapter, generating the 40 images, retaining the adapter for re-rolls, retaining the source photos, and using any of it to improve the base model are five distinct purposes. Consent to the first is not consent to the fifth. Bundled consent is the single most common defect in products of this shape.
• Whether consent can be withdrawn, and what withdrawal is required to undo. If withdrawal must unwind a base model that was trained on the user’s photos, that is not a deletion job, it is a retraining project — which is the strongest possible argument for never mixing user uploads into base-model training in the first place.
• The retention periods your jurisdiction permits, which may be shorter than the 90 days Per user economics end to end prices, and which may differ between the source photos, the adapter and the outputs.
• Where the data may physically live, since data-residency rules can forbid the NVMe cache tier of Multi tenancy thousands of adapters one base model on a machine in the wrong region.
• Whether minors can reach the product at all, because biometric processing of a minor is a categorically different legal regime in most places and the answer is an age-gate requirement, not a model change.

Load-bearing, and each one invalidates the design rather than degrading it:

• That the adapter is in scope for deletion. It does not look like a photograph, so it is the artifact teams forget. If it is in scope and you do not delete it, you have retained biometric data after a valid erasure request. Nothing in the system will alert you; the deletion audit query is the only thing that will.
• That the deletion path reaches every tier. The object-store lifecycle rule is the easy one. VRAM and NVMe caches, the generated outputs sitting on the CDN — the content delivery network, the fleet of caches near users that actually serves the images — backups, and any analytics copy are the ones that survive. A deletion that covers four of six locations is not a partial success — legally it is a failure with a paper trail saying you thought you had handled it.
• That user uploads never enter base-model training. The entire deletion story depends on the user’s data living only in artifacts you can enumerate and drop. The moment a selfie contributes a gradient to the shared base, deletion becomes unachievable and the product’s privacy claims become false.
• That consent is specific, informed and withdrawable rather than bundled into terms of service. Every control above is downstream of it: if the consent does not hold, the adapter should never have existed, and no amount of correct deletion machinery repairs having trained it.
• That you have the right to hold the public-figure gallery you match against. It is itself a biometric database, assembled about people who did not upload anything to you. The control that protects one group of people is built out of data about another group, and that is not a technicality — it is the reason the gallery needs its own lawful basis, its own retention rule and its own audit.

10. Alternatives considered and rejected

Every design below is worth naming, and each one dies to a specific number or legal fact. The middle column matters as much as the right one. An alternative you cannot state the appeal of is one you have not actually considered, and an interviewer will hear the difference. The first four rows are the major forks; the rest are the tuning decisions that go wrong most often.

11. Interviewer pushback

These are the questions this design is most often asked, what each one is probing, and the answer in the form you would actually say it out loud.

Use them as a recall test: cover the answers, read a question, and see whether you can reconstruct the arithmetic rather than recall the conclusion.

“Why LoRA and not a full fine-tune? Full fine-tune gets better identity.” Testing: whether you can price a decision instead of asserting it. It does — 0.73 versus 0.68 on ArcFace. It also costs 5.2 GB per user against 42 MB, which at 50,000 users a day and 90-day retention is 23 petabytes versus 189 terabytes, roughly $538k a month against $4.3k. And it forces batch size 1 at serve time, because two users no longer share a weight matrix, which costs about 1.9x per image in MFU alone. Five hundredths of an identity point does not buy either of those.

“Derive LoRA’s parameter saving.” Testing: whether the number is memorized or reconstructible. The update to a d · d matrix is constrained to rank r, so dW = B·A with B at d · r and A at r · d. That is 2dr parameters against d^2, a ratio of 2r/d. With d = 2048 and r = 16 that is 32/2048 = 1.56%. Across 320 attention matrices, 1.34B trainable becomes 21.0M, which is 42 MB in fp16 against 5.2 GB for the whole model. And the same 2r/d shows up again as the FLOP overhead of keeping the adapter unmerged at inference, because it is the same two skinny matrices.

“So LoRA makes training 100x cheaper?” Testing: whether you actually know what LoRA saves. It is a trap. No — about 23% cheaper. You still forward through all 2.6B parameters and still backpropagate through every layer; you only skip computing weight gradients for the frozen matrices. What LoRA saves is memory: 5.5 GB of state against 41.6 GB, with the frozen base shared. That is what lets 26 training jobs share one 80 GB card, and co-batching eight users raises MFU from 22% to 48%, which is where the real 2.2x cost reduction comes from. LoRA’s saving is memory, and memory buys batching.

“How do you serve a thousand different users from one GPU?” Testing: the multi-tenancy mechanism. Base model resident once, adapters unmerged and hot-swapped through a VRAM/NVMe/S3 cache. Swapping is free relative to generation: 42 MB is 0.84 ms over PCIe, 8 ms from NVMe, 42 ms from object storage, against 4,060 ms to generate one image — and a request generates 48. VRAM holds about 1,570 adapters alongside the base. And the base GEMM is shared across the batch, so 64 requests from 64 different users go through one matmul with a per-request rank-16 residual at 1.56% overhead.

“Why not merge the adapters? That’s free.” Testing: whether you notice the second-order effect. Because merging makes every user a different model, which pins batch size at 1. Batch 1 runs at roughly 25% MFU and batch 64 at 48%, so merged is about 1.9x more expensive per image after accounting for the 1.56% you saved. Paying 1.56% to keep the base matmul shared is the best trade in the design.

“Walk me through the cost per user.” Testing: end-to-end arithmetic, live. Training is 1,200 steps at batch 4, 13 TFLOP per image, so 62.5 PFLOP; co-batched eight ways at 475 TFLOP/s that is 132 GPU-seconds, $0.09. Generation is 26.8 TFLOP per pass, 60 passes with CFG, 1,608 TFLOP per image, 4.06 seconds each; 48 images to deliver 40 is 195 GPU-seconds, $0.14. Upscaling adds under a cent. That is $0.24 at full utilization, $0.43 at 55%, and $0.47 after reruns and re-rolls. Storage and egress add under two cents. Call it $0.49 against a $29 price.

“98% margin. So you’re done?” Testing: whether you know which number is actually load-bearing. No, and the P&L says why. Against $29, refunds at 9% are $2.61 and acquisition is around $15, so contribution is about $10.66 and compute is 4.6% of it. Halving the GPU bill moves contribution 2.3%; cutting refunds from 9% to 5% moves it 11%. Refunds here are almost entirely “it doesn’t look like me,” so the identity gate — which costs $0.041 at the fleet’s 55% utilization and saves $0.78 — is worth roughly five times any compute optimization. The ladder choice still matters, but through fleet storage, latency, and compliance surface, not through per-user dollars.

“Every image has the user’s kitchen in it. What happened?” Testing: mechanism, not vocabulary. The loss rewards any weight change that lowers reconstruction error on 15 images, and nothing in it separates the face from what co-occurs with the face. The trigger token is in every caption and the backsplash is in 13 of 15 photos, so the token becomes the model’s best handle on the backsplash too. Fixes in order of effect: caption the nuisance variables so “kitchen” can claim that attention mass instead, mask the loss toward the face region, add prior-preservation images, and lower rank or stop earlier. Rank matters here — at rank 4 the leak rate is 11% versus 21% at rank 16, because low rank limits how much of the training set the adapter is able to memorize.

“Just train longer to get better identity, then.” Testing: whether you select checkpoints on one axis. Identity is monotone in steps and prompt adherence is not. At 2,000 steps identity reaches 0.72 but adherence falls to 0.54 and 58% of outputs leak the training background, so every “hiking on a mountain trail” prompt comes back as an indoor portrait. Selecting on identity picks the overfit checkpoint every time. I select on the knee of the joint curve, which lands at 1,200 steps and rank 16.

“You report ArcFace cosine 0.68. Is that good?” Testing: whether you know a metric needs anchors. Only against the two distributions that define the scale on the same encoder: different people average about 0.02, and two real photos of the same person average about 0.65. So 0.68 means the outputs are about as similar to the references as two genuine photos of the person are to each other. Without those anchors 0.68 is not a number — the same error as quoting PR-AUC without the prevalence.

“Your identity scores are lower for darker-skinned users. Is the generator biased?” Testing: whether you conflate metric bias with model bias. Partly, and I cannot tell how much from that metric, because ArcFace itself has higher error rates on darker-skinned faces — some of the gap is the ruler, not the thing being measured. So I would look at the metrics whose instrument is a person: downloads per user fell from 7.1 to 4.2 and refunds went from 5.4% to 11.3%, and neither of those routes through ArcFace. Those say there is a real gap. Then I would run a human same-person study on a balanced panel to separate the two, fix the base model’s styling prior with balanced fine-tuning data, and calibrate per-bucket gate thresholds against the human study rather than against ArcFace. And I would never let the launch gate read an aggregate.

“Can you make this instant?” Testing: whether you know the whole ladder, not just your pick. Yes, by dropping to the encoder-based rung — project a face-recognition embedding into cross-attention through an adapter trained once offline. Zero per-user training, zero storage, ten-second turnaround. The cost is identity: about 0.52 against 0.68, and the ceiling is structural, since a recognition embedding is trained to be invariant to exactly the within-person variation that makes someone recognizable. So I would ship it as an instant preview that converts the user and run the LoRA for the delivered set. It also has the smallest compliance surface on the ladder, which is a real argument on its own.

“A user asks you to delete their data. What exactly do you delete?” Testing: whether you treat the adapter as data. The uploads, the adapter, every cached copy of the adapter in VRAM and on NVMe, and the generated outputs — and the deletion has to emit an audit record. The adapter is a model derived from biometric data, so it is in scope in most jurisdictions and people forget it because it does not look like a photo. I would write the audit query that joins the deletion log against the adapter store and run it as a scheduled test, because this failure is silent: nothing errors when an adapter outlives a deletion request, and you find out from a regulator rather than from a log line.

The assumption ledger

Here, collected in one place, is every assumption the chapter has leaned on — so you can state the design’s foundations in twenty seconds and say what replaces the design when each one fails.

Each row is sorted into one of three bins, the same three used in ch 01:

• State it — you are free to pick, and being wrong costs you a re-derivation.
• Ask it — the answer changes the architecture, so it is worth an interviewer’s time.
• Load-bearing — if it is wrong, the design is not suboptimal, it is invalid.

The privacy rows are marked and grouped first, because in this product they are the ones where being wrong is a legal problem rather than a tuning problem. Read the last column of those first five rows as a single sentence: there is no version of this system that survives getting them wrong.

The sentence that makes this visible to an interviewer: “This design rests on four things. One, that nothing in the training loss separates the face from the kitchen behind it — which is why I stop at the knee rather than at maximum identity, and why every guard in the recipe exists. Two, that a frozen shared base is what makes co-batching and heterogeneous serving possible, so LoRA’s real saving is memory, not FLOPs. Three, that compute is under 5% of contribution, so the identity gate that cuts refunds is worth five times any GPU optimization. And four — the one that is a legal problem rather than an engineering one — that the per-user adapter is biometric-derived data: it is in scope for deletion, the deletion has to reach the VRAM and NVMe caches and not just the object store, user uploads must never touch the base model, and the consent that authorized all of it has to be specific and withdrawable rather than buried in terms of service. Get the first three wrong and the product is expensive. Get the fourth wrong and there is no product.”

Next: 11 — Text-to-Video — the same problem with a temporal axis, where the compute blow-up decides the product shape before any modelling choice does.