You can implement differential privacy flawlessly and still leak the people you most wanted to protect.
Not through a bug in your training loop. Through a single line of preprocessing that runs before the model sees anything. It passes every test. Your privacy accountant prints a small, reassuring number. And that number is describing something other than your patients.
This is really a lesson about adaptive data. How you reshape your data before training can rewrite the guarantee you thought you had, and nothing in your code will warn you.
Differentially private SGD makes one promise: no single person can move the model much. It keeps that promise with two steps on every batch. First it clips each example's gradient to a fixed length C, so one record can only push the weights so far. Then it adds noise sized to C. The privacy accountant watches C and the noise and reports a budget, epsilon. Smaller epsilon means stronger privacy.
The whole thing rests on one assumption. One training example is one person. Clip the example, and you have clipped the person.
Hold onto that sentence. Everything breaks when it stops being true.
Real medical data is lopsided. Most people are healthy, few are sick, and the model learns to predict "healthy" and call it a day. The usual fix is SMOTE: it invents new minority examples by drawing a line between two real minority records and dropping a synthetic point somewhere on it.
That fix quietly voids the assumption above. Each synthetic row is built from two real people. One real patient can seed dozens of synthetic rows. So when DP-SGD clips a row, it is no longer clipping a person. It is clipping one of the many shadows that person casts.
Move the slider below. Watch one patient turn into many rows.
At six times oversampling a typical minority patient is spread across roughly eight rows. Differential privacy has a name for this. It is called group privacy, and it says that if one person occupies k rows, the guarantee you can make about that person is not epsilon. It is k times epsilon. Cross out the small number you were about to publish and multiply it by eight.
TensorFlow Privacy will even tell you, in the text of its own report, that no user-level guarantee is possible without a bound on how many rows a single user can occupy. SMOTE removes exactly that bound.
Group privacy is the headline problem. There are two more, and both push in the same direction: they make the printed epsilon look better than it is.
The accountant needs to know how many examples you trained on, because privacy depends on the sampling rate q = batch size / dataset size. Feed it the post-SMOTE count and q shrinks. A smaller q reads as stronger privacy. So oversampling from 10,000 real records to 17,000 synthetic ones hands the accountant a number that is too large, and it obligingly reports an epsilon that is too small, before you even get to the group-privacy multiplier.
The last gap has nothing to do with SMOTE, but it stacks on top. Your accountant reports two epsilons and you probably quoted the friendlier one.
One assumes Poisson sampling, where each example lands in a batch by an independent coin flip. The other assumes you shuffle and cut fixed-size batches, which is what almost everyone's code actually does. The Poisson number is smaller. The shuffled number describes your pipeline. People quote the Poisson number.
The widget below lets you play all three effects against each other. Pick a batch size and a noise level, then flip the dataset size and the sampling assumption and watch the reported epsilon drift away from the honest one.
The two numbers are the same training run. One is what a hurried author writes in the abstract. The other is what the run actually guarantees. Thirty times apart, and every step of the gap looked reasonable on its own.
You do not need SMOTE. Two options keep one row per person and cost nothing in privacy.
Weight the loss so the minority class counts for more per example. The gradient still belongs to one person, so the accountant's assumption survives. Or leave the model alone and move the decision threshold after training, since where you cut the probability is a free parameter that spends no budget. Both handle imbalance without smearing a patient across rows.
The larger habit is the one worth keeping. Under differential privacy, every operation on your data is part of the mechanism, not a step that happens before it. Resampling, augmentation, deduplication, and imputation all touch the thing the guarantee is about. Treat them as first-class decisions with consequences you can state, and adaptive data stops being the place your privacy quietly leaks.