You can write a differentially private mechanism that compiles, trains, converges, and prints a clean epsilon, and still guarantees nothing.
I know because I wrote one. It was an adaptive clipping algorithm: instead of fixing the gradient clip bound by hand, it watched training and adjusted the bound on its own. It ran. It reported a small budget. It was not differentially private, and none of that was visible from the outside.
This is a post about adaptive intelligence, the idea that a good algorithm tunes itself while it learns. Self-tuning is powerful. It is also the exact place privacy leaks, because tuning means looking at the data, and under differential privacy every look has a price.
Here is the rule. Every time your mechanism reads the private data to make a decision, it spends privacy budget, whether or not you wrote down the cost.
Fixed DP-SGD obeys this without trying. The clip bound is a constant you chose in advance. The noise is a constant you chose in advance. The only thing that touches the data is the gradient, and that one access is exactly what the accountant charges you for.
The moment your algorithm gets clever, it starts reading the data in new places. It sets the clip bound from the gradients it just saw. It scales the noise by how uncertain the model feels. Each of these is a fresh look at private data, and if the accountant does not know about it, that look is free to you and expensive to the people in your dataset.
Five ways I have watched this go wrong. Click each one.
The move: C = mean(gradient_norms), so the bound tracks the data.
Why it leaks: the clip bound is now a function of the private gradients. Anyone who sees C learns about them. The Gaussian mechanism assumes C was fixed before you looked.
The fix: keep C constant, or estimate it through its own noisy sub-mechanism that pays budget.
The move: sigma = base * (1 + model_uncertainty).
Why it leaks: the noise level itself now carries information. An observer reads the amount of noise and infers the signal that set it. Noise is supposed to hide the data, not encode it.
The fix: sigma is a constant you pick to buy a target epsilon. It never depends on the data.
The move: give one group a larger clip bound, then size the noise to the smaller one.
Why it leaks: sensitivity is the largest amount any one record can move the sum. If some records are clipped to a bigger bound, the noise must match that bigger bound. Size it to the smaller one and the larger-bound group is under-protected.
The fix: calibrate noise to max over all per-group bounds, not the average and not the minimum.
The move: epsilon_total = sum(epsilon_epoch).
Why it leaks: naive summation is both loose and, once parameters adapt on the data, invalid. Each step's mechanism depends on choices made from earlier private data, so the steps are not the independent pieces the sum assumes.
The fix: compose with a Renyi accountant that tracks the whole adaptive procedure, not a running total of unrelated numbers.
The move: the adaptation reads the data to update a knob, but only the gradient release is accounted.
Why it leaks: the knob is a second output computed from private data. It free-rides on the budget the gradients paid. Two releases, one bill.
The fix: account for every release. The next section shows how to split one budget across both.
Defects one and two are the same disease: a knob set from the data with no entry in the ledger. Defect three is a calibration slip that any per-group scheme invites. Defect four is a composition shortcut. Defect five is the one people miss most, so it gets its own section.
Good adaptive clipping actually does the right thing. To tune the bound safely, it privately estimates how many gradients got clipped last step, then nudges the bound toward a target. That estimate is noisy on purpose, because it is a read of the data and has to be paid for.
So every step releases two things. The noisy gradient sum, and the noisy clipped count. The standard privacy statement in most libraries accounts for one Gaussian mechanism, the gradients. Hand it your noise multiplier and it happily reports an epsilon that ignores the second release. The number is too good, and defect five is hiding inside it.
The fix is to split one budget across both releases and account for the sum. Under Renyi composition, releasing two independent Gaussians combines cleanly:
You pick the total you want to be charged for, hand a slice of it to the count, and the gradient noise grows a little to keep the books balanced. There is a real trap here. A popular default sets the count noise to records-per-round over twenty, which was tuned for federated rounds holding thousands of records. At a batch of thirty-two it demands more budget than exists and the split has no solution.
Move the sliders. Watch the records-over-twenty default fall off a cliff, and watch the budget-share fix stay sane.
The right column always has a solution, because the budget share is bounded below one by construction. The left column blows up the moment the batch is small, which is precisely when you are doing on-device or memory-tight training. The fix costs almost nothing: a five percent slice for the count inflates the gradient noise by under three percent, and the composition still recomposes to the sigma you asked for.
Before I call anything differentially private, I ask four questions. Does every read of the private data have a line in the budget? Is the noise calibrated to the largest amount one record can move the output? Does the composition account for the whole adaptive procedure rather than a sum of parts? And does my split of the budget actually recompose to the total I claim?
That last one is the cheapest insurance in the whole field. Derive the pieces, add them back up, and check that they equal the number you are about to publish. If they do not, you have found a leak before your reviewers did.
Adaptive intelligence is worth the trouble. A mechanism that tunes its own clipping beats one you hand-tuned, and it does it while spending a fixed budget more wisely than a static schedule ever could. The catch is that self-tuning and privacy pull against each other, and the only way to have both is to pay for every look. Budget it, calibrate it, compose it, and check that it adds up. Then the number you print is the number you can defend.