Probability, sensitivity, and composition
Privacy guarantees are distributional statements about neighboring datasets.
Privacy and federated learning
Moving data or gradients is a systems choice, not by itself a privacy guarantee. This chapter connects formal bounds, protocols, attacks, utility, and accounting.
12 maintained papers · literature reviewed through 2 October 2026
Keeping data on-device—or omitting the original training files—is not a privacy guarantee. Differential privacy defines a measurable information bound; federated optimization changes where updates are computed; membership and teacher-inversion methods expose residual leakage or model-dependent surrogate data. A defensible system states adversary, protected unit, model access, budget, trust boundary, utility, and composition.
Prerequisite contract
Distributed optimization
Client sampling, aggregation, and failures define the actual protocol.
How the field developed
2006
Differential-privacy mechanisms
Sensitivity-calibrated noise turns privacy into a parameterized guarantee over neighboring datasets.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 01.01 Calibrating Noise to Sensitivity in Private Data Analysis The paper’s epsilon-indistinguishability criterion bounds how much the distribution of an entire interaction transcript can change when one database row changes. It quantifies over every neighboring pair, adversary, and transcript, then connects that distributional contract to simulatable and semantic formulations in the appendix. Privacy is a property of the complete randomized interaction, not a claim that an adversary learns nothing about the population. — Work: Derive the Laplace mechanism for one bounded query and identify the protected unit.
Exit evidence: State adjacency, epsilon, delta, clipping or sensitivity, and composition.
2016–2017
Private and federated optimization
DP-SGD clips and perturbs gradients; FedAvg aggregates decentralized client updates under heterogeneous data and communication.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 02.01 Deep Learning with Differential Privacy DP-SGD bounds each example’s influence by clipping its gradient to an L2 norm C, averages clipped gradients, adds calibrated Gaussian noise, and spends a predeclared privacy budget across training. The paper’s moments accountant tracks log moments of privacy loss under subsampling and composition, yielding substantially tighter epsilon bounds than the generic strong-composition calculation in its experiments. Privacy is a property of the complete randomized mechanism and accounting assumptions, not of noise alone. Google differential privacy 02.02 Communication-Efficient Learning of Deep Networks from Decentralized Data Federated Averaging repeatedly broadcasts one global model, lets a sampled subset of clients perform several local SGD updates on their own examples, and averages the returned parameters in proportion to client data size. More local computation can dramatically reduce communication rounds. The method addresses decentralized optimization under unbalanced, non-IID data; keeping raw data on devices is an architectural boundary, not by itself a differential-privacy guarantee. — Work: Compare centralized SGD, FedAvg, DP-SGD, and their combination at matched utility and rounds.
Exit evidence: Report privacy accountant, client sampling, failures, bytes, and quality together.
2017
Dropout-tolerant secure aggregation
Pairwise masks cancel in the sum, threshold shares repair masks for dropped clients, and private masks prevent false-dropout disclosure; the server obtains an aggregate rather than plaintext individual updates.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 03.01 Practical Secure Aggregation for Privacy-Preserving Machine Learning Secure aggregation lets one server recover the element-wise sum of many clients' model-update vectors without learning any surviving client's vector in the clear. Every pair of clients derives cancelling pseudorandom masks, while each client also adds a private self-mask. Threshold secret sharing gives the server just enough recovery material to remove masks belonging to dropped clients and the surviving self-masks, but not enough to isolate one honest survivor. It is a cryptographic aggregation primitive: it is not differential privacy, Byzantine robustness, or permission for the server to inspect individual gradients. Version-of-record manuscript · DOI Work: Trace every mask, survivor decision, threshold, and reconstructed secret through one failed-client round.
Exit evidence: Separate confidentiality of submitted updates from correctness, availability, differential privacy, and final-model leakage.
2017–2019
Empirical membership and gradient leakage attacks
Membership inference probes model outputs; DLG optimizes dummy data until its parameter gradients match a directly observed small-batch gradient.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 04.01 Membership Inference Attacks Against Machine Learning Models A membership-inference attacker receives a complete candidate record and black-box access to a target classifier, queries the record, and predicts whether it was in the target’s training set. Shadow models create labeled examples of member and non-member output behavior; per-class attack models learn the confidence patterns that distinguish them. The attack operationalizes model memorization as a privacy risk without needing target parameters. — 04.02 Deep Leakage from Gradients Deep Leakage from Gradients (DLG) reconstructs a training example by treating unknown input pixels and the unknown label as optimization variables. Given the current model parameters and a shared gradient, the attacker computes gradients from dummy data, minimizes the squared distance between dummy and observed gradients, and backpropagates through that gradient calculation. When the gradient fields match, the dummy input can become visually and semantically close to the private example. The attack demonstrates that a gradient is a data-dependent observation, not an automatically anonymous update. It therefore tests the gradient interface used by federated averaging and motivates controls such as secure aggregation. Final proceedings PDF · Authors' implementation Work: Define attacker access, reproduce both attacks at matched utility, then vary aggregation, batch, local steps, compression, and protection.
Exit evidence: Separate a successful or failed empirical attack from a formal privacy guarantee.
2019–2020
Teacher inversion and data-free transfer
Data-Free Learning of Student Networks trains a teacher-guided generator using confidence, activation, and class-balance priors while the student imitates teacher responses; Zero-Shot KD optimizes class-conditioned Data Impressions; DeepInversion adds batch-normalization feature-distribution priors and image regularizers. Removing direct access to original files is an access condition, not a privacy guarantee, and higher-fidelity synthesis can make leakage more urgent.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 05.01 Data-Free Learning of Student NetworksPeer-reviewed ICCV 2019 proceedings paper. Data-Free Learning of Student Networks (DAFL) replaces unavailable training images with a generator optimized through a fixed teacher. The generator searches for inputs that make the teacher confident, strongly activated, and class-balanced; the student then matches teacher predictions on those synthetic inputs. The paper moves KD from 'teacher plus transfer data' to 'teacher as the source of a transfer set,' while still requiring internal teacher features for its full loss. Author manuscript 05.02 Zero-Shot Knowledge Distillation in Deep Networks Zero-Shot Knowledge Distillation handles a specific access constraint: the trained teacher is available, but neither its training examples nor metadata derived from them can be used. It synthesizes a transfer set by sampling plausible class-probability vectors from a teacher-derived Dirichlet model and optimizing random inputs until the teacher emits those vectors. The resulting Data Impressions then replace real examples in ordinary soft-target distillation. — 05.03 Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion DeepInversion synthesizes class-conditional images from a trained classifier when the original dataset is unavailable. It starts from noise and backpropagates through the frozen teacher, combining a desired class prediction with image priors and a feature-distribution prior derived from the running means and variances stored in batch-normalization layers. Adaptive DeepInversion additionally searches for examples on which teacher and student disagree. Official code Work: Compare teacher-guided learned synthesis, class-conditioned impressions, and feature-prior inversion while varying model access, class prior, internal statistics, initialization, and regularization; then run leakage audits.
Exit evidence: State what original-data access was removed, which logits, weights, gradients, or internal statistics remain exposed, what recognizable structure is recovered, and which privacy claim—if any—has evidence.
2020
Optimization under heterogeneous federated clients
FedProx regularizes local drift and permits variable client work; SCAFFOLD uses server/client control variates. Neither data locality nor optimization stability is itself a privacy guarantee.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 06.01 Federated Optimization in Heterogeneous NetworksPublished in the MLSys 2020 proceedings. Core mechanism: a proximal local objective. FedProx modifies federated averaging by adding a quadratic penalty that keeps each client's local model near the current server model and by allowing heterogeneous clients to return solutions of differing accuracy. This reading boundary matters: the method stabilizes optimization under statistical and systems heterogeneity; it does not itself encrypt updates, prevent leakage, or guarantee fairness across clients. Official implementation 06.02 SCAFFOLD: Stochastic Controlled Averaging for Federated LearningPublished in the ICML 2020 proceedings, PMLR 119; no canonical first-party implementation was verified. Core mechanism: server and client control variates. SCAFFOLD maintains control variates at the server and each client, using their difference to correct local stochastic gradients so non-IID clients do not systematically drift away from the global optimization direction. This reading boundary matters: the method corrects a specific optimization bias under its protocol and assumptions; it is not privacy protection, personalization, Byzantine robustness, or a guarantee for every local solver. — Work: Compare FedAvg, FedProx, and SCAFFOLD under controlled client heterogeneity, participation, local work, failures, and communication; add a separate privacy mechanism and attack evaluation only when making a privacy claim.
Exit evidence: Report per-client and aggregate utility, rounds, bytes, local steps, drift or control-variate state, fairness, failure recovery, attacker access, and any formal privacy budget separately.
2025–2026
Generative scene representations and coordinate privacy
Autoregressive Gaussian scene generation and inversion of scene-coordinate regressors expose different consequences of learned 3D representations: what a model can generate, and what its outputs reveal about a private scene.
Key papers in this section
Order Primary paper · authors · year · venue/status Core idea and why to read it Official resources 07.01 Seeing Through the Weights: Privacy Leakage in Scene Coordinate RegressionAccepted to ECCV 2026; oral presentation verified in the official program. This record and review use the conference-hosted manuscript checked 2026-09-12; a separate Springer proceedings record was not verified. The official repository was a README with a forthcoming-code notice when checked; a runnable implementation was not verified. A scene stored only in neural-network weights can still leak its geometry. This paper shows that querying scene-coordinate regression models with unrelated proxy images produces coordinate samples that accumulate into recognizable scene structure; access to descriptors or gradients improves filtering and refinement. It challenges privacy by representation, not the cryptographic security of a protected model. Official ECCV oral presentation · Official ECCV paper record · arXiv record · Official project · Official code repository (release announced, not yet implemented) Work: Which view, geometry or coordinate information is accessible, and do appearance quality and geometric fidelity agree?
Exit evidence: Image FID does not certify 3D coverage; coordinate-output access is stronger than a pose-only service interface.
Research branches
Formal privacy
Which individual or group change is bounded, and under what composition?
Calibrating Noise to Sensitivity in Private Data Analysis · Deep Learning with Differential Privacy
Federated optimization and secure aggregation
Who sees raw updates, aggregate updates, metadata, failures, and recovery shares?
Communication-Efficient Learning of Deep Networks from Decentralized Data · Practical Secure Aggregation for Privacy-Preserving Machine Learning
Leakage measurement
What adversary, access level, base rate, batch, shared object, and utility point define the attack?
Membership Inference Attacks Against Machine Learning Models · Deep Leakage from Gradients
Model inversion and synthesized data
Which logits, weights, features, priors, or gradients are exposed, and can they recover recognizable or membership-linked structure?
Data-Free Learning of Student Networks · Zero-Shot Knowledge Distillation in Deep Networks · Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion
Optimization under heterogeneous federated clients
Is the intervention correcting local optimization drift, reducing communication, or providing a quantified confidentiality guarantee?
Federated Optimization in Heterogeneous Networks · SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
Generative scene representations and coordinate privacy
Which view, geometry or coordinate information is accessible, and do appearance quality and geometric fidelity agree?
Seeing Through the Weights: Privacy Leakage in Scene Coordinate Regression
Comparison table
| Family or stage | Scientific distinction | Main risk | Anchors |
|---|---|---|---|
| Differential privacy | Formal bound on output-distribution change. | Wrong unit or incomplete accounting voids the interpretation. | 2006 · 2016 |
| Federated learning | Moves computation to clients and aggregates updates. | Updates and metadata can still leak information. | 2017 |
| Secure aggregation | Cryptographically reveals an aggregate while concealing qualifying individual updates. | It does not validate inputs, guarantee availability, or bound what the aggregate and final model reveal. | 2017 |
| Membership / gradient attacks | Measures concrete empirical leakage from outputs or directly observed updates. | A weak failed attack does not establish privacy; a successful small-batch attack does not quantify every deployment. | 2017 · 2019 |
| Teacher inversion / data-free synthesis | Constructs a surrogate transfer distribution from teacher outputs, parameters, or internal statistics without opening the original training files. | Data-free transfer is an access condition, not a formal privacy guarantee; generated samples may expose or approximate memorized structure. | 2019 · 2019 · 2020 |
| Optimization under heterogeneous federated clients | FedProx regularizes heterogeneous local objectives and permits variable work; SCAFFOLD uses server/client control variates to correct client drift. | Better federated convergence and keeping raw data local do not bound leakage from updates, aggregates, metadata, or the final model. | 2020 · 2020 |
| Generative scene representations and coordinate privacy | Autoregressive Gaussian scene generation and inversion of scene-coordinate regressors expose different consequences of learned 3D representations: what a model can generate, and what its outputs reveal about a private scene. | Image FID does not certify 3D coverage; coordinate-output access is stronger than a pose-only service interface. | 2026 |
Concept lineage
Calibrating Noise to Sensitivity in Private Data AnalysisDeep Learning with Differential Privacy
A query mechanism is adapted to iterative gradient learning through clipping, noise, subsampling, and accounting.
Communication-Efficient Learning of Deep Networks from Decentralized DataMembership Inference Attacks Against Machine Learning Models
Changing data location changes the attack surface but does not remove the need to measure output and update leakage.
Communication-Efficient Learning of Deep Networks from Decentralized DataPractical Secure Aggregation for Privacy-Preserving Machine Learning
Distributed averaging is wrapped in a dropout-tolerant cryptographic protocol so the coordinator obtains only the aggregate under stated thresholds.
Deep Leakage from GradientsPractical Secure Aggregation for Privacy-Preserving Machine Learning
Direct gradient reconstruction motivates hiding qualifying individual updates, while aggregate leakage and final-model privacy still require separate controls.
Membership Inference Attacks Against Machine Learning ModelsZero-Shot Knowledge Distillation in Deep Networks
Output leakage testing is complemented by an explicit construction that inverts teacher behavior into a synthetic transfer set; neither access condition alone supplies a formal guarantee.
Data-Free Learning of Student NetworksDreaming to Distill: Data-Free Knowledge Transfer via DeepInversion
Teacher-guided learned synthesis is complemented by explicit batch-normalization feature priors and image regularization for more natural transfer inputs.
Failure modes to audit
- Equating federated learning with privacy.
- Calling a method private merely because it is data-free or omits the original files.
- Reporting epsilon without adjacency, delta, accountant, or unit.
- Comparing private and non-private models at unequal utility.
- Ignoring malicious clients, server trust, and metadata.
Questions for every paper
- Who is protected from whom?
- What is the neighboring dataset?
- What information crosses the trust boundary?
- How does privacy compose over time?
Complete maintained paper route
Section titled “Complete maintained paper route”12 canonical readings · 2006–2026· route reviewed through October 2, 2026.
This is the complete structured list for this chapter, ordered chronologically from earlier foundations toward current work. The explanatory ladder above highlights the shortest conceptual route; this list also includes important branches, benchmarks, and current endpoints. It is generated from the same deduplicated catalog used by the AI background registry and My Library, so a future catalog update appears here automatically.
Show all 12 readings in maintained order
Calibrating Noise to Sensitivity in Private Data Analysis
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith
Metadata checked
Deep Learning with Differential Privacy
Martín Abadi et al.
Official artifacts: Google differential privacy
Metadata checked
Communication-Efficient Learning of Deep Networks from Decentralized Data
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas
Metadata checked
Membership Inference Attacks Against Machine Learning Models
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov
Metadata checked
Practical Secure Aggregation for Privacy-Preserving Machine Learning
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth
Official artifacts: Version-of-record manuscript · DOI
Metadata checked
Zero-Shot Knowledge Distillation in Deep Networks
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, R. Venkatesh Babu, and Anirban Chakraborty
Metadata checked
Deep Leakage from Gradients
Ligeng Zhu, Zhijian Liu, and Song Han
Official artifacts: Final proceedings PDF · Authors' implementation
Metadata checked
Data-Free Learning of Student Networks
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian
Peer-reviewed ICCV 2019 proceedings paper.
Official artifacts: Author manuscript
Metadata checked
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion
Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz
Official artifacts: Official code
Metadata checked
Federated Optimization in Heterogeneous Networks
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith
Published in the MLSys 2020 proceedings.
Official artifacts: Official implementation
Metadata checked
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh
Published in the ICML 2020 proceedings, PMLR 119; no canonical first-party implementation was verified.
Metadata checked
Seeing Through the Weights: Privacy Leakage in Scene Coordinate Regression
Oleksii Nasypanyi, Jaemin Cho, Utku Ozbulak, Byungkon Kang, and Francois Rameau
Accepted to ECCV 2026; oral presentation verified in the official program. This record and review use the conference-hosted manuscript checked 2026-09-12; a separate Springer proceedings record was not verified. The official repository was a README with a forthcoming-code notice when checked; a runnable implementation was not verified.
Official artifacts: Official ECCV oral presentation · Official ECCV paper record · arXiv record · Official project · Official code repository (release announced, not yet implemented)
Metadata checked