Membership Inference and Data Extraction
Privacy attacks that reveal whether a specific record was in a model's training data, or that make the model regurgitate memorized training text.
Also membership inference attack · MIA · training data extraction · memorization · privacy attack
Shokri et al., 2017
If a student remembers one exam question unusually well, word for word, you can guess that exact question was in their practice set. Models give themselves away in the same way.
Where it fits
+All relationships as a list
How it works
A trained model is a compressed summary of its training data, and sometimes it remembers too much. Two related privacy attacks exploit that.
Membership inference asks: was this record part of the training set? Models tend to be a bit more confident, or have lower loss, on examples they trained on. An attacker who sees those signals can guess membership better than chance. That alone can be sensitive: being in a dataset of cancer patients reveals that you are one.
Training data extraction goes further: get a generative model to output memorized training data. Researchers extracted names, phone numbers and email addresses from GPT-2, and near-copies of training images from diffusion models. Text or images that appear many times in the training data are the most likely to be memorized.
What helps:
- Less overfitting: regularization and more data.
- Deduplicating training data.
- Differential privacy, which mathematically limits how much any single record can influence the model.
- Removing personal data before training.
Regulators increasingly treat models trained on personal data as possibly containing personal data themselves, which makes these attacks a compliance issue too.
Everyday use cases
- Case 01
Medical models
Learning that a person's record was in a hospital's HIV-clinic training set already reveals sensitive information.
- Case 02
Auditing LLMs
Researchers extracted names, phone numbers and addresses that GPT-2 had memorized from web text.
Strengths, weaknesses and when to use it
Strengths
- A practical yardstick for how much a model leaks about its training data.
Weaknesses
- Attack success varies widely; low average risk can hide high risk for outlier records.
When to use itBefore releasing models trained on personal or confidential data.
Compare with
Check yourself
- Q01
Why do overfit models leak more membership information?
Show answer
A: They behave noticeably differently on examples they were trained on