Bias and Fairness
Models trained on historical data can treat groups of people unequally. Fairness work measures those differences and tries to reduce them.
Also algorithmic bias · fairness metrics · demographic parity · equal opportunity · disparate impact
If a teacher only ever saw homework from one class, they might wrongly assume students from other classes do worse. A model trained on lopsided examples makes the same kind of unfair guesses.
Where it fits
+All relationships as a list
Try it
How it works
A model learns whatever patterns its data contains, including unfair ones. Bias creeps in through several doors:
- Who is in the data: a face dataset with mostly light-skinned faces leads to worse accuracy on darker skin. The 2018 Gender Shades audit found exactly this in commercial systems.
- What the labels encode: if past hiring decisions were biased, a model trained to imitate them learns the bias.
- Proxies: removing a sensitive attribute is not enough; postcode, name or hobbies can stand in for it.
To manage what you cannot see, you measure. Common group fairness metrics:
- Demographic parity: each group gets positive decisions at the same rate.
- Equal opportunity: among people who truly qualify, each group is approved at the same rate (equal true-positive rates).
These can conflict with each other and with overall accuracy, especially when groups differ in base rates, so fairness is partly a value judgment, not only a technical fix. Practical tools include rebalancing training data, adjusting decision thresholds per group, and auditing results before and after launch. The widget shows how a skewed training set produces skewed decisions.
Everyday use cases
- Case 01
Face analysis
A 2018 audit found commercial gender classifiers erred far more often on darker-skinned women than on lighter-skinned men.
- Case 02
Hiring and lending
Models trained on past decisions can learn to penalize groups that were historically disadvantaged.
Famous examples
- 2015Photo Library Search
Typing 'dog', 'beach' or 'receipt' into your photo app and finding matching pictures you never tagged, powered by image classifiers and image embeddings.
- 1989Credit Scoring
Predicting how likely a borrower is to repay, traditionally with interpretable scorecards based on logistic regression, increasingly with gradient boosting, under strict fairness and explainability rules.
Strengths, weaknesses and when to use it
Strengths
- Clear group metrics make unfairness measurable and auditable.
Weaknesses
- Different fairness definitions can conflict; you must choose deliberately.
- Removing a sensitive attribute does not remove bias, because other features act as proxies.
When to use itFor any model whose decisions affect people's opportunities, safety or treatment.
Check yourself
- Q01
Why does deleting the "gender" column often fail to remove gender bias?
Show answer
A: Other features (like job titles or hobbies) can act as proxies for gender
- Q02
What does "equal opportunity" compare across groups?
Show answer
B: True-positive rates (how many qualified people are approved)