Explainability
Methods that show why a model made a decision, such as which inputs mattered most, so people can check, debug and trust it.
Also XAI · interpretability · SHAP · LIME · saliency maps · feature importance
When a teacher marks your answer wrong, you want to know why. Explainability makes the model show its working.
Where it fits
+All relationships as a list
Is a kind of / part of
Kinds and parts
Siblings
Applies to
Used by
How it works
Some models explain themselves: a small decision tree or a linear model can be read directly. For gradient boosting or neural networks, we need tools:
- Feature importance: shuffle one input column and see how much accuracy drops. Big drop, important feature.
- SHAP: split one prediction fairly among the inputs (“income +12%, debt −20%, age +3%”), based on Shapley values from game theory.
- LIME: fit a simple, readable model that imitates the complex one just around one example.
- Saliency maps: for images, highlight which pixels most affected the output.
These methods have caught real bugs, such as image models that learned to spot a watermark or a ruler instead of the actual object. But they explain the model, which may itself be wrong, and approximations can mislead.
A newer research field, mechanistic interpretability, tries to reverse-engineer what individual neurons and circuits inside networks compute, for example finding the attention heads that track which word “it” refers to.
Everyday use cases
- Case 01
Loan decisions
"Declined mainly because of high existing debt and short credit history", as regulators often require.
- Case 02
Debugging models
A saliency map revealed a classifier relying on a watermark instead of the object.
Famous examples
Strengths, weaknesses and when to use it
Strengths
- Builds trust, enables audits and catches shortcut learning.
Weaknesses
- Explanations can be approximate or misleading; they describe the model, not the world.
- Explaining large neural networks is still an open research problem.
When to use itWhenever decisions must be justified, audited or debugged.
Compare with
Check yourself
- Q01
What does a SHAP value for one feature tell you?
Show answer
A: How much that feature pushed this particular prediction up or down