Mechanistic Interpretability
Reverse-engineering what is happening inside a neural network: which internal features it represents and which circuits of neurons compute its behavior.
Also circuits · features · sparse autoencoders · superposition · induction heads · probing
Instead of only checking whether a calculator gives the right answers, you open it up and trace the wires to see exactly how it adds. Here the 'calculator' is a neural network with billions of wires.
Where it fits
+All relationships as a list
Is a kind of / part of
Applies to
How it works
Most explainability methods treat a model as a black box: which input features pushed this prediction up or down? Mechanistic interpretability opens the box and asks how the network computes its answer.
Key ideas:
- Features: concepts the network represents internally, such as “this text is in French” or “this is a curve in an image”. They often correspond to directions in a layer’s activations, not to single neurons.
- Superposition: networks pack more features than they have neurons by overlapping them, so individual neurons respond to several unrelated things. Sparse autoencoders are trained to pull those mixed signals apart into cleaner features.
- Circuits: groups of features and connections that implement a specific behavior. A well-known example in Transformers is the induction head, which helps explain in-context learning.
Researchers test their understanding by intervening: amplify or suppress a feature and check whether the behavior changes as predicted.
Why it matters for safety: if we could reliably read a model’s internal reasoning, we could check for deception or dangerous capabilities directly instead of only testing outputs. This connects interpretability closely to alignment. The field is young, and fully explaining a large model remains far off.
Everyday use cases
- Case 01
Finding what a model knows
Researchers located features in LLMs for concepts such as specific landmarks, code bugs or deception, and changed behavior by turning them up or down.
- Case 02
Auditing for hidden behaviors
Checking whether a model has internal mechanisms for unwanted behavior that tests did not reveal.
Strengths, weaknesses and when to use it
Strengths
- Explains how a model computes, not just which inputs mattered.
Weaknesses
- Labor-intensive and still far from fully explaining large models.
When to use itResearch on model behavior, safety audits, and debugging surprising failures.
Compare with
Check yourself
- Q01
What is superposition in neural networks?
Show answer
A: Representing more features than there are neurons by overlapping them