LLM Interpretability, Layer by Layer — Part 0: What Does It Mean to Understand a Model?
November 7, 2026 · 7 min read · Interpretability, Large Language Models, Machine Learning
Type "Michael Jordan plays the sport of" into GPT-2, a small language model from 2019, and it answers "basketball." Somewhere in its 124 million numbers, the model connected a name to a sport. Where? How? And if it had answered "golf," could we have found the mistake and fixed it?
Interpretability is the study of questions like these: what happens inside a neural network between the input and the output. This series covers mechanistic interpretability, the branch that tries to explain a model's behaviour in terms of its internal parts, the way a biologist explains a heartbeat in terms of cells and signals.
This post covers four things every later part relies on:
- why we want to look inside models at all,
- the ladder of evidence: how to tell a real explanation from a pretty picture,
- a short primer on the transformer's residual stream,
- the model, tools and notation used throughout the series.
You need some comfort with vectors and matrix multiplication, and a rough idea of what a neural network is. No prior knowledge of transformers is needed.
1. Why look inside?#
A language model is tested mostly from the outside: give it inputs, score its outputs. That works until it doesn't. Three reasons to look inside:
- Debugging. When a model gets something wrong, outside testing tells you that it failed, not why. Knowing why tells you whether more data, a different prompt, or a fix to the model will help.
- Safety and trust. A model can behave well on every test you thought of and still carry a behaviour you never tested, such as a hidden trigger, a shortcut, or a bias that only shows up in rare cases. Looking inside is one of the few ways to catch what you didn't think to test. Part 7 does exactly this.
- Science. Models learn things nobody programmed. Understanding how they represent facts, grammar or arithmetic tells us something about learning itself.
ELI5: a mechanic who only listens to the engine can say "something's wrong." A mechanic who opens the hood can say "the fuel pump is failing, and here's why." Interpretability is opening the hood.
2. The ladder of evidence#
Looking inside a network is easy. You can print any number you like. The hard part is knowing whether what you see explains anything. A neuron that lights up whenever the word "basketball" appears might be how the model recognizes basketball. Or it might be a side effect, like a warning light wired to the wrong sensor.
This series organizes every method by the strength of the evidence it gives. Each rung adds something the one below can't provide:
- Observe. We read internal values and see what they look like. Useful for forming ideas, but it proves nothing.
- Correlate. We find a direction or feature inside the model that tracks a concept, such as "this athlete plays basketball." It shows the information is there, not that the model uses it.
- Cause. We change one internal value and see the output change as predicted. Now we know the model uses it.
- Control. We can push the model's behaviour in a chosen direction on purpose, which shows we understand the part well enough to use it.
- Audit. We find a behaviour nobody told us to look for, using only the tools. This is the real test of whether interpretability works.
The most common mistake in interpretability is stopping at "correlate" and calling it an explanation. Every part of this series names which rung its method reaches and what it can't show.
3. A five-minute primer: the residual stream#
Language models like GPT-2 are transformers. Here is the one picture you need for this series:
Text is split into tokens: words or pieces of words. "Michael Jordan plays the sport of" becomes 6 tokens: Michael, Jordan, plays, the, sport, of. (The space is part of the token.) For each token, the model keeps one vector of 768 numbers. That vector is the residual stream.
The stream starts as the token's embedding, a learned vector for each of the 50,257 tokens in GPT-2's vocabulary. Then it passes through 12 layers. Each layer has two blocks:
- Attention moves information between tokens. The vector for
ofcan pull in information fromJordanearlier in the sentence. Each layer has 12 attention heads that do this in parallel, each with its own pattern of which tokens to look at. - MLP (a small two-layer network) works on each token separately. It's where much of the model's stored knowledge seems to live.
The key point is in the arrows. Every block reads the stream and adds its output back to it. Nothing is overwritten. After layer , the stream at each token is just a sum:
At the end, the final vector at the last token is multiplied by the unembedding matrix to give one score (a logit) for every token in the vocabulary. A softmax turns the logits into probabilities for the next token.
ELI5: picture a long corridor in an office. Each token has its own folder that travels down the corridor. At each desk (a block), a worker reads the folder, writes a sticky note, and adds it to the folder. Nothing is ever thrown away. The attention workers can also copy notes between folders. At the end of the corridor, someone reads the last folder and announces the next word. Interpretability is reading those notes, and later, swapping and forging them to see what happens.
Because the stream is a running sum, it is a natural place to look. Any information the model will use later must be written into it somewhere. Most methods in this series read, compare or edit the residual stream.
4. Model, tools and notation#
One model throughout: GPT-2 small. It has 12 layers, 12 attention heads per layer, a 768-dimensional residual stream and 124M parameters. It's old and weak by today's standards, but that's the point. It runs on a laptop CPU, every snippet in this series runs in seconds to minutes, and some of the best-understood results in interpretability were found in it.
Tools. We use TransformerLens, which gives every internal activation a name and makes it easy to read or edit, and SAELens for Part 3. Since version 4, TransformerLens loads models through TransformerBridge:
import torch
from transformer_lens.model_bridge import TransformerBridge
torch.set_grad_enabled(False)
model = TransformerBridge.boot_transformers("gpt2", device="cpu")
model.enable_compatibility_mode() # classic hook names, e.g. blocks.6.hook_resid_pre
logits, cache = model.run_with_cache("Michael Jordan plays the sport of")
print(model.to_str_tokens("Michael Jordan plays the sport of"))
print(cache["blocks.6.hook_resid_pre"].shape) # [batch, position, d_model]
top = logits[0, -1].softmax(-1).topk(3)
print([(model.to_string(i), round(p.item(), 2)) for p, i in zip(top.values, top.indices)])
# ['<|endoftext|>', 'Michael', ' Jordan', ' plays', ' the', ' sport', ' of']
# torch.Size([1, 7, 768])
# [(' basketball', 0.5), (' football', 0.05), (' golf', 0.04)]
The <|endoftext|> token at the start is a "beginning of text" marker that TransformerLens adds. run_with_cache runs the model once and stores every internal activation by name.
Notation used in every part:
| Symbol | Meaning |
|---|---|
| residual stream at token position , before layer | |
| layer; "head 9.9" means layer 9, head 9 | |
| unembedding matrix: residual stream → one logit per vocabulary token | |
| logit difference | logit(right answer) − logit(wrong answer); our main way to score a behaviour |
| clean / corrupted run | the same prompt with one key detail changed, e.g. Kobe Bryant vs Tiger Woods |
| feature / direction | a vector in the residual stream that corresponds to a concept |
Our running example. GPT-2 small knows surprisingly few facts reliably. It gives "Paris" only a 7% chance after "The Eiffel Tower is in the city of." But it knows sports stars well. So Parts 1–4 and 6 follow one fact, athlete → sport, from where it is stored to where it is used and how to change it. Part 5 studies a different, well-mapped behaviour (finding the right name in a sentence), and Part 7 audits a model with a behaviour we hid in it ourselves.
Recap#
- Mechanistic interpretability explains a model's behaviour in terms of its internal parts.
- Reasons to do it: debugging, safety (catching what you didn't test for), and science.
- The ladder of evidence: observe → correlate → cause → control → audit. Correlation shows information is present; only causal tests show the model uses it.
- A transformer keeps a residual stream, one vector per token, that every attention and MLP block reads from and adds to. Attention moves information between tokens; MLPs process each token separately.
- The series uses GPT-2 small with TransformerLens, and follows the fact "Michael Jordan plays the sport of → basketball" through the model.
Next up — Part 1: Reading the Residual Stream. The simplest look inside: decode the residual stream at every layer as if it were the last one, and watch "basketball" appear, layer by layer. We also see what attention patterns show and what they don't.
