Quy-Anh Dang

Quy-Anh Dang

Researcher, Knovel Engineering


← All work

LLM Interpretability, Layer by Layer — Part 5: Mechanisms, Finding a Circuit

November 7, 2026 · 7 min read · Interpretability, Large Language Models, Machine Learning

Part 4 told us where GPT-2 recalls an athlete's sport: early MLPs at the name, late attention at the last token. But a heatmap of important places isn't a mechanism. A mechanism says how the parts work together, step by step, like a wiring diagram.

In interpretability, such a diagram is called a circuit: a small set of components (attention heads, MLPs) and the connections between them that together carry out one behaviour. This post looks at two of the best-understood circuits in any language model, both in GPT-2 small:

  1. induction heads, which let a model copy patterns it has seen earlier in the text,
  2. the indirect object identification (IOI) circuit, which completes "When Mary and John went to the store, John gave a drink to" with Mary,

and then asks how we know a circuit is right: faithfulness, completeness and minimality.

This is still the cause rung, but now at the level of individual heads and how they connect.

1. A warm-up: induction heads#

Take a sequence of 50 random tokens and repeat it. The first time through, GPT-2 can't predict anything: its loss is 12.0 (worse than guessing at random, because the text looks nothing like real language). The second time through, the loss drops to 0.22. The model recognizes the repeat and copies.

The heads responsible are induction heads (Olsson et al., 2022). An induction head implements a simple rule: "if the current token is A, find an earlier A, look at the token that came right after it (B), and predict B." So after seeing "… Harry Potter … Harry", it predicts "Potter".

We can find these heads directly. On the repeated sequence, an induction head attends from each token in the second copy to the token after that token's first occurrence. We measure how much attention goes there:

Induction score for every head of GPT-2 small. Heads 5.1, 5.5, 6.9, 7.2 and 7.10 score above 0.85; a few heads in layers 9 to 11 score around 0.5.

Five heads stand out with scores between 0.85 and 0.94: 5.1, 5.5, 6.9, 7.2 and 7.10. These match the induction heads found in earlier work on GPT-2 small.

Induction heads need help. To find "the token after the earlier A," a head must know, at each earlier position, which token came before it. That's done by a previous-token head in an earlier layer, which copies each token's identity into the next position's stream. Two heads in two layers, connected through the residual stream: that's the smallest real circuit. It's also thought to be a main source of in-context learning, a model's ability to pick up patterns from its prompt.

import torch

seq = torch.randint(1000, 10000, (10, 50))
bos = torch.full((10, 1), model.tokenizer.bos_token_id)
tokens = torch.cat([bos, seq, seq], dim=1)                   # random tokens, then the same again
_, cache = model.run_with_cache(tokens)

q = torch.arange(51, 101)                                     # positions in the second copy
k = q - 49                                                    # the token after the earlier copy
scores = torch.stack([cache["pattern", l][:, :, q, k].mean((0, 2)) for l in range(12)])
top = scores.flatten().topk(5)
print([(i // 12, i % 12, round(v, 2)) for v, i in zip(top.values.tolist(), top.indices.tolist())])
# [(5, 5, 0.94), (6, 9, 0.92), (7, 10, 0.92), (5, 1, 0.91), (7, 2, 0.85)]   (layer, head, score)

2. The IOI task#

Now a real-language behaviour. Read this sentence and fill the gap:

When Mary and John went to the store, John gave a drink to ___

You say Mary. Not because of grammar alone, but by a bit of logic: two people were introduced, one of them (John, the subject, S) is mentioned again as the giver, so the receiver must be the other one (Mary, the indirect object, IO).

GPT-2 small does this reliably. On 60 such prompts with varied names, places and objects, it ranks the right name first 97% of the time, with an average logit difference of 3.38.

Wang et al. (2022) reverse-engineered how, in Interpretability in the Wild. Their circuit has 26 attention heads in several teams. The core idea, in plain words:

  1. Duplicate-token and induction heads notice that the subject's name (John) appears twice.
  2. S-inhibition heads (around layers 7–8) pass that on to the final position: "John is the repeated one, so don't say John."
  3. Name mover heads (layers 9–10) attend from the final token to the names in the sentence and copy a name into the output, except the one that S-inhibition told them to avoid. So they copy Mary.

The rest of this post checks this story with our own experiments.

3. Name movers: who writes the answer?#

First, direct logit attribution. Because the residual stream is a sum (Part 0), the final logit difference is a sum of contributions from each head's output at the last position. We can compute each head's share directly: project its output onto the direction (scaled by the final normalization):

Direct effect of each head on the IO-minus-S logit difference. Head 9.9 contributes +3.4, head 9.6 +1.4 and head 10.0 +0.6; head 10.7 contributes -2.5 and head 11.10 -0.8.

A handful of heads matter, all in the last three layers:

  • Name movers push toward Mary: 9.9 (+3.40), 9.6 (+1.41), 10.0 (+0.58).
  • Negative name movers push against the answer: 10.7 (−2.54) and 11.10 (−0.81). Like the last MLP in Part 4, they seem to keep the model from being overconfident.

Their attention patterns show what they're doing:

Attention from the final token. Head 9.9 puts most of its attention on the indirect object, the first name. Head 7.9 attends mostly to the second mention of the subject.

From the final to, name mover 9.9 puts 74% of its attention on the IO name (orange) and very little on either mention of the subject (blue). It looks at the right name and copies it. S-inhibition head 7.9 does something different: it attends to the second mention of the subject. That's the "John is the repeated one" signal being read.

Activation patching (Part 4's tool, now per head) confirms the causal role. We corrupt with the ABC version of each prompt, "When Lisa and Paul went to the store, Kate gave a drink to": three different names, so nothing points to Mary or John. Patching head 9.9's clean output into the corrupted run alone restores 98% of the logit difference. Head 9.6 restores 41%, head 10.0 18%.

4. Does the circuit explain the behaviour?#

Finding important heads is easy. Showing that these heads are the mechanism is harder. Three standard tests (Wang et al., 2022):

  • Faithfulness: run the model with only the circuit active. Does it still do the task?
  • Completeness: does the circuit include everything that matters, or are there important parts outside it?
  • Minimality: is every part of the circuit needed?

To "turn off" a head, we use mean ablation. We replace its output with its average output over the ABC prompts. That removes the information specific to this prompt while keeping the head's output in a normal range. (Setting outputs to zero can break the model in unnatural ways.)

Model runLogit difference (IO − S)
Full model3.38
Only the 26 circuit heads (other 118 mean-ablated)2.61 (77%)
26 random heads instead−0.05
All 144 heads mean-ablated−0.26
Circuit without the 3 main name movers2.30 (68%)

Faithfulness: fairly good. The 26 heads alone keep 77% of the behaviour, while 26 random heads keep none of it. The circuit is clearly the right place, but it isn't everything. Wang et al. reported 87% with a slightly different setup. The missing quarter is spread across many small contributions, and that's typical.

A surprise: self-repair. Remove the three main name movers (9.9, 9.6, 10.0), which carry most of the direct effect, and the circuit still keeps 68%. Other heads, the backup name movers, take over when the main ones are gone. This is a real complication for interpretability: knocking out a part and seeing little change does not prove the part wasn't doing anything. The model may simply have a spare. Later work found this kind of self-repair in many models.

5. How circuits are found#

We used a circuit that was already published and checked it. Finding one from scratch used to take months of manual patching. Newer methods automate the search:

  • Path patching patches only the path from one head to another (e.g. from an S-inhibition head into the name movers' queries), not the head's whole effect. That's how Wang et al. found which teams connect to which.
  • ACDC (Conmy et al., 2023) starts from the full model and repeatedly removes connections whose patching barely changes the output, until a small circuit remains.
  • Attribution graphs (Anthropic, 2025) replace the model's MLPs with interpretable features (Part 3) and trace how features at one position and layer feed features at the next, producing a graph of the computation for one prompt.

All of them face the questions from section 4. A circuit is a hypothesis, judged by how much of the behaviour it explains (faithfulness), whether it misses important parts (completeness), and whether it includes unneeded ones (minimality).

6. In code: direct logit attribution#

prompt = "When Mary and John went to the store, John gave a drink to"
io, s = model.to_single_token(" Mary"), model.to_single_token(" John")
_, cache = model.run_with_cache(prompt)

direction = model.W_U[:, io] - model.W_U[:, s]                 # "Mary rather than John"
scale = cache["ln_final.hook_scale"][0, -1]                    # the final LayerNorm's scaling
for l, h in [(9, 9), (9, 6), (10, 0), (10, 7), (11, 10)]:
    z = cache[f"blocks.{l}.attn.hook_z"][0, -1, h]              # head output before W_O
    contribution = (z @ model.W_O[l, h]) @ direction / scale
    print(f"head {l}.{h}: {contribution.item():+.2f}")
# head 9.9: +2.10
# head 9.6: +2.11
# head 10.0: +0.82
# head 10.7: -2.28
# head 11.10: -0.84

On this single sentence, heads 9.9 and 9.6 share the work about equally. Averaged over 60 sentences (the figure above), 9.9 does most of it. Individual prompts vary, which is why circuit results should always be averaged over a dataset.

Recap#

  • A circuit is a small set of components and connections that implements a behaviour.
  • Induction heads (5.1, 5.5, 6.9, 7.2, 7.10 in GPT-2 small) copy patterns that repeat in the context. They work with earlier previous-token heads, and drop the loss on a repeated random sequence from 12.0 to 0.22.
  • The IOI circuit: duplicate-token and induction heads notice the repeated name, S-inhibition heads pass on "not that one," and name movers (9.9, 9.6, 10.0) copy the other name. Head 9.9 alone restores 98% of the behaviour when patched.
  • Judge circuits by faithfulness (77% with only the 26 heads), completeness and minimality.
  • Self-repair (68% kept even without the main name movers) means that ablating a part and seeing little change doesn't prove it was unimportant.

Next up — Part 6: Steering and Editing. We've found where facts live and how heads move them. Now we use that knowledge to change behaviour on purpose: flip Kobe Bryant to golf, make the model's writing more positive or negative, and measure what breaks along the way.


© 2026 Quy-Anh Dang. All rights reserved.