transformer-lens-interpretability
Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.
Install this skill
Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.
npx skills add KalarisLabs/research-agent-skills --skill transformer-lens-interpretabilityuvx research-agent-skills install transformer-lens-interpretabilityDocumentation & Instructions
TransformerLens: Mechanistic Interpretability for Transformers
TransformerLens is the de facto standard library for mechanistic interpretability research on GPT-style language models. Created by Neel Nanda and maintained by Bryce Meyer, it provides clean interfaces to inspect and manipulate model internals via HookPoints on every activation.
GitHub: TransformerLensOrg/TransformerLens
When to Use TransformerLens
Use TransformerLens when you need to:
- Reverse-engineer algorithms learned during training
- Perform activation patching / causal tracing experiments
- Study attention patterns and information flow
- Analyze circuits (e.g., induction heads, IOI circuit)
- Cache and inspect intermediate activations
- Apply direct logit attribution
Consider alternatives when:
- You need to work with non-transformer architectures → Use nnsight or pyvene
- You want to train/analyze Sparse Autoencoders → Use SAELens
- You need remote execution on massive models → Use nnsight with NDIF
- You want higher-level causal intervention abstractions → Use pyvene
Installation
pip install transformer-lens
For development version:
pip install git+https://github.com/TransformerLensOrg/TransformerLens
Core Concepts
HookedTransformer
The main class that wraps transformer models with HookPoints on every activation:
from transformer_lens import HookedTransformer
# Load a model
model = HookedTransformer.from_pretrained("gpt2-small")
# For gated models (LLaMA, Mistral)
import os
os.environ["HF_TOKEN"] = "your_token"
model = HookedTransformer.from_pretrained("meta-llama/Llama-2-7b-hf")
Supported Models (50+)
| Family | Models |
|---|---|
| GPT-2 | gpt2, gpt2-medium, gpt2-large, gpt2-xl |
| LLaMA | llama-7b, llama-13b, llama-2-7b, llama-2-13b |
| EleutherAI | pythia-70m to pythia-12b, gpt-neo, gpt-j-6b |
| Mistral | mistral-7b, mixtral-8x7b |
| Others | phi, qwen, opt, gemma |
Activation Caching
Run the model and cache all intermediate activations:
# Get all activations
tokens = model.to_tokens("The Eiffel Tower is in")
logits, cache = model.run_with_cache(tokens)
# Access specific activations
residual = cache["resid_post", 5] # Layer 5 residual stream
attn_pattern = cache["pattern", 3] # Layer 3 attention pattern
mlp_out = cache["mlp_out", 7] # Layer 7 MLP output
# Filter which activations to cache (saves memory)
logits, cache = model.run_with_cache(
tokens,
names_filter=lambda name: "resid_post" in name
)
ActivationCache Keys
| Key Pattern | Shape | Description |
|---|---|---|
resid_pre, layer |
[batch, pos, d_model] | Residual before attention |
resid_mid, layer |
[batch, pos, d_model] | Residual after attention |
resid_post, layer |
[batch, pos, d_model] | Residual after MLP |
attn_out, layer |
[batch, pos, d_model] | Attention output |
mlp_out, layer |
[batch, pos, d_model] | MLP output |
pattern, layer |
[batch, head, q_pos, k_pos] | Attention pattern (post-softmax) |
q, layer |
[batch, pos, head, d_head] | Query vectors |
k, layer |
[batch, pos, head, d_head] | Key vectors |
v, layer |
[batch, pos, head, d_head] | Value vectors |
Workflow 1: Activation Patching (Causal Tracing)
Identify which activations causally affect model output by patching clean activations into corrupted runs.
Step-by-Step
from transformer_lens import HookedTransformer, patching
import torch
model = HookedTransformer.from_pretrained("gpt2-small")
# 1. Define clean and corrupted prompts
clean_prompt = "The Eiffel Tower is in the city of"
corrupted_prompt = "The Colosseum is in the city of"
clean_tokens = model.to_tokens(clean_prompt)
corrupted_tokens = model.to_tokens(corrupted_prompt)
# 2. Get clean activations
_, clean_cache = model.run_with_cache(clean_tokens)
# 3. Define metric (e.g., logit difference)
paris_token = model.to_single_token(" Paris")
rome_token = model.to_single_token(" Rome")
def metric(logits):
return logits[0, -1, paris_token] - logits[0, -1, rome_token]
# 4. Patch each position and layer
results = torch.zeros(model.cfg.n_layers, clean_tokens.shape[1])
for layer in range(model.cfg.n_layers):
for pos in range(clean_tokens.shape[1]):
def patch_hook(activation, hook):
activation[0, pos] = clean_cache[hook.name][0, pos]
return activation
patched_logits = model.run_with_hooks(
corrupted_tokens,
fwd_hooks=[(f"blocks.{layer}.hook_resid_post", patch_hook)]
)
results[layer, pos] = metric(patched_logits)
# 5. Visualize results (layer x position heatmap)
Checklist
- Define clean and corrupted inputs that differ minimally
- Choose metric that captures behavior difference
- Cache clean activations
- Systematically patch each (layer, position) combination
- Visualize results as heatmap
- Identify causal hotspots
Workflow 2: Circuit Analysis (Indirect Object Identification)
Replicate the IOI circuit discovery from "Interpretability in the Wild".
Step-by-Step
from transformer_lens import HookedTransformer
import torch
model = HookedTransformer.from_pretrained("gpt2-small")
# IOI task: "When John and Mary went to the store, Mary gave a bottle to"
# Model should predict "John" (indirect object)
prompt = "When John and Mary went to the store, Mary gave a bottle to"
tokens = model.to_tokens(prompt)
# 1. Get baseline logits
logits, cache = model.run_with_cache(tokens)
john_token = model.to_single_token(" John")
mary_token = model.to_single_token(" Mary")
# 2. Compute logit difference (IO - S)
logit_diff = logits[0, -1, john_token] - logits[0, -1, mary_token]
print(f"Logit difference: {logit_diff.item():.3f}")
# 3. Direct logit attribution by head
def get_head_contribution(layer, head):
# Project head output to logits
head_out = cache["z", layer][0, :, head, :] # [pos, d_head]
W_O = model.W_O[layer, head] # [d_head, d_model]
W_U = model.W_U # [d_model, vocab]
# Head contribution to logits at final position
contribution = head_out[-1] @ W_O @ W_U
return contribution[john_token] - contribution[mary_token]
# 4. Map all heads
head_contributions = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)
for layer in range(model.cfg.n_layers):
for head in range(model.cfg.n_heads):
head_contributions[layer, head] = get_head_contribution(layer, head)
# 5. Identify top contributing heads (name movers, backup name movers)
Checklist
- Set up task with clear IO/S tokens
- Compute baseline logit difference
- Decompose by attention head contributions
- Identify key circuit components (name movers, S-inhibition, induction)
- Validate with ablation experiments
Workflow 3: Induction Head Detection
Find induction heads that implement [A][B]...[A] → [B] pattern.
from transformer_lens import HookedTransformer
import torch
model = HookedTransformer.from_pretrained("gpt2-small")
# Create repeated sequence: [A][B][A] should predict [B]
repeated_tokens = torch.tensor([[1000, 2000, 1000]]) # Arbitrary tokens
_, cache = model.run_with_cache(repeated_tokens)
# Induction heads attend from final [A] back to first [B]
# Check attention from position 2 to position 1
induction_scores = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)
for layer in range(model.cfg.n_layers):
pattern = cache["pattern", layer][0] # [head, q_pos, k_pos]
# Attention from pos 2 to pos 1
induction_scores[layer] = pattern[:, 2, 1]
# Heads with high scores are induction heads
top_heads = torch.topk(induction_scores.flatten(), k=5)
Common Issues & Solutions
Issue: Hooks persist after debugging
# WRONG: Old hooks remain active
model.run_with_hooks(tokens, fwd_hooks=[...]) # Debug, add new hooks
model.run_with_hooks(tokens, fwd_hooks=[...]) # Old hooks still there!
# RIGHT: Always reset hooks
model.reset_hooks()
model.run_with_hooks(tokens, fwd_hooks=[...])
Issue: Tokenization gotchas
# WRONG: Assuming consistent tokenization
model.to_tokens("Tim") # Single token
model.to_tokens("Neel") # Becomes "Ne" + "el" (two tokens!)
# RIGHT: Check tokenization explicitly
tokens = model.to_tokens("Neel", prepend_bos=False)
print(model.to_str_tokens(tokens)) # ['Ne', 'el']
Issue: LayerNorm ignored in analysis
# WRONG: Ignoring LayerNorm
pre_activation = residual @ model.W_in[layer]
# RIGHT: Include LayerNorm
ln_scale = model.blocks[layer].ln2.w
ln_out = model.blocks[layer].ln2(residual)
pre_activation = ln_out @ model.W_in[layer]
Issue: Memory explosion with large models
# Use selective caching
logits, cache = model.run_with_cache(
tokens,
names_filter=lambda n: "resid_post" in n or "pattern" in n,
device="cpu" # Cache on CPU
)
Key Classes Reference
| Class | Purpose |
|---|---|
HookedTransformer |
Main model wrapper with hooks |
ActivationCache |
Dictionary-like cache of activations |
HookedTransformerConfig |
Model configuration |
FactoredMatrix |
Efficient factored matrix operations |
Integration with SAELens
TransformerLens integrates with SAELens for Sparse Autoencoder analysis:
from transformer_lens import HookedTransformer
from sae_lens import SAE
model = HookedTransformer.from_pretrained("gpt2-small")
sae = SAE.from_pretrained("gpt2-small-res-jb", "blocks.8.hook_resid_pre")
# Run with SAE
tokens = model.to_tokens("Hello world")
_, cache = model.run_with_cache(tokens)
sae_acts = sae.encode(cache["resid_pre", 8])
Reference Documentation
For detailed API documentation, tutorials, and advanced usage, see the references/ folder:
| File | Contents |
|---|---|
| references/README.md | Overview and quick start guide |
| references/api.md | Complete API reference for HookedTransformer, ActivationCache, HookPoints |
| references/tutorials.md | Step-by-step tutorials for activation patching, circuit analysis, logit lens |
External Resources
Tutorials
- Main Demo Notebook
- Activation Patching Demo
- ARENA Mech Interp Course - 200+ hours of tutorials
Papers
- A Mathematical Framework for Transformer Circuits
- In-context Learning and Induction Heads
- Interpretability in the Wild (IOI)
Official Documentation
Version Notes
- v2.0: Removed HookedSAE (moved to SAELens)
- v3.0 (alpha): TransformerBridge for loading any nn.Module
Agent operating procedure
- Check the environment. Confirm the model, checkpoint, benchmark version and evaluation harness version.
- Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
- Run a small version first. Evaluate on a small subset to verify prompts, parsing and scoring.
- Execute the full task using the instructions and references above.
- Validate the result. Use the benchmark's official splits and metrics; check for contamination; report variance across seeds or prompts.
- Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
| If this happens | Do this |
|---|---|
| Scores differ from published numbers | Check prompt format, few-shot settings and harness version before concluding. |
| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API. |
| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |
Integrity rules
- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
- Report evaluation settings in full; never cherry-pick prompts, seeds or subsets.
- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.
Related skills
pyvene-interventions: Provides guidance for performing causal interventions on PyTorch models using pyvene's declarative intervention framework.nnsight-remote-interpretability: Provides guidance for interpreting and manipulating neural network internals using nnsight with optional NDIF remote execution.sparse-autoencoder-training: Provides guidance for training and analyzing Sparse Autoencoders (SAEs) using SAELens to decompose neural network activations into interpre…
Trigger Prompts & Evaluation Corpus
These evaluation prompts test that the agent correctly selects transformer-lens-interpretability for relevant tasks and leaves it out for near misses.
Should trigger (3)
- should trigger
“How do I cache all activations for gpt2-small on a prompt and look at layer 5 attention patterns?”
expects Loads HookedTransformer 'gpt2' and uses run_with_cache, then indexes cache['pattern', 5] with shape [batch, head, q_pos, k_pos] (post-softmax). Notes the tokenization/BOS caveat and suggests visualizing per-head patterns.
- should trigger
“I'm working in ioi_patching.ipynb with gpt2-small on 200 IOI prompts (e.g. 'When Mary and John went to the store, John gave a drink to'). I want to run path/activation patching over resid_pre at every layer and position and plot a heatmap of logit-difference recovery, but my corrupted and clean prompts have different token lengths and I get a shape mismatch error in the hook. What should I do?”
expects Explains that clean and corrupted prompts must tokenize to the same length (check with to_tokens and fix the name/template choice), caches the clean run, uses a patching hook on hook_name resid_pre with run_with_hooks per (layer, pos), and normalizes the logit-difference metric between corrupted and clean baselines. Mentions resetting hooks and states assumptions or verifies on a small subset first.
- should trigger
“I trained a tiny 2-layer attention-only transformer on repeated random token sequences and want to figure out which heads copy the token that followed the previous occurrence of the current token, and prove they're doing so causally. How should I go about it?”
expects Recognizes this as induction-head detection and recommends loading the model in TransformerLens (HookedTransformer with a custom config), running repeated random-token sequences, scoring heads by attention to the offset-by-seq_len-minus-one diagonal in the pattern cache, then validating via ablation or patching and checking with direct logit attribution/OV circuits.
Near misses (2)
- near miss
“I need to run activation patching on Llama-3-70B but I have no local GPU. How can I run the interventions remotely?”
expects Recommends nnsight with NDIF for remote execution on very large models instead of TransformerLens, which needs local weights loaded; briefly outlines the nnsight remote workflow and notes the API key requirement.
- near miss
“I want to do interchange intervention training on a PyTorch model with a declarative config of intervention locations and a trainable rotation. How do I set that up?”
expects Identifies this as a pyvene use case (IntervenableConfig, trainable interchange interventions) rather than TransformerLens hooks, and guides setup accordingly.