Skip to content
Research Agent Skillsby Kalaris Labs
Install
ML Evaluation and Safetyv1.0.0AdaptedMIT

transformer-lens-interpretability

Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.

Install this skill

Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.

skills.sh CLI (Recommended)
npx skills add KalarisLabs/research-agent-skills --skill transformer-lens-interpretability
uvx (Python tool runner)
uvx research-agent-skills install transformer-lens-interpretability

Documentation & Instructions

TransformerLens: Mechanistic Interpretability for Transformers

TransformerLens is the de facto standard library for mechanistic interpretability research on GPT-style language models. Created by Neel Nanda and maintained by Bryce Meyer, it provides clean interfaces to inspect and manipulate model internals via HookPoints on every activation.

GitHub: TransformerLensOrg/TransformerLens

When to Use TransformerLens

Use TransformerLens when you need to:

  • Reverse-engineer algorithms learned during training
  • Perform activation patching / causal tracing experiments
  • Study attention patterns and information flow
  • Analyze circuits (e.g., induction heads, IOI circuit)
  • Cache and inspect intermediate activations
  • Apply direct logit attribution

Consider alternatives when:

  • You need to work with non-transformer architectures → Use nnsight or pyvene
  • You want to train/analyze Sparse Autoencoders → Use SAELens
  • You need remote execution on massive models → Use nnsight with NDIF
  • You want higher-level causal intervention abstractions → Use pyvene

Installation

pip install transformer-lens

For development version:

pip install git+https://github.com/TransformerLensOrg/TransformerLens

Core Concepts

HookedTransformer

The main class that wraps transformer models with HookPoints on every activation:

from transformer_lens import HookedTransformer

# Load a model
model = HookedTransformer.from_pretrained("gpt2-small")

# For gated models (LLaMA, Mistral)
import os
os.environ["HF_TOKEN"] = "your_token"
model = HookedTransformer.from_pretrained("meta-llama/Llama-2-7b-hf")

Supported Models (50+)

Family Models
GPT-2 gpt2, gpt2-medium, gpt2-large, gpt2-xl
LLaMA llama-7b, llama-13b, llama-2-7b, llama-2-13b
EleutherAI pythia-70m to pythia-12b, gpt-neo, gpt-j-6b
Mistral mistral-7b, mixtral-8x7b
Others phi, qwen, opt, gemma

Activation Caching

Run the model and cache all intermediate activations:

# Get all activations
tokens = model.to_tokens("The Eiffel Tower is in")
logits, cache = model.run_with_cache(tokens)

# Access specific activations
residual = cache["resid_post", 5]  # Layer 5 residual stream
attn_pattern = cache["pattern", 3]  # Layer 3 attention pattern
mlp_out = cache["mlp_out", 7]  # Layer 7 MLP output

# Filter which activations to cache (saves memory)
logits, cache = model.run_with_cache(
    tokens,
    names_filter=lambda name: "resid_post" in name
)

ActivationCache Keys

Key Pattern Shape Description
resid_pre, layer [batch, pos, d_model] Residual before attention
resid_mid, layer [batch, pos, d_model] Residual after attention
resid_post, layer [batch, pos, d_model] Residual after MLP
attn_out, layer [batch, pos, d_model] Attention output
mlp_out, layer [batch, pos, d_model] MLP output
pattern, layer [batch, head, q_pos, k_pos] Attention pattern (post-softmax)
q, layer [batch, pos, head, d_head] Query vectors
k, layer [batch, pos, head, d_head] Key vectors
v, layer [batch, pos, head, d_head] Value vectors

Workflow 1: Activation Patching (Causal Tracing)

Identify which activations causally affect model output by patching clean activations into corrupted runs.

Step-by-Step

from transformer_lens import HookedTransformer, patching
import torch

model = HookedTransformer.from_pretrained("gpt2-small")

# 1. Define clean and corrupted prompts
clean_prompt = "The Eiffel Tower is in the city of"
corrupted_prompt = "The Colosseum is in the city of"

clean_tokens = model.to_tokens(clean_prompt)
corrupted_tokens = model.to_tokens(corrupted_prompt)

# 2. Get clean activations
_, clean_cache = model.run_with_cache(clean_tokens)

# 3. Define metric (e.g., logit difference)
paris_token = model.to_single_token(" Paris")
rome_token = model.to_single_token(" Rome")

def metric(logits):
    return logits[0, -1, paris_token] - logits[0, -1, rome_token]

# 4. Patch each position and layer
results = torch.zeros(model.cfg.n_layers, clean_tokens.shape[1])

for layer in range(model.cfg.n_layers):
    for pos in range(clean_tokens.shape[1]):
        def patch_hook(activation, hook):
            activation[0, pos] = clean_cache[hook.name][0, pos]
            return activation

        patched_logits = model.run_with_hooks(
            corrupted_tokens,
            fwd_hooks=[(f"blocks.{layer}.hook_resid_post", patch_hook)]
        )
        results[layer, pos] = metric(patched_logits)

# 5. Visualize results (layer x position heatmap)

Checklist

  • Define clean and corrupted inputs that differ minimally
  • Choose metric that captures behavior difference
  • Cache clean activations
  • Systematically patch each (layer, position) combination
  • Visualize results as heatmap
  • Identify causal hotspots

Workflow 2: Circuit Analysis (Indirect Object Identification)

Replicate the IOI circuit discovery from "Interpretability in the Wild".

Step-by-Step

from transformer_lens import HookedTransformer
import torch

model = HookedTransformer.from_pretrained("gpt2-small")

# IOI task: "When John and Mary went to the store, Mary gave a bottle to"
# Model should predict "John" (indirect object)

prompt = "When John and Mary went to the store, Mary gave a bottle to"
tokens = model.to_tokens(prompt)

# 1. Get baseline logits
logits, cache = model.run_with_cache(tokens)

john_token = model.to_single_token(" John")
mary_token = model.to_single_token(" Mary")

# 2. Compute logit difference (IO - S)
logit_diff = logits[0, -1, john_token] - logits[0, -1, mary_token]
print(f"Logit difference: {logit_diff.item():.3f}")

# 3. Direct logit attribution by head
def get_head_contribution(layer, head):
    # Project head output to logits
    head_out = cache["z", layer][0, :, head, :]  # [pos, d_head]
    W_O = model.W_O[layer, head]  # [d_head, d_model]
    W_U = model.W_U  # [d_model, vocab]

    # Head contribution to logits at final position
    contribution = head_out[-1] @ W_O @ W_U
    return contribution[john_token] - contribution[mary_token]

# 4. Map all heads
head_contributions = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)
for layer in range(model.cfg.n_layers):
    for head in range(model.cfg.n_heads):
        head_contributions[layer, head] = get_head_contribution(layer, head)

# 5. Identify top contributing heads (name movers, backup name movers)

Checklist

  • Set up task with clear IO/S tokens
  • Compute baseline logit difference
  • Decompose by attention head contributions
  • Identify key circuit components (name movers, S-inhibition, induction)
  • Validate with ablation experiments

Workflow 3: Induction Head Detection

Find induction heads that implement [A][B]...[A] → [B] pattern.

from transformer_lens import HookedTransformer
import torch

model = HookedTransformer.from_pretrained("gpt2-small")

# Create repeated sequence: [A][B][A] should predict [B]
repeated_tokens = torch.tensor([[1000, 2000, 1000]])  # Arbitrary tokens

_, cache = model.run_with_cache(repeated_tokens)

# Induction heads attend from final [A] back to first [B]
# Check attention from position 2 to position 1
induction_scores = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)

for layer in range(model.cfg.n_layers):
    pattern = cache["pattern", layer][0]  # [head, q_pos, k_pos]
    # Attention from pos 2 to pos 1
    induction_scores[layer] = pattern[:, 2, 1]

# Heads with high scores are induction heads
top_heads = torch.topk(induction_scores.flatten(), k=5)

Common Issues & Solutions

Issue: Hooks persist after debugging

# WRONG: Old hooks remain active
model.run_with_hooks(tokens, fwd_hooks=[...])  # Debug, add new hooks
model.run_with_hooks(tokens, fwd_hooks=[...])  # Old hooks still there!

# RIGHT: Always reset hooks
model.reset_hooks()
model.run_with_hooks(tokens, fwd_hooks=[...])

Issue: Tokenization gotchas

# WRONG: Assuming consistent tokenization
model.to_tokens("Tim")  # Single token
model.to_tokens("Neel")  # Becomes "Ne" + "el" (two tokens!)

# RIGHT: Check tokenization explicitly
tokens = model.to_tokens("Neel", prepend_bos=False)
print(model.to_str_tokens(tokens))  # ['Ne', 'el']

Issue: LayerNorm ignored in analysis

# WRONG: Ignoring LayerNorm
pre_activation = residual @ model.W_in[layer]

# RIGHT: Include LayerNorm
ln_scale = model.blocks[layer].ln2.w
ln_out = model.blocks[layer].ln2(residual)
pre_activation = ln_out @ model.W_in[layer]

Issue: Memory explosion with large models

# Use selective caching
logits, cache = model.run_with_cache(
    tokens,
    names_filter=lambda n: "resid_post" in n or "pattern" in n,
    device="cpu"  # Cache on CPU
)

Key Classes Reference

Class Purpose
HookedTransformer Main model wrapper with hooks
ActivationCache Dictionary-like cache of activations
HookedTransformerConfig Model configuration
FactoredMatrix Efficient factored matrix operations

Integration with SAELens

TransformerLens integrates with SAELens for Sparse Autoencoder analysis:

from transformer_lens import HookedTransformer
from sae_lens import SAE

model = HookedTransformer.from_pretrained("gpt2-small")
sae = SAE.from_pretrained("gpt2-small-res-jb", "blocks.8.hook_resid_pre")

# Run with SAE
tokens = model.to_tokens("Hello world")
_, cache = model.run_with_cache(tokens)
sae_acts = sae.encode(cache["resid_pre", 8])

Reference Documentation

For detailed API documentation, tutorials, and advanced usage, see the references/ folder:

File Contents
references/README.md Overview and quick start guide
references/api.md Complete API reference for HookedTransformer, ActivationCache, HookPoints
references/tutorials.md Step-by-step tutorials for activation patching, circuit analysis, logit lens

External Resources

Tutorials

Papers

Official Documentation

Version Notes

  • v2.0: Removed HookedSAE (moved to SAELens)
  • v3.0 (alpha): TransformerBridge for loading any nn.Module

Agent operating procedure

  1. Check the environment. Confirm the model, checkpoint, benchmark version and evaluation harness version.
  2. Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
  3. Run a small version first. Evaluate on a small subset to verify prompts, parsing and scoring.
  4. Execute the full task using the instructions and references above.
  5. Validate the result. Use the benchmark's official splits and metrics; check for contamination; report variance across seeds or prompts.
  6. Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
If this happens Do this
Scores differ from published numbers Check prompt format, few-shot settings and harness version before concluding.
A function, flag or endpoint in these instructions is missing in the installed version Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API.
A required input, identifier or parameter is ambiguous Ask the user, or state the assumption explicitly before running.

Integrity rules

  • Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
  • Report evaluation settings in full; never cherry-pick prompts, seeds or subsets.
  • Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
  • Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.

Related skills

  • pyvene-interventions: Provides guidance for performing causal interventions on PyTorch models using pyvene's declarative intervention framework.
  • nnsight-remote-interpretability: Provides guidance for interpreting and manipulating neural network internals using nnsight with optional NDIF remote execution.
  • sparse-autoencoder-training: Provides guidance for training and analyzing Sparse Autoencoders (SAEs) using SAELens to decompose neural network activations into interpre…

Trigger Prompts & Evaluation Corpus

These evaluation prompts test that the agent correctly selects transformer-lens-interpretability for relevant tasks and leaves it out for near misses.

Should trigger (3)

  • should trigger

    “How do I cache all activations for gpt2-small on a prompt and look at layer 5 attention patterns?”

    expects Loads HookedTransformer 'gpt2' and uses run_with_cache, then indexes cache['pattern', 5] with shape [batch, head, q_pos, k_pos] (post-softmax). Notes the tokenization/BOS caveat and suggests visualizing per-head patterns.

  • should trigger

    “I'm working in ioi_patching.ipynb with gpt2-small on 200 IOI prompts (e.g. 'When Mary and John went to the store, John gave a drink to'). I want to run path/activation patching over resid_pre at every layer and position and plot a heatmap of logit-difference recovery, but my corrupted and clean prompts have different token lengths and I get a shape mismatch error in the hook. What should I do?”

    expects Explains that clean and corrupted prompts must tokenize to the same length (check with to_tokens and fix the name/template choice), caches the clean run, uses a patching hook on hook_name resid_pre with run_with_hooks per (layer, pos), and normalizes the logit-difference metric between corrupted and clean baselines. Mentions resetting hooks and states assumptions or verifies on a small subset first.

  • should trigger

    “I trained a tiny 2-layer attention-only transformer on repeated random token sequences and want to figure out which heads copy the token that followed the previous occurrence of the current token, and prove they're doing so causally. How should I go about it?”

    expects Recognizes this as induction-head detection and recommends loading the model in TransformerLens (HookedTransformer with a custom config), running repeated random-token sequences, scoring heads by attention to the offset-by-seq_len-minus-one diagonal in the pattern cache, then validating via ablation or patching and checking with direct logit attribution/OV circuits.

Near misses (2)

  • near miss

    “I need to run activation patching on Llama-3-70B but I have no local GPU. How can I run the interventions remotely?”

    expects Recommends nnsight with NDIF for remote execution on very large models instead of TransformerLens, which needs local weights loaded; briefly outlines the nnsight remote workflow and notes the API key requirement.

  • near miss

    “I want to do interchange intervention training on a PyTorch model with a declarative config of intervention locations and a trainable rotation. How do I set that up?”

    expects Identifies this as a pyvene use case (IntervenableConfig, trainable interchange interventions) rather than TransformerLens hooks, and guides setup accordingly.