sparse-autoencoder-training
Provides guidance for training and analyzing Sparse Autoencoders (SAEs) using SAELens to decompose neural network activations into interpretable features. Use when discovering interpretable features, analyzing superposition, or studying monosemantic representations in language models.
Install this skill
Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.
npx skills add KalarisLabs/research-agent-skills --skill sparse-autoencoder-traininguvx research-agent-skills install sparse-autoencoder-trainingDocumentation & Instructions
SAELens: Sparse Autoencoders for Mechanistic Interpretability
SAELens is the primary library for training and analyzing Sparse Autoencoders (SAEs) - a technique for decomposing polysemantic neural network activations into sparse, interpretable features. Based on Anthropic's groundbreaking research on monosemanticity.
GitHub: jbloomAus/SAELens
The Problem: Polysemanticity & Superposition
Individual neurons in neural networks are polysemantic - they activate in multiple, semantically distinct contexts. This happens because models use superposition to represent more features than they have neurons, making interpretability difficult.
SAEs solve this by decomposing dense activations into sparse, monosemantic features - typically only a small number of features activate for any given input, and each feature corresponds to an interpretable concept.
When to Use SAELens
Use SAELens when you need to:
- Discover interpretable features in model activations
- Understand what concepts a model has learned
- Study superposition and feature geometry
- Perform feature-based steering or ablation
- Analyze safety-relevant features (deception, bias, harmful content)
Consider alternatives when:
- You need basic activation analysis → Use TransformerLens directly
- You want causal intervention experiments → Use pyvene or TransformerLens
- You need production steering → Consider direct activation engineering
Installation
pip install sae-lens
Requirements: Python 3.10+, transformer-lens>=2.0.0
Core Concepts
What SAEs Learn
SAEs are trained to reconstruct model activations through a sparse bottleneck:
Input Activation → Encoder → Sparse Features → Decoder → Reconstructed Activation
(d_model) ↓ (d_sae >> d_model) ↓ (d_model)
sparsity reconstruction
penalty loss
Loss Function: MSE(original, reconstructed) + L1_coefficient × L1(features)
Key Validation (Anthropic Research)
In "Towards Monosemanticity", human evaluators found 70% of SAE features genuinely interpretable. Features discovered include:
- DNA sequences, legal language, HTTP requests
- Hebrew text, nutrition statements, code syntax
- Sentiment, named entities, grammatical structures
Workflow 1: Loading and Analyzing Pre-trained SAEs
Step-by-Step
from transformer_lens import HookedTransformer
from sae_lens import SAE
# 1. Load model and pre-trained SAE
model = HookedTransformer.from_pretrained("gpt2-small", device="cuda")
sae, cfg_dict, sparsity = SAE.from_pretrained(
release="gpt2-small-res-jb",
sae_id="blocks.8.hook_resid_pre",
device="cuda"
)
# 2. Get model activations
tokens = model.to_tokens("The capital of France is Paris")
_, cache = model.run_with_cache(tokens)
activations = cache["resid_pre", 8] # [batch, pos, d_model]
# 3. Encode to SAE features
sae_features = sae.encode(activations) # [batch, pos, d_sae]
print(f"Active features: {(sae_features > 0).sum()}")
# 4. Find top features for each position
for pos in range(tokens.shape[1]):
top_features = sae_features[0, pos].topk(5)
token = model.to_str_tokens(tokens[0, pos:pos+1])[0]
print(f"Token '{token}': features {top_features.indices.tolist()}")
# 5. Reconstruct activations
reconstructed = sae.decode(sae_features)
reconstruction_error = (activations - reconstructed).norm()
Available Pre-trained SAEs
| Release | Model | Layers |
|---|---|---|
gpt2-small-res-jb |
GPT-2 Small | Multiple residual streams |
gemma-2b-res |
Gemma 2B | Residual streams |
| Various on HuggingFace | Search tag saelens |
Various |
Checklist
- Load model with TransformerLens
- Load matching SAE for target layer
- Encode activations to sparse features
- Identify top-activating features per token
- Validate reconstruction quality
Workflow 2: Training a Custom SAE
Step-by-Step
from sae_lens import SAE, LanguageModelSAERunnerConfig, SAETrainingRunner
# 1. Configure training
cfg = LanguageModelSAERunnerConfig(
# Model
model_name="gpt2-small",
hook_name="blocks.8.hook_resid_pre",
hook_layer=8,
d_in=768, # Model dimension
# SAE architecture
architecture="standard", # or "gated", "topk"
d_sae=768 * 8, # Expansion factor of 8
activation_fn="relu",
# Training
lr=4e-4,
l1_coefficient=8e-5, # Sparsity penalty
l1_warm_up_steps=1000,
train_batch_size_tokens=4096,
training_tokens=100_000_000,
# Data
dataset_path="monology/pile-uncopyrighted",
context_size=128,
# Logging
log_to_wandb=True,
wandb_project="sae-training",
# Checkpointing
checkpoint_path="checkpoints",
n_checkpoints=5,
)
# 2. Train
trainer = SAETrainingRunner(cfg)
sae = trainer.run()
# 3. Evaluate
print(f"L0 (avg active features): {trainer.metrics['l0']}")
print(f"CE Loss Recovered: {trainer.metrics['ce_loss_score']}")
Key Hyperparameters
| Parameter | Typical Value | Effect |
|---|---|---|
d_sae |
4-16× d_model | More features, higher capacity |
l1_coefficient |
5e-5 to 1e-4 | Higher = sparser, less accurate |
lr |
1e-4 to 1e-3 | Standard optimizer LR |
l1_warm_up_steps |
500-2000 | Prevents early feature death |
Evaluation Metrics
| Metric | Target | Meaning |
|---|---|---|
| L0 | 50-200 | Average active features per token |
| CE Loss Score | 80-95% | Cross-entropy recovered vs original |
| Dead Features | <5% | Features that never activate |
| Explained Variance | >90% | Reconstruction quality |
Checklist
- Choose target layer and hook point
- Set expansion factor (d_sae = 4-16× d_model)
- Tune L1 coefficient for desired sparsity
- Enable L1 warm-up to prevent dead features
- Monitor metrics during training (W&B)
- Validate L0 and CE loss recovery
- Check dead feature ratio
Workflow 3: Feature Analysis and Steering
Analyzing Individual Features
from transformer_lens import HookedTransformer
from sae_lens import SAE
import torch
model = HookedTransformer.from_pretrained("gpt2-small", device="cuda")
sae, _, _ = SAE.from_pretrained(
release="gpt2-small-res-jb",
sae_id="blocks.8.hook_resid_pre",
device="cuda"
)
# Find what activates a specific feature
feature_idx = 1234
test_texts = [
"The scientist conducted an experiment",
"I love chocolate cake",
"The code compiles successfully",
"Paris is beautiful in spring",
]
for text in test_texts:
tokens = model.to_tokens(text)
_, cache = model.run_with_cache(tokens)
features = sae.encode(cache["resid_pre", 8])
activation = features[0, :, feature_idx].max().item()
print(f"{activation:.3f}: {text}")
Feature Steering
def steer_with_feature(model, sae, prompt, feature_idx, strength=5.0):
"""Add SAE feature direction to residual stream."""
tokens = model.to_tokens(prompt)
# Get feature direction from decoder
feature_direction = sae.W_dec[feature_idx] # [d_model]
def steering_hook(activation, hook):
# Add scaled feature direction at all positions
activation += strength * feature_direction
return activation
# Generate with steering
output = model.generate(
tokens,
max_new_tokens=50,
fwd_hooks=[("blocks.8.hook_resid_pre", steering_hook)]
)
return model.to_string(output[0])
Feature Attribution
# Which features most affect a specific output?
tokens = model.to_tokens("The capital of France is")
_, cache = model.run_with_cache(tokens)
# Get features at final position
features = sae.encode(cache["resid_pre", 8])[0, -1] # [d_sae]
# Get logit attribution per feature
# Feature contribution = feature_activation × decoder_weight × unembedding
W_dec = sae.W_dec # [d_sae, d_model]
W_U = model.W_U # [d_model, vocab]
# Contribution to "Paris" logit
paris_token = model.to_single_token(" Paris")
feature_contributions = features * (W_dec @ W_U[:, paris_token])
top_features = feature_contributions.topk(10)
print("Top features for 'Paris' prediction:")
for idx, val in zip(top_features.indices, top_features.values):
print(f" Feature {idx.item()}: {val.item():.3f}")
Common Issues & Solutions
Issue: High dead feature ratio
# WRONG: No warm-up, features die early
cfg = LanguageModelSAERunnerConfig(
l1_coefficient=1e-4,
l1_warm_up_steps=0, # Bad!
)
# RIGHT: Warm-up L1 penalty
cfg = LanguageModelSAERunnerConfig(
l1_coefficient=8e-5,
l1_warm_up_steps=1000, # Gradually increase
use_ghost_grads=True, # Revive dead features
)
Issue: Poor reconstruction (low CE recovery)
# Reduce sparsity penalty
cfg = LanguageModelSAERunnerConfig(
l1_coefficient=5e-5, # Lower = better reconstruction
d_sae=768 * 16, # More capacity
)
Issue: Features not interpretable
# Increase sparsity (higher L1)
cfg = LanguageModelSAERunnerConfig(
l1_coefficient=1e-4, # Higher = sparser, more interpretable
)
# Or use TopK architecture
cfg = LanguageModelSAERunnerConfig(
architecture="topk",
activation_fn_kwargs={"k": 50}, # Exactly 50 active features
)
Issue: Memory errors during training
cfg = LanguageModelSAERunnerConfig(
train_batch_size_tokens=2048, # Reduce batch size
store_batch_size_prompts=4, # Fewer prompts in buffer
n_batches_in_buffer=8, # Smaller activation buffer
)
Integration with Neuronpedia
Browse pre-trained SAE features at neuronpedia.org:
# Features are indexed by SAE ID
# Example: gpt2-small layer 8 feature 1234
# → neuronpedia.org/gpt2-small/8-res-jb/1234
Key Classes Reference
| Class | Purpose |
|---|---|
SAE |
Sparse Autoencoder model |
LanguageModelSAERunnerConfig |
Training configuration |
SAETrainingRunner |
Training loop manager |
ActivationsStore |
Activation collection and batching |
HookedSAETransformer |
TransformerLens + SAE integration |
Reference Documentation
For detailed API documentation, tutorials, and advanced usage, see the references/ folder:
| File | Contents |
|---|---|
| references/README.md | Overview and quick start guide |
| references/api.md | Complete API reference for SAE, TrainingSAE, configurations |
| references/tutorials.md | Step-by-step tutorials for training, analysis, steering |
External Resources
Tutorials
Papers
- Towards Monosemanticity - Anthropic (2023)
- Scaling Monosemanticity - Anthropic (2024)
- Sparse Autoencoders Find Highly Interpretable Features - Cunningham et al. (ICLR 2024)
Official Documentation
- SAELens Docs
- Neuronpedia - Feature browser
SAE Architectures
| Architecture | Description | Use Case |
|---|---|---|
| Standard | ReLU + L1 penalty | General purpose |
| Gated | Learned gating mechanism | Better sparsity control |
| TopK | Exactly K active features | Consistent sparsity |
# TopK SAE (exactly 50 features active)
cfg = LanguageModelSAERunnerConfig(
architecture="topk",
activation_fn="topk",
activation_fn_kwargs={"k": 50},
)
Agent operating procedure
- Check the environment. Confirm the model, checkpoint, benchmark version and evaluation harness version.
- Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
- Run a small version first. Evaluate on a small subset to verify prompts, parsing and scoring.
- Execute the full task using the instructions and references above.
- Validate the result. Use the benchmark's official splits and metrics; check for contamination; report variance across seeds or prompts.
- Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
| If this happens | Do this |
|---|---|
| Scores differ from published numbers | Check prompt format, few-shot settings and harness version before concluding. |
| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API. |
| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |
Integrity rules
- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
- Report evaluation settings in full; never cherry-pick prompts, seeds or subsets.
- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.
Related skills
nnsight-remote-interpretability: Provides guidance for interpreting and manipulating neural network internals using nnsight with optional NDIF remote execution.unsloth: Provides guidance on fine-tuning large language models with Unsloth, a library for faster, lower-memory training using LoRA and QLoRA, base…transformer-lens-interpretability: Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookP…
Trigger Prompts & Evaluation Corpus
These evaluation prompts test that the agent correctly selects sparse-autoencoder-training for relevant tasks and leaves it out for near misses.
Should trigger (3)
- should trigger
“How do I load a pretrained SAE for GPT-2 small and see which features fire on a sentence?”
expects Loads GPT-2 small with TransformerLens/HookedSAETransformer and a matching SAE (e.g. release gpt2-small-res-jb) for the chosen layer and hook point. Encodes activations, lists top-activating features per token, and checks reconstruction quality. Notes that API names should be confirmed against the installed SAELens version.
- should trigger
“I'm training an SAE on layer 8 residual stream of Gemma 2B with SAELens (d_model 2048, d_sae 16384). After 20k steps, about 40% of my features are dead and the L0 is ~600. Here's my config: l1_coefficient=2e-4, lr=5e-4, l1_warm_up_steps=0. What should I change and what metrics should I check?”
expects Diagnoses the problems: no L1 warm-up and an L1 coefficient that is probably mismatched with the sparsity target. Suggests adding L1 warm-up (500-2000 steps), tuning the L1 coefficient, and possibly adjusting lr or resampling and ghost-grad-style fixes. Says to track L0 (target roughly 50-200), CE loss recovered (80-95%), dead feature fraction (<5%) and explained variance (>90%). Does not invent config flags, and asks about or states assumptions about the SAELens version.
- should trigger
“I want to break apart a language model's internal activations into human-understandable concepts, because I think individual neurons are encoding multiple unrelated things at once. How should I go about it, and how do I check the concepts are really meaningful?”
expects Recommends a sparse dictionary-learning approach (an SAE with a sparse bottleneck, trained on activations from a chosen layer) to address polysemanticity and superposition. Covers choosing the layer, expansion factor, L1 or TopK sparsity, and validation by L0, CE recovery, dead features and manual or Neuronpedia inspection of features. Is honest that interpretability is not guaranteed.
Near misses (2)
- near miss
“I just need to cache residual stream activations from GPT-2 and do activation patching on layer 6 attention heads to find which heads matter for indirect object identification. No SAEs involved. How do I set that up?”
expects Points to TransformerLens hooks (run_with_cache, HookPoint patching) for activation patching and circuit analysis, with no SAE training or loading. Recognizes this as a causal intervention task for TransformerLens, not SAELens.
- near miss
“What's the best way to fine-tune Llama 3 8B with QLoRA on my own dataset using limited VRAM?”
expects Gives QLoRA fine-tuning guidance, such as Unsloth or the PEFT setup, 4-bit quantization and memory settings. Does not involve sparse autoencoders or interpretability.