Neel Nanda: Mechanistic Interpretability of Neural Networks

Understanding Mechanistic Interpretability

• 💡 Mechanistic interpretability aims to reverse engineer neural networks, hypothesizing they contain human-comprehensible algorithms despite appearing as black boxes.
• 🧠 The goal is to make the rich, interpretable internal structure of models legible, similar to decompiling a program binary to its source code.
• 🎯 This field seeks to achieve a rigorous understanding of how AI models function, balancing depth with practical utility.

Core Concepts in Neural Network Architecture

• 🤖 Transformers, like those behind ChatGPT, Gemini, and Claude, process sequences of tokens, using a residual stream to accumulate knowledge across layers.
• 🌐 Attention layers move information between words, while MLP layers process information locally, performing much of the network's "thinking."
• 🧩 Features are like variables stored in activations, and circuits are the algorithms implemented in weights to compute and use these features.

Unpacking Model Algorithms: The Grokking Phenomenon

• 🔬 The phenomenon of grokking shows models can suddenly switch from memorization to perfect generalization after extended training on algorithmic tasks like modular addition.
• 🔑 Analysis revealed models learn elegant, comprehensible algorithms, such as representing modular addition through Fourier transforms and rotations around a unit circle.
• ✅ This demonstrates that even complex behaviors can stem from interpretable internal mechanisms, which can be reverse-engineered by examining weights and activations.

Sparse Autoencoders for Feature Discovery

• 📊 Polysemanticity, where single neurons represent multiple unrelated concepts, challenges direct interpretation; models often use superposition to compress many features into shared dimensions.
• 🛠️ Sparse autoencoders (SAEs) are trained to reconstruct model activations with sparsity constraints, effectively decomposing them into independently meaningful and composable features.
• 🚀 SAEs have been successfully scaled to frontier models like GPT-4 and Claude, enabling techniques like steering models by activating or suppressing specific features.

Advanced Techniques and Future Directions

• 🔗 Activation patching is a key causal intervention technique that uses contrast pairs to test the sufficiency and necessity of specific activations for model behavior.
• ⚠️ A primary motivation for mechanistic interpretability is AI safety, helping to distinguish genuinely aligned behavior from deceptive alignment in advanced models.
• 🌱 Ongoing research focuses on improving SAEs, developing better methods for measuring their performance, finding circuits more scalably, and applying them to real-world tasks like detecting hallucinations or jailbreaks.