A recent survey paper, "Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety," published on arXiv, investigates whether dissecting the internal computations of AI models contributes to their safety. The authors, a group of 27 researchers from various institutions, argue that while mechanistic interpretability offers a path to understanding, its direct impact on making AI systems safer remains largely unproven.
The paper distinguishes mechanistic interpretability from behavioral evaluations, which primarily assess a model's outputs. Behavioral methods, while scalable, cannot reliably predict how a model will behave with unseen inputs. Mechanistic interpretability, conversely, aims to reverse-engineer the internal representations and computational pathways that produce these outputs.
The survey introduces foundational concepts in mechanistic interpretability, including features, polysemanticity, transformer computation, and circuits. Features refer to directions in activation space that appear to encode specific concepts. Polysemanticity describes the phenomenon where a single neuron can activate for multiple unrelated concepts, complicating simple one-to-one interpretations. The concept of transformer computation and circuits involves tracing a specific capability to a particular subgraph within the neural network.
The researchers survey existing methods for feature extraction and analysis, circuit discovery, and causal verification. They also examine techniques for explaining intermediate representations in natural language, analyzing learning dynamics, and attributing outputs to training data.
Beyond reviewing current approaches, the paper proposes a unified framework for evaluating claims made within mechanistic interpretability. It characterizes six distinct research orientations based on their goals, assumptions, and trade-offs. The authors also released a repository of code notebooks to allow readers to experiment with key findings from the surveyed methods.
The survey considers how mechanistic understanding could support AI safety through knowledge editing, steering, and monitoring. However, it also assesses the limitations, potential failure modes, and open problems within the field. The authors caution that interpretability research could inadvertently accelerate AI capabilities, present dual-use concerns, or lead to overconfidence if explanations are overestimated.
The paper's central question is whether understanding how models work makes them safer. The authors suggest that while internal analysis can reveal and sometimes influence processes that output tests might miss, the assumption that mechanistic understanding directly leads to safer systems requires further validation. They emphasize that mechanistic interpretability may be more relevant to risks arising within a specific model rather than broader systemic risks, which would necessitate complementary approaches.