Interpretability Over Time

A scientific understanding of language models needs more than post-hoc analysis of a trained checkpoint. It requires knowing how behaviors arise over training, and that in turn requires the artifacts to study it: intermediate checkpoints, controlled repeated runs, and the exact data seen at each step. Our Pythia suite pioneered this approach and has become the standard instrument for research on learning dynamics, memorization, and how capabilities form. The position paper Don’t Just “Fix It in Post”: A Science of AI Must Study Training Dynamics (ICML 2026 oral) lays out the argument.

We work on this at two levels of abstraction. Learning mechanics treats neural networks as mathematical objects: optimization dynamics, loss landscapes, hyperparameter regimes, representation geometry, feature learning, and scaling behavior. Training dynamics is concerned with the development of semantic behaviors such as memorization, factual recall, compositional generalization, and bias, and is often sensitive to the contents of the training data. Foundational results from the first level tend to inspire methods at the second.

Data attribution connects specific model behaviors to the training data that shaped them, and we see it as a central tool for these questions. Our Bergson library contains the first public implementation of many leading attribution techniques. We use it both to study attribution as a method and to ask how data contents shape model behavior, including as a way to select what to filter out of pretraining data in the first place.

We also treat models as the product of multiple stages of training, from pretraining through midtraining, supervised fine-tuning, and preference optimization, rather than as a single artifact. Understanding how each stage contributes to final behavior is undervalued in the field and is something we deliberately center.