Training Data

Training data shapes what a model learns. We build and document open corpora so researchers can inspect that foundation and study how changing it changes a model.

The Pile brought together diverse sources of text in an 800GB language-modeling dataset. The Common Pile extends our work on open training data with an 8TB collection of public domain and openly licensed text.

Releasing a corpus also means documenting where its contents came from and how they were selected. This connects dataset construction to research on filtering, licensing, and the effects of data composition.

Selected Papers

Newest first. Showing 12 of 38.

Training Data

Selected Papers

Bergson: An Open Source Library for Data Attribution

Where Does Social Reasoning Come From? Capability Provenance in Language Models

Quantifying the Effect of Test Set Contamination on Generative Evaluations

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution

Mitigating Emergent Misalignment with Data Attribution

The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon

Bridging the Data Provenance Gap Across Text, Speech, and Video

Aria-MIDI: A Dataset of MIDI Files for Symbolic Music Modeling

Robin: a Suite of Multi-Scale Vision-Language Models and the CHIRP Evaluation Benchmark