Open Models & Infrastructure

Open research on large models requires more than released weights. It requires open data to train on, open code to train and evaluate with, and models whose full development is public. We have built all three layers since 2020.

The Pile established that a curated, documented, openly available pretraining corpus could support competitive models, and GPT-NeoX-20B was for a time the largest open-source language model in the world. The Common Pile continues that work with an 8TB corpus of public domain and openly licensed text, and the Comma models trained on it show what is possible when licensing is taken seriously from the start.

Our training framework, GPT-NeoX, and our high-performance computing team provide training infrastructure as a service to our own research teams and to external groups. That infrastructure is what makes controlled model suites such as Pythia, the filtering scaling suites used in open-weight safety research, and model organisms for behavioral safety possible. We also contribute to architecture research, including the RWKV family of recurrent models and work on optimizers and hyperparameter transfer.

Current Directions

Pythia 2.0
A successor model suite spanning multiple training stages, with parallel runs using different post-training algorithms so researchers can study how those choices shape behavior.
Openly licensed data
Growing the Common Pile and studying how license-clean data affects model quality across languages and domains.
Infrastructure as a service
Training and evaluation infrastructure for internal teams and for the external research communities we support.

Selected Papers

Newest first. Showing 12 of 57.

No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance

Lopardo, Frentzen Salim, Srivastava, Jiang, Sharma, and Arnett. EMNLP Findings, 2026

Beetle: Structured Exposure Pretraining in Bilingual Language Models for Modelling L2 Language Processing

Salhan, Arnett, Michaelov, and Buttery. EMNLP, 2026

All 57 papers in this area