Eval Harness
In Progress
Projects
GPT-Neo
Completed
An implementation of model & data-parallel GPT-2 and GPT-3-like models with Mesh Tensorflow.
GPT-NeoX
In Progress
An implementation of 3D-parallel GPT-3-like models for distributed GPUs.
OpenWebText2
Completed
An enhanced version of OpenWebTextCorpus.
The Pile
Completed
A large, diverse, open-source language modelling dataset.