Eval Harness

In Progress

GPT-Neo

Completed

An implementation of model & data-parallel GPT-2 and GPT-3-like models with Mesh Tensorflow.

GPT-NeoX

In Progress

An implementation of 3D-parallel GPT⁠-⁠3-like models for distributed GPUs.

OpenWebText2

Completed

An enhanced version of OpenWebTextCorpus.

The Pile

Completed

A large, diverse, open-source language modelling dataset.