EleutherAI

A grassroots collective of researchers working to open source AI research.

News & Announcements

  • 2021⁠-⁠03⁠-⁠31
  • GPT⁠-⁠Neo
    GPT⁠-⁠Neo 1.3B and 2.7B are now available on Hugging Face Model Hub! Run the models with Transformers or call for them through their on-demand Inference API.
    EleutherAI on Model Hub >

  • 2021⁠-⁠03⁠-⁠21
  • GPT⁠-⁠Neo
    GPT⁠-⁠Neo 1.3B and 2.7B, trained on the Pile, are now available to run with the GPT⁠-⁠Neo framework.
    GPT-Neo on Github >

  • 2021⁠-⁠01⁠-⁠01
  • The Pile
    We are proud to announce the release of the Pile, a free and publicly available 800GB dataset of diverse English text for language modeling!
    Visit the Pile >

Projects

Large Language models

  • GPT⁠-⁠Neo

    • alt
    • An implementation of model & data-parallel GPT⁠-⁠2 and GPT⁠-⁠3-like models, utilizing Mesh Tensorflow for distributed support.
    • completed
  • GPT⁠-⁠NeoX

    • alt
    • An implementation of 3D-parallel GPT⁠-⁠3-like models on distributed GPUs, based upon DeepSpeed and Megatron-LM.
    • in progress

  • The Pile

    • alt
    • The Pile is an 825 GiB diverse, open source language modelling dataset consisting of data from 22 high quality sources. It is useful for both training and benchmarking large language models.
    • completed
  • OpenWebText2

    • alt
    • OpenWebText2 is a dataset inspired by WebText, created by scraping URLs extracted from Reddit submissions up until April 2020 with a minimum score of 3 as a proxy for quality. It features content from multiple languages, document metadata, multiple dataset versions, and open source replication code.
    • completed