Multilingual NLP

Training Data: Quality over Quantity

In a large-scale collaboration, we demonstrated that in addition to being limited in quantity, data for lower-resource languages tends to be lower in quality. This compounds the disparities in performance between high- and low-resource languages. One source of low quality data is incorrect language labels, coming from poor language identification models. Recently, we developed a new language identification benchmark, CommonLID, which shows that most existing models perform even more poorly on web data. This new benchmark can help guide the development of better models, in turn helping us create larger and higher-quality datasets for more languages.

Multilingual Models

Our work also includes influential multilingual models, such as BLOOM, which was pioneering in its scale and degree of openness. As part of the release, the training corpus, ROOTS, was also made available. Follow up work demonstrated the ability to adapt a multilingual model to learn a new language through fine-tuning.

Going Beyond Translation in Evaluation

We have developed benchmarks that go beyond translation to evaluate language models in a culturally relevant way. For example, KMMLU evaluates world knowledge specific to Korea, such as law and history. Most recently, we developed Global PIQA, created in collaboration with over 300 researchers from around the world, encompassing over 140 languages, including some that are rarely represented in NLP. Global PIQA evaluates physical commonsense reasoning, with a focus on culturally specific items and situations, and we found that even the top proprietary systems still perform very poorly on this task for under-represented languages.

These evaluations and more are available in the LM Eval Harness, which represents tasks in over 150 languages.

Learn more about our work across the training stack

• Data curation
  • Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
  • The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
  • Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources
  • CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
• Tokenization
  • BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
  • Evaluating Morphological Alignment of Tokenizers in 70 Languages
  • Explaining and Mitigating Cross-Linguistic Tokenizer Inequalities
• Model Training and Training Dynamics
  • BLOOM: A 176b-parameter open-access multilingual language model
  • A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models
  • On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
  • Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
  • Beetle: Structured Exposure Pretraining in Bilingual Language Models for Modelling L2 Language Processing
• Language Adaptation and Post-Training
  • BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting
  • Crosslingual Generalization through Multitask Finetuning
• Evaluation
  • You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings
  • Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages
  • HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models
  • Re-Evaluating Evaluation for Multilingual Summarization
  • KMMLU: Measuring Massive Multitask Language Understanding in Korean
  • Global PIQA: Evaluating Physical Commonsense Reasoning Across 100+ Languages and Cultures
  • Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
  • Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
• Speech
  • BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models