Nanotron Research

HF PRESS

The Ultra-Scale Playbook: Training LLMs on GPU Clusters

Essential reading for anyone scaling ML infrastructure

The knowledge on how to efficiently scale training to large GPU clusters has been well kept within a handful big industry labs. With this book, we set out to lift the veil and release a comprehensive resource on distributed training.

AUTHORS
Nouamane Tazi, Ferdinand Mom, Haolun Zhao, Phuc Nguyen, Mohamed Mekkouri, Leandro Werra, Thomas Wolf

AFFILIATION
Hugging Face

PUBLISHED
Jul 30, 2025

---

This book PDF is accessible with a PRO subscription.

Download Subscribe to PRO

The Nanotron team focus on sharing open knowledge and developping open-source libraries for efficient distributed training of large-scale AI models.

Some of its contributions are:

• the Nanotron library
• the Picotron library
• the Ultrascale-Playbook, a comprehensive book covering all distributed/parallelisation and low-level techniques that can be used to efficiently train models at the largest scales.