Training Compute-Optimal Large Language Models
Google DeepMind · · Research
DeepMind showed most existing LLMs (including GPT-3) were significantly undertrained — too many parameters, too little data. For compute-optimal training, model size and training tokens should scale equally.
Why it matters
Corrected the Kaplan scaling laws. Reshaped how the entire field allocated compute budgets. Directly inspired LLaMA, Mistral, and the "smaller but better trained" open-source ecosystem.
This is LLM World's historical reading, not a claim by Google DeepMind. The facts above and the sources below are what it rests on.
Sources
Record status: verified · confidence 95/100
arXiv paper 2203.15556, submitted 2022-03-29. The paper is the primary source.