Scaling Laws for Neural Language Models
OpenAI · · Research
OpenAI researchers showed that language model loss follows predictable power laws across parameters, dataset size, and compute — spanning more than seven orders of magnitude.
Why it matters
Gave the field a quantitative framework for predicting how much compute a model needs. Directly predicted GPT-3's optimal size four months later. Compute allocation became a strategic decision rather than guesswork.
This is LLM World's historical reading, not a claim by OpenAI. The facts above and the sources below are what it rests on.
Sources
Record status: verified · confidence 95/100
arXiv paper 2001.08361, submitted 2020-01-23. The paper is the primary source.