Peer-reviewedReviewed
Training Compute-Optimal Large Language Models (Chinchilla)
Corrected the earlier scaling prescription by showing parameters and training tokens should be scaled in roughly equal proportion, revealing that then-current large models were badly undertrained for their compute budgets.
Executive summary
DeepMind trained over 400 models at varied size and token counts and re-estimated the compute-optimal frontier, finding that for a given compute budget parameters and data should grow together at about a 1:1 ratio rather than favoring size. To demonstrate it they trained Chinchilla, a 70B model on 1.4 trillion tokens, which outperformed the 280B Gopher and other larger models while being cheaper to run at inference. The result redirected the field toward training smaller models on far more data.
Antecedents
Challenged, corrected, or evaluated by
- challenges Scaling Laws for Neural Language Models — Chinchilla corrects Kaplan optimal N/D allocationDirect evidence
What it enabled — descendants
- enables Scaling Data-Constrained Language Models — Chinchilla token hunger raises the data-scarcity questionStrongly supported
- enables LLaMA — Chinchilla-informed over-training shapes LLaMADirect evidence