Peer-reviewedReviewed
Language Models are Few-Shot Learners (GPT-3)
Showed that scaling an autoregressive language model to 175B parameters yields in-context few-shot learning, where tasks are specified through examples in the prompt rather than by fine-tuning weights.
Executive summary
GPT-3 trains a 175-billion-parameter Transformer on a filtered Common Crawl plus other corpora, keeping the next-token objective but scaling roughly 100x over GPT-2. Given a natural-language instruction and a handful of demonstrations in its context window, it performs translation, question answering, arithmetic, and other tasks without weight updates, with accuracy generally rising as more examples are shown. This removed the per-task fine-tuning and labeled-data requirement for many uses and made prompting the primary interface to large models.
Antecedents
Challenged, corrected, or evaluated by
- scales Language Models are Unsupervised Multitask Learners — Scaled to 175B; few-shot in-context learning emergesDirect evidence
- provides evidence for In-context Learning and Induction Heads — Induction heads give a mechanistic account of in-context learningStrongly supported
- challenges On the Dangers of Stochastic Parrots — Critiques the scale-first paradigm on cost bias and environmental groundsStrongly supported
What it enabled — descendants
- provides evidence for Scaling Laws for Neural Language Models — GPT-3-era models motivate the scaling-law studyDirect evidence
- enables Chain-of-Thought Prompting Elicits Reasoning — Chain-of-thought emerges only at scaleDirect evidence
- depends on WebGPT — WebGPT fine-tunes GPT-3 to browse and answer with citationsDirect evidence
- applies to Do As I Can, Not As I Say — SayCan grounds a language model in a robot value function of affordancesDirect evidence
Appears in reading paths
- Complete beginner