Peer-reviewedReviewed
Attention Is All You Need
Introduced the Transformer, a sequence model built entirely on self-attention that removed the sequential recurrence of RNNs and enabled full parallelization of training across positions.
Executive summary
The paper replaces recurrent and convolutional encoders with stacked multi-head self-attention and feed-forward layers, using positional encodings to retain word order. Because attention relates all positions at once rather than stepping through a sequence, training parallelizes across the sequence and long-range dependencies are captured in a constant number of operations. It set new machine-translation results on WMT English-German and English-French at lower training cost, and became the base architecture for essentially all later large language models.
Antecedents
- generalizes Neural Machine Translation by Jointly Learning to Align and Translate — Self-attention generalizes content-based soft alignmentDirect evidence
- extends Sequence to Sequence Learning with Neural Networks — Encoder-decoder transduction framing carried over from seq2seqDirect evidence
- depends on Deep Residual Learning for Image Recognition — Residual sublayers (x + Sublayer(x)) in every blockDirect evidence
- depends on Layer Normalization — LayerNorm applied around each sublayerDirect evidence
- depends on Adam — Adam optimizer with warmup scheduleDirect evidence
- depends on Neural Machine Translation of Rare Words with Subword Units — Subword (BPE) vocabulary for open-vocab modelingDirect evidence
- depends on Efficient Estimation of Word Representations — Learned token embeddings feed the inputDirect evidence
- depends on Understanding the Difficulty of Training Deep Nets — Principled init enables deep Transformer trainingStrongly supported
What it enabled — descendants
- applies to Improving Language Understanding by Generative Pre-Training — Decoder-only autoregressive LM built on Transformer decoderDirect evidence
- applies to BERT — Encoder-only masked LM built on Transformer encoderDirect evidence
- applies to Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — Encoder-decoder text-to-text modelDirect evidence
- applies to BART — Encoder-decoder denoising autoencoderDirect evidence
- makes efficient Root Mean Square Layer Normalization — RMSNorm simplifies LayerNorm used in TransformerDirect evidence
- improves GLU Variants Improve Transformer — SwiGLU replaces ReLU FFN in TransformerDirect evidence
- makes efficient Train Short Test Long — ALiBi replaces positional encodings for length extrapolationDirect evidence
- makes efficient On Layer Normalization in the Transformer Architecture — Pre-norm placement stabilizes deep Transformer trainingDirect evidence
- makes efficient Fast Transformer Decoding — MQA shares K/V heads to shrink KV cacheDirect evidence
- applies to Vision Transformer — Transformer applied to image patchesDirect evidence
- makes efficient FlashAttention — FlashAttention makes exact attention IO-efficientDirect evidence
- makes efficient Sparse Attention — Sparse attention reduces attention costDirect evidence
- makes efficient Linearized Attention — Linear attention approximates softmax attentionDirect evidence
- depends on GLM — GLM is built on the Transformer architectureStrongly supported
- depends on A Mathematical Framework for Transformer Circuits — The framework reverse-engineers the Transformer attention blockDirect evidence
- depends on Toy Models of Superposition — Superposition is studied in small Transformer-style networksStrongly supported
- depends on Zero-Shot Text-to-Image Generation — DALL-E autoregressively models image tokens with a TransformerDirect evidence
- depends on Robust Speech Recognition via Large-Scale Weak Supervision — Whisper is an encoder-decoder Transformer for speechDirect evidence
- depends on Highly Accurate Protein Structure Prediction with AlphaFold — AlphaFold Evoformer and structure module are built on attentionStrongly supported
- depends on Titans — Titans augments an attention backbone with a learned long-term memory moduleDirect evidence
- depends on BLOOM — BLOOM is a decoder-only TransformerStrongly supported
Appears in reading paths
- Complete beginner
- ML engineer