Peer-reviewed2017
Attention Is All You Need
Introduced the Transformer, a sequence model built entirely on self-attention that removed the sequential recurrence of RNNs and enabled full parallelization of training across positions.
The directory
211 records — 211 editorially reviewed with confirmed primary-source links. Filtering and sorting happen instantly in your browser.
211 papers
Peer-reviewed2022
FlashAttention (1/2/3): IO-Aware Exact Attention
Computes exact attention with an IO-aware tiling and online-softmax algorithm that avoids materializing the full attention matrix in GPU high-bandwidth memory, removing the memory and bandwidth bottleneck that made long sequences slow and costly.
Preprint2015
Distilling the Knowledge in a Neural Network
Showed that a small student network can be trained to match a large teacher’s softened output distribution, transferring most of the teacher’s accuracy into a far cheaper model and turning model size into a deployment choice rather than a fixed cost.
Technical report2023
LLaMA: Open and Efficient Foundation Language Models
Showed that training smaller models on far more publicly available tokens produces compute-efficient foundation models, removing the assumption that strong performance required proprietary data or the largest parameter counts.
Foundational paper1986
Backpropagation (Learning representations by back-propagating errors)
Introduced the backpropagation algorithm for training multi-layer neural networks by propagating output error backward through the network to compute weight gradients, removing the barrier that networks with hidden layers had no practical learning rule.
Peer-reviewed2014
Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau attention)
Introduced an attention mechanism that lets the decoder compute a weighted combination of all encoder hidden states at each output step, removing the fixed-length bottleneck vector that capped encoder-decoder translation quality on long sentences.
Peer-reviewed2015
Deep Residual Learning for Image Recognition (ResNet)
Introduced residual learning with identity skip connections so layers fit a residual function rather than a full mapping, removing the degradation problem that made very deep networks harder to optimize and less accurate.
Peer-reviewed2014
Adam: A Method for Stochastic Optimization
Introduced Adam, an optimizer that maintains per-parameter adaptive learning rates from running estimates of the first and second moments of the gradients, removing much of the manual learning-rate tuning and poor conditioning that hampered SGD on sparse or noisy gradients.
Foundational paper1936
On Computable Numbers (Turing Machine)
Turing defined a precise abstract model of mechanical computation and used it to characterize which functions can be computed and to prove that some well-defined problems cannot.
Architecture report1945
First Draft of a Report on the EDVAC (stored-program architecture)
Von Neumann's report set out an architecture in which a computer's instructions are stored in the same memory as its data, letting a single machine be reprogrammed without rewiring.
Foundational paper1948
A Mathematical Theory of Communication (information theory)
Shannon founded information theory by defining information quantitatively and proving limits on how much data a channel can carry and how far messages can be compressed.
Peer-reviewed2020
Language Models are Few-Shot Learners (GPT-3)
Showed that scaling an autoregressive language model to 175B parameters yields in-context few-shot learning, where tasks are specified through examples in the prompt rather than by fine-tuning weights.
Peer-reviewed2018
BERT: Pre-training of Deep Bidirectional Transformers
Introduced BERT, a Transformer encoder pre-trained with masked-language-modeling and next-sentence prediction to produce deeply bidirectional representations usable across tasks via light fine-tuning.
Peer-reviewed2022
Training Compute-Optimal Large Language Models (Chinchilla)
Corrected the earlier scaling prescription by showing parameters and training tokens should be scaled in roughly equal proportion, revealing that then-current large models were badly undertrained for their compute budgets.
Peer-reviewed2022
InstructGPT: Training LMs to Follow Instructions with Human Feedback
Applied reinforcement learning from human feedback to GPT-3 so a much smaller aligned model followed user intent better than the raw base model, closing the gap between next-token pretraining and doing what a user actually asked.
Peer-reviewed2021
CLIP: Learning Transferable Visual Models from NL Supervision
Introduced CLIP, which trains image and text encoders jointly on 400 million web image-caption pairs with a contrastive objective, removing the need for fixed labeled classification datasets and enabling zero-shot recognition from natural-language prompts.
Peer-reviewed2009
ImageNet: A Large-Scale Hierarchical Image Database
Built a labelled image database of unprecedented scale and hierarchical structure, giving computer vision the large supervised benchmark it needed and creating the competition that deep learning went on to win.
Peer-reviewed2021
RoFormer: Rotary Position Embedding (RoPE)
Introduced rotary position embedding, which encodes position by rotating query and key vectors so their dot product depends only on relative distance, removing the need for additive position embeddings that generalize poorly to longer sequences.
Preprint2020
Scaling Laws for Neural Language Models
Established that language-model test loss follows smooth power laws in model size, dataset size, and training compute, turning architecture-and-scale guesswork into predictable extrapolation.
Peer-reviewed2018
Mixed Precision Training
Showed that most neural-network operations can run in 16-bit floating point rather than 32-bit, roughly halving memory use and speeding training on tensor hardware without accuracy loss.
Infrastructure2019
Megatron-LM: Training Multi-Billion Parameter LMs Using Model Parallelism
Introduced a simple tensor (intra-layer) model-parallel scheme for Transformers that splits individual matrix multiplications across GPUs, letting models grow past a single device's memory using only standard framework primitives.
Infrastructure2020
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Removed the memory redundancy of data-parallel training by partitioning optimizer states, gradients, and parameters across devices instead of replicating them, making it feasible to train models with orders of magnitude more parameters.
Peer-reviewed2022
Chain-of-Thought Prompting Elicits Reasoning
Showed that prompting a large model to emit intermediate reasoning steps before its answer unlocks multi-step reasoning that direct-answer prompting fails at, without any fine-tuning.
Technical report2025
DeepSeek-R1: Incentivizing Reasoning via RL (RLVR)
It showed that a base language model can be trained to produce long chains of reasoning largely through reinforcement learning against automatically checkable rewards (correct answers, passing tests), without first needing large amounts of human-written reasoning demonstrations.
Peer-reviewed2021
Vision Transformer (An Image is Worth 16x16 Words)
Introduced the Vision Transformer, showing that a plain Transformer applied to fixed-size image patches can match or beat convolutional networks on image classification, removing the assumption that vision required convolution-specific inductive biases.
Peer-reviewed2020
Retrieval-Augmented Generation (RAG)
Introduced retrieval-augmented generation, coupling a parametric sequence-to-sequence model with a dense-retrieval index so a language model can pull in external documents at generation time instead of relying only on facts baked into its weights.
Peer-reviewed2023
PagedAttention / vLLM
Manages the key-value cache in non-contiguous fixed-size blocks like virtual-memory paging, removing the fragmentation that wasted most KV-cache memory under contiguous per-request allocation.
Peer-reviewed2023
Speculative Decoding / Medusa / EAGLE
Introduced speculative decoding, which uses a small fast draft model to propose several tokens that the large target model verifies in parallel, cutting the number of expensive sequential steps without changing the output distribution.
Peer-reviewed2022
Inference Quantization (GPTQ/AWQ/SmoothQuant/bitsandbytes)
Introduced GPTQ, a one-shot post-training weight quantization method that compresses large language model weights to about 3-4 bits using approximate second-order (Hessian) information, letting big models run on far less memory with little accuracy loss.
Peer-reviewed2022
LoRA / QLoRA (parameter-efficient fine-tuning)
Freezes the pretrained weights and injects trainable low-rank matrices into each layer, removing the storage and memory cost of full fine-tuning while adding no inference latency.
Essay2019
The Bitter Lesson
Argued from 70 years of AI history that general methods leveraging more computation reliably beat approaches built on human-designed knowledge, a lesson repeatedly relearned and central to the scaling era.
Peer-reviewed2009
The Unreasonable Effectiveness of Data
Argued that for many hard problems, simple models trained on far more data outperform sophisticated models on less, shifting emphasis from clever algorithms to data scale.
Technical report2024
Llama 2 & Llama 3
A detailed engineering report on Meta's Llama family of open-weight dense transformer models, documenting the data, scaling, and post-training recipe that made a widely downloadable model competitive with the strongest closed models of its time.
Technical report2024
DeepSeek (V2/V3; MLA + efficient MoE + FP8)
A technical report on DeepSeek-V3, a large mixture-of-experts model that combined multi-head latent attention, a load-balanced sparse expert design, and FP8 mixed-precision training to reach frontier-level quality at substantially lower training cost.
Technical report2024
Qwen (Qwen/Qwen2/Qwen2.5)
A technical report on Alibaba's Qwen2.5 family, a set of open-weight models spanning many sizes trained on a much larger pretraining corpus, with strong multilingual, coding, math, and long-context performance.
Peer-reviewed2014
Sequence to Sequence Learning with Neural Networks
Introduced a general end-to-end LSTM encoder-decoder that maps an input sequence to a fixed-length vector and decodes it into an output sequence, removing the need for fixed-alignment, feature-engineered pipelines in sequence transduction.
Preprint2016
Layer Normalization
Introduced layer normalization, which normalizes activations across the features of a single training example rather than across the batch, removing batch-size dependence and making normalization usable in recurrent networks and at inference on single examples.
Peer-reviewed2015
Neural Machine Translation of Rare Words with Subword Units (BPE)
Introduced byte pair encoding as a subword segmentation method for translation, letting a fixed vocabulary represent rare and unseen words as sequences of learned subword units and removing the out-of-vocabulary problem in open-vocabulary translation.
Peer-reviewed2012
AlexNet: ImageNet Classification with Deep CNNs
Showed that a large deep convolutional network trained on GPUs could win the ImageNet classification benchmark by a wide margin, demonstrating that scale of data, model, and compute together made deep learning practical for large-scale image recognition.
Peer-reviewed1997
Long Short-Term Memory (LSTM)
Introduced the Long Short-Term Memory recurrent architecture, which uses a memory cell protected by multiplicative gates to preserve error signals over long time lags, addressing the vanishing and exploding gradients that prevented standard recurrent networks from learning long-range dependencies.
Thesis1937
A Symbolic Analysis of Relay and Switching Circuits
Shannon showed that networks of relay switches correspond directly to expressions in Boolean logic, so circuits can be designed and simplified using algebra instead of trial and error.
Foundational paper1943
A Logical Calculus of Ideas Immanent in Nervous Activity (artificial neuron)
McCulloch and Pitts proposed a mathematical model of neurons as simple threshold logic units and showed that networks of them can compute logical functions.
Vision essay1945
As We May Think (Memex) + Science the Endless Frontier
Bush proposed the Memex, an imagined desk-based device for storing documents and linking them by association, anticipating the idea of navigable connected information.
Foundational paper1950
Computing Machinery and Intelligence (Turing Test)
Turing reframed the question 'can machines think' as an operational test of whether a machine's conversational responses are indistinguishable from a human's.
Institutional milestone1945
The Manhattan Project / Los Alamos (big-science template)
The Manhattan Project organized dispersed scientific talent, industrial-scale engineering, and government funding into a single goal-directed effort, establishing the template for state-backed 'big science' that later underwrote computing, national laboratories, and large research programs.
Method1949
The Monte Carlo Method / Metropolis Algorithm
The Monte Carlo method and the Metropolis algorithm introduced the idea of solving hard mathematical and physical problems by drawing large numbers of random samples, giving computers a general way to estimate quantities that cannot be calculated exactly.
Founding event1956
The Dartmouth Workshop (birth of Artificial Intelligence)
The 1956 Dartmouth Summer Research Project named the field of artificial intelligence and set its founding agenda, proposing that learning and intelligence could be described precisely enough for a machine to simulate them.
Foundational paper1958
The Perceptron
Rosenblatt's Perceptron introduced a trainable model that adjusts numerical weights from examples to classify inputs, giving a concrete mechanism for machine learning and serving as the direct ancestor of neural networks.
Technical report2018
Improving Language Understanding by Generative Pre-Training (GPT)
Showed that a single Transformer decoder pre-trained on unlabeled text with a language-modeling objective, then fine-tuned per task, could beat task-specific architectures across diverse NLP benchmarks.
Technical report2019
Language Models are Unsupervised Multitask Learners (GPT-2)
Demonstrated that scaling a decoder-only language model and its training corpus lets it perform many NLP tasks with no gradient updates, purely by conditioning on a prompt.
Peer-reviewed2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
Introduced T5, which casts every NLP task as text-to-text generation, allowing one model, objective, and decoding procedure to cover classification, translation, summarization, and question answering.
Peer-reviewed2017
Outrageously Large Neural Networks: Sparsely-Gated MoE
Introduced the sparsely-gated mixture-of-experts layer, where a trainable gating network routes each input to a few of thousands of expert sub-networks, letting capacity grow enormously without a proportional increase in per-example compute.
Peer-reviewed2022
Switch Transformer
Introduced the Switch Transformer, which simplifies mixture-of-experts routing to a single expert per token and pairs it with stability and distributed-training techniques to train sparse models up to a trillion parameters.
Software release2018
SentencePiece
It packaged subword tokenization (BPE and unigram language-model segmentation) into a single self-contained tool that trains directly from raw untokenized text and treats input as a reversible byte/character stream, removing the dependence on language-specific pre-tokenizers and making tokenization reproducible and fully invertible across languages.
Peer-reviewed2022
Deduplicating Training Data Makes Language Models Better
It showed that standard language-model training corpora contain large amounts of near- and exact-duplicate text, and that removing these duplicates with scalable exact and approximate (suffix-array and MinHash) methods reduces memorization, cleans up train/test overlap, and lets models reach the same or better quality with fewer training steps.
Dataset paper2021
The Pile
Assembled an 800GB English text corpus from 22 curated, diverse sources, giving open language-model training a large, documented, multi-domain dataset instead of ad hoc web scrapes.
Dataset paper2023
The RefinedWeb Dataset for Falcon LLM
It showed that aggressive filtering and deduplication of Common Crawl alone can produce web text good enough to train competitive LLMs, challenging the assumption that curated sources like books and Wikipedia are required.
Technical report2023
Textbooks Are All You Need (phi)
It showed that training a small code model on a small corpus of textbook-quality and synthetically generated exercises yields coding ability far above what its parameter and token count would predict, shifting emphasis from data quantity to data quality.
Peer-reviewed2019
GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism
GPipe partitions a deep model across accelerators as pipeline stages and splits each minibatch into micro-batches, letting a network too large for one device be trained while keeping most devices busy.
Infrastructure2018
Mesh-TensorFlow / GSPMD
GSPMD lets a user annotate a few tensors with sharding specifications and then automatically propagates and compiles those into a partitioned program, so the same model graph can be split across devices by data, operator, or pipeline dimensions without rewriting it.
Infrastructure2023
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
PyTorch FSDP shards each layer's parameters, gradients, and optimizer state across data-parallel workers and reconstructs full parameters on the fly only for the layer being computed, cutting per-device memory so large models fit under standard data-parallel training.
Preprint2016
Training Deep Nets with Sublinear Memory Cost (Activation Checkpointing)
This work trades compute for memory by discarding most intermediate activations during the forward pass and recomputing them during the backward pass, cutting activation memory for an n-layer network from linear to about the square root of n.
Peer-reviewed2022
Finetuned Language Models Are Zero-Shot Learners (FLAN) / Self-Instruct
It showed that fine-tuning a language model on a broad collection of tasks phrased as natural-language instructions makes the model follow instructions for unseen tasks it was never trained on, removing the need for task-specific examples or prompt engineering to steer a model.
Technical report2022
Constitutional AI: Harmlessness from AI Feedback (RLAIF)
Replaced most human-labeled harmlessness feedback with model-generated feedback governed by a written set of principles, removing the need for large volumes of human labels on harmful content to train a harmless assistant.
Peer-reviewed2023
Direct Preference Optimization (DPO)
Showed the RLHF objective can be solved by a single classification-style loss on preference pairs, eliminating the separate reward model and reinforcement-learning loop.
Peer-reviewed2023
ReAct: Synergizing Reasoning and Acting in LMs
Interleaves free-text reasoning traces with discrete actions in a single prompting loop, removing the split between chain-of-thought reasoning (which cannot gather new information) and action-only agents (which cannot plan or revise).
Peer-reviewed2022
Orca: Continuous (Iteration-Level) Batching
Introduced iteration-level (continuous) batching for transformer serving, letting the server add and retire requests at each generation step instead of processing a batch to completion, which sharply raised GPU throughput for LLM inference.
Software release2023
Edge/Format Ecosystem (llama.cpp/GGUF/TensorRT-LLM/SGLang)
This ecosystem built the runtimes and file formats that let LLMs run efficiently on consumer and production hardware, with SGLang contributing RadixAttention, which reuses shared prefix key-value cache across requests via a radix tree to speed up structured and multi-call generation.
Benchmark2024
SWE-bench (repository-level SWE)
Evaluates language models on resolving real GitHub issues by requiring a repository-wide code patch that makes the project's actual test suite pass, removing the artificiality of self-contained single-function coding tasks.
Peer-reviewed2021
Highly Accurate Protein Structure Prediction with AlphaFold
Showed that a deep-learning system built on attention could predict a protein 3D structure from its amino-acid sequence at near-experimental accuracy, solving a decades-old grand challenge in biology and demonstrating that attention-based models generalize far beyond language.
Peer-reviewed1992
Simple Statistical Gradient-Following Algorithms (REINFORCE)
Introduced REINFORCE, a family of algorithms that adjust a policy's parameters directly along an unbiased estimate of the reward gradient, enabling reinforcement learning in networks that output probabilities over actions.
Peer-reviewed2015
Human-level Control through Deep RL (DQN)
Introduced the Deep Q-Network (DQN), which combined Q-learning with a convolutional neural network to learn control policies directly from raw pixels, reaching human-level play across many Atari games with one architecture.
Peer-reviewed2016
Mastering the Game of Go (AlphaGo/AlphaZero)
Introduced AlphaGo and later AlphaZero, systems that combined deep neural networks with Monte Carlo tree search to master Go and other board games, with AlphaZero learning entirely from self-play with no human game data.
Peer-reviewed2014
Generative Adversarial Networks (GAN)
Introduced generative adversarial networks (GANs), which train a generator and a discriminator against each other so the generator learns to produce data indistinguishable from real examples.
Foundational book1949
The Organization of Behavior (Hebbian learning)
Hebb proposed that learning happens by strengthening the connection between neurons that are repeatedly active together, giving a physical mechanism for memory and association.
Wartime engineering1943
Enigma / Bletchley Park codebreaking (Bombe / Colossus)
The Bletchley Park codebreaking effort, using electromechanical Bombes and the electronic Colossus, showed that machines could automate large-scale symbolic reasoning and search, and it drove practical advances in programmable computing hardware.
Foundational book1944
Theory of Games and Economic Behavior (game theory)
'Theory of Games and Economic Behavior' founded game theory as a formal framework for analyzing decisions among competing agents, providing the mathematical language of strategies, payoffs, and equilibria used across economics, AI, and multi-agent systems.
Peer-reviewed2019
Root Mean Square Layer Normalization (RMSNorm)
Introduced RMSNorm, which normalizes activations by their root-mean-square alone and drops the mean-centering and bias of LayerNorm, cutting normalization cost while keeping training stability.
Preprint2020
GLU Variants Improve Transformer (SwiGLU)
Showed that replacing the transformer feed-forward layer's ReLU with gated linear unit variants, especially SwiGLU, yields better quality at matched compute.
Peer-reviewed2020
On Layer Normalization in the Transformer Architecture (pre-norm)
It showed analytically and empirically that placing layer normalization inside the residual branch (Pre-LN) instead of after it (Post-LN) keeps gradients well-scaled at initialization, removing the need for the learning-rate warm-up stage that Post-LN Transformers required for stable training.
Peer-reviewed2023
GQA: Training Generalized Multi-Query Transformer Models
Introduced grouped-query attention, an interpolation between multi-head and multi-query attention that shares each key/value head across a small group of query heads, plus a recipe to uptrain existing multi-head checkpoints into this form.
Technical report2024
Mixtral of Experts
It released an open-weight sparse mixture-of-experts language model in which a router picks 2 of 8 expert feed-forward blocks per token per layer, giving the quality of a large (~47B-parameter) model while only activating about 13B parameters per token, cutting the compute and latency cost of high-quality inference.
Peer-reviewed2017
Deep Reinforcement Learning from Human Preferences
Showed a reward model can be trained from humans' pairwise comparisons of agent trajectory segments, letting RL optimize behaviors too hard to hand-specify without ever writing an explicit reward function.
Peer-reviewed2023
Verifiers & Process Supervision (GSM8K / Let's Verify Step by Step)
It compared rewarding only the final answer (outcome supervision) against rewarding each intermediate reasoning step (process supervision) and showed that step-level feedback trains a more reliable verifier that selects correct solutions to hard math problems more often.
System card2024
OpenAI o1 / o3 Reasoning Systems
These were the first widely deployed systems trained to spend additional inference-time computation on an internal chain of reasoning before answering, trading longer per-query compute for markedly higher accuracy on math, science, and coding problems.
Peer-reviewed2022
Flamingo: a Visual Language Model for Few-Shot Learning
Introduced Flamingo, a vision-language model that connects a frozen pretrained image encoder to a frozen pretrained language model via trainable cross-attention layers, letting a single model handle interleaved image-and-text prompts and learn new tasks from a few in-context examples.
Peer-reviewed2022
Latent Diffusion (Stable Diffusion) / Diffusion Transformers (DiT)
Introduced latent diffusion (the basis of Stable Diffusion), which runs the diffusion process in a compressed autoencoder latent space instead of pixel space, cutting the compute cost of training and sampling high-resolution image generators by a large factor.
System card2024
Native/Omni Multimodality (Gemini / GPT-4V / GPT-4o)
Presented Gemini, a family of models trained from the start on interleaved text, images, audio, and video rather than bolting separate modality models together, aiming to remove the boundaries between modality-specific systems in a single model.
Technical report2021
Codex: Evaluating LLMs Trained on Code (HumanEval)
Introduced Codex and the HumanEval benchmark, evaluating a GPT model fine-tuned on public code for generating programs from docstrings and measuring correctness by actually running unit tests, establishing execution-based functional evaluation instead of text-overlap metrics.
Dataset paper2022
LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models
Released an openly available dataset of 5.85 billion image-text pairs, giving the research community the web-scale multimodal data that had previously been locked inside a few large labs and enabling open text-to-image and vision-language models.
Peer-reviewed2024
Mamba / Mamba-2 (Selective State Spaces)
Introduced a selective state-space sequence model whose parameters depend on the input, matching Transformer quality on language while scaling linearly in sequence length and avoiding the quadratic attention bottleneck.
Technical report2020
Zoom In: An Introduction to Circuits
Argued that neural networks can be reverse-engineered into human-understandable circuits — features connected by weights — turning interpretability from vague intuition into a study of concrete mechanisms.
Technical report2021
A Mathematical Framework for Transformer Circuits
Provided a mathematical account of attention-only Transformers as compositions of interpretable operations, making the attention mechanism analyzable rather than opaque.
Technical report2022
In-context Learning and Induction Heads
Identified induction heads — attention heads that complete repeated patterns — and argued they are a primary mechanism behind in-context learning in Transformers.
Technical report2022
Toy Models of Superposition
Showed that neural networks pack more features than they have dimensions by putting them in superposition, explaining why individual neurons are often polysemantic and hard to read.
Technical report2023
Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
Used sparse dictionary learning to decompose a language model’s activations into many monosemantic features, giving a scalable way to extract human-interpretable concepts from superposed representations.
Essay2020
The Scaling Hypothesis
Argued that the capabilities of large language models are largely a smooth function of scale, so continuing to scale compute and data should keep producing gains without new architectural ideas.
Technical report2018
AI and Compute
Documented that the compute used in the largest AI training runs had grown roughly exponentially, quantifying the trend that later scaling laws and frontier models would ride.
Critique2021
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
Argued that ever-larger language models carry underexamined costs — environmental, financial, and social — and that fluency without grounding risks mistaking pattern completion for understanding.
Preprint2021
Zero-Shot Text-to-Image Generation (DALL-E)
Showed that a single autoregressive Transformer over combined text and image tokens could generate coherent images from natural-language captions, demonstrating zero-shot text-to-image generation at scale.
Preprint2022
Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)
Trained a single speech model on a very large, weakly-labeled multilingual corpus, achieving robust zero-shot transcription and translation without dataset-specific fine-tuning.
Technical report2023
Mistral 7B / Mixtral
A compact 7-billion-parameter open-weight model that used grouped-query attention and sliding-window attention to match or beat larger models, showing that architectural efficiency could substitute for raw parameter count.
Technical report2024
Open Ecosystem (Gemma/Phi/OLMo/Falcon/Command R/Yi/GLM/InternLM)
This open ecosystem produced strong sub-frontier base models under varied openness levels, with OLMo going furthest by releasing not just weights but the full training data, code, logs, and checkpoints so the entire training process is reproducible.
Peer-reviewed2021
GLM: General Language Model Pretraining with Autoregressive Blank Infilling / GLM-130B
Introduced the GLM pretraining objective — autoregressive blank infilling — which unifies bidirectional context and left-to-right generation in a single model, and scaled it into GLM-130B, an openly released English–Chinese bilingual foundation model.
Technical report2025
GLM-4.5 to GLM-5.2 (Zhipu AI / Z.ai open-weight MoE series)
Put a frontier line of large mixture-of-experts models into the open-weight ecosystem: GLM-4.5 through the GLM-5.2 flagship, released by Zhipu AI / Z.ai under an MIT license with downloadable weights rather than API-only access.
Technical report2022
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Produced a 176-billion-parameter multilingual language model through an open 1000-researcher collaboration, releasing the weights, training code, and data so a frontier-scale model was no longer the preserve of a single well-resourced lab.
Peer-reviewed2018
Subword Regularization / Unigram LM Tokenization
It introduced probabilistic subword segmentation, sampling multiple tokenizations of the same text during training as regularization, and a unigram language-model tokenizer that produces those alternative segmentations.
Peer-reviewed2024
The Curse of Recursion / Model Collapse
It identified and analyzed model collapse, the degenerative process where training successive generations of models on their predecessors' outputs makes them progressively forget the true data distribution, especially its tails.
Peer-reviewed2023
DoReMi: Optimizing Data Mixtures with Group DRO
DoReMi trains a small proxy model with Group DRO to learn domain weights for a pretraining corpus, so a large model can be trained once on a tuned data mixture instead of tuned by trial and error.
Peer-reviewed2019
PipeDream: Generalized Pipeline Parallelism for DNN Training
PipeDream keeps the pipeline full by injecting a new minibatch at every stage as soon as it is free and using weight stashing to keep forward and backward passes consistent, removing the pipeline-flush bubble of synchronous schemes.
Technical report2023
FP8-LM / FP8 Training
FP8-LM shows large language models can be trained with 8-bit floating point for gradients, optimizer state, and communication by adding per-tensor scaling and precision decoupling, reducing memory and communication versus 16-bit training without loss degradation.
Peer-reviewed2003
A Neural Probabilistic Language Model
Proposed learning a distributed vector representation for each word jointly with a neural network that predicts the next word, addressing the curse of dimensionality that made count-based n-gram language models generalize poorly to unseen word combinations.
Preprint2013
Efficient Estimation of Word Representations (Word2Vec)
Introduced the continuous bag-of-words and skip-gram models (Word2Vec), simple shallow architectures that learn word vectors from raw text far more cheaply than prior neural language models, removing the compute cost that had limited embedding training to small corpora.
Peer-reviewed2014
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Introduced dropout, a regularization method that randomly deactivates a fraction of units on each training pass, addressing the tendency of large neural networks to overfit by co-adapting their units to the training data.
Peer-reviewed2010
Understanding the Difficulty of Training Deep Nets (Xavier/He init)
Analyzed how activation-function choice and weight-initialization scale cause activations and gradients to shrink or explode across layers, and introduced the Xavier/Glorot initialization scheme that keeps signal variance stable, removing a common cause of failed deep-network training.
Peer-reviewed2010
Rectified Linear Units (ReLU)
Established the rectified linear unit (max of zero and the input) as an activation function for deep networks, addressing the vanishing-gradient and slow-training problems caused by saturating sigmoid and tanh units.
Peer-reviewed2015
Batch Normalization
Introduced batch normalization, which standardizes each layer's inputs using the mean and variance of the current mini-batch, addressing the internal covariate shift that forced small learning rates and careful initialization when training deep networks.
Peer-reviewed1998
LeNet: Gradient-Based Learning Applied to Document Recognition
Presented LeNet, a convolutional neural network trained end to end with backpropagation for handwritten digit and document recognition, showing that learned convolutional features could replace hand-engineered feature extractors for image tasks.
Peer-reviewed1994
Learning Long-Term Dependencies is Difficult (vanishing gradients)
Bengio et al. analyzed why recurrent networks fail to learn dependencies across long time gaps, showing that the gradients propagated back through many steps tend to vanish or explode exponentially, which explained a fundamental obstacle to training networks on long sequences.
Peer-reviewed2006
Deep Belief Nets / Reducing Dimensionality
Showed that a deep network can be initialized by greedily pre-training one layer at a time as restricted Boltzmann machines and then fine-tuned as a whole, addressing the difficulty of training deep networks from random weights and enabling autoencoders that learn low-dimensional data codes.
Peer-reviewed1993
The Mathematics of Statistical MT (IBM alignment models)
Brown et al. formalized statistical machine translation as a probabilistic model estimated from bilingual text, introducing the IBM Models 1-5 that learn word-to-word translation probabilities and alignments from parallel corpora, solving how to translate by learning from data rather than hand-written rules.
Peer-reviewed1988
Learning to Predict by the Methods of Temporal Differences
Introduced temporal-difference (TD) learning, a method that lets a system learn to predict long-run outcomes by adjusting each prediction toward the next prediction rather than waiting for the final result.
Peer-reviewed1992
Q-learning
Introduced Q-learning, an off-policy algorithm that learns the value of taking each action in each state and proved it converges to optimal control even while the agent explores with a different policy.
Peer-reviewed2013
Auto-Encoding Variational Bayes (VAE)
Introduced the variational autoencoder (VAE), which trains a neural network to both encode data into a probabilistic latent space and generate new samples from it, made trainable by the reparameterization trick.
Foundational treatise1854
Boolean Algebra (The Laws of Thought)
Boole recast logical reasoning as algebra, showing that propositions and their combinations obey formal symbolic laws that can be manipulated like equations.
Foundational design1843
Analytical Engine & the First Algorithm
Babbage designed a general-purpose mechanical computer controlled by punched cards, and Lovelace's notes on it described how such a machine could be programmed to carry out an arbitrary sequence of operations.
Foundational book1948
Cybernetics: Control and Communication in the Animal and the Machine
Wiener named and unified the study of feedback and control, arguing that goal-directed behavior in machines and living things can be described by the same mathematics of information and regulation.
Critique1969
Perceptrons (Minsky-Papert critique)
Minsky and Papert's book analyzed the mathematical capabilities of single-layer perceptrons, proving they cannot compute functions that are not linearly separable, such as XOR, which clarified the fundamental limits of that model.
Foundational paper1964
Algorithmic Information Theory (Solomonoff / Kolmogorov)
Algorithmic information theory defined the complexity of an object as the length of the shortest program that produces it, giving a formal account of randomness, compression, and simplicity that underlies modern ideas about learning and inductive inference.
Peer-reviewed2021
Train Short Test Long: Attention with Linear Biases (ALiBi)
ALiBi replaces learned or sinusoidal position embeddings with a fixed linear penalty added to attention scores based on key-query distance, letting a model trained on short sequences run on much longer ones without retraining.
Peer-reviewed2018
Self-Attention with Relative Position Representations / Transformer-XL
Transformer-XL adds a recurrence mechanism that reuses hidden states from prior segments plus a relative positional encoding, removing the fixed-length context limit and the context fragmentation caused by processing text in independent chunks.
Preprint2019
Fast Transformer Decoding: One Write-Head is All You Need (MQA)
Introduced multi-query attention, which shares a single key and value head across all query heads so autoregressive decoding needs far less memory bandwidth for the attention cache.
Preprint2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach
It showed that BERT was substantially undertrained and that with more data, longer training, larger batches, removal of the next-sentence-prediction objective, and dynamic masking, the same architecture reaches much higher accuracy, establishing that pretraining recipe choices, not architecture, drove most of the remaining gains.
Peer-reviewed2020
ELECTRA: Pre-training Text Encoders as Discriminators
ELECTRA trains an encoder with replaced-token detection, a discriminator that classifies every token as original or generator-substituted, extracting learning signal from all positions instead of only the ~15% masked ones and cutting pretraining compute.
Peer-reviewed2020
BART: Denoising Seq2Seq Pre-training
BART pretrains a full encoder-decoder Transformer as a denoising autoencoder that reconstructs original text from arbitrarily corrupted input, unifying BERT-style bidirectional encoding with GPT-style autoregressive generation for both understanding and generation tasks.
Peer-reviewed2021
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
DeBERTa introduces disentangled attention, representing each token by separate content and relative-position vectors whose attention is computed across both, plus an enhanced mask decoder that reinjects absolute positions, improving over BERT and RoBERTa at equal or smaller scale.
Peer-reviewed2022
Emergent Abilities of Large Language Models
This paper documents that certain capabilities are absent in smaller language models yet appear once scale crosses a threshold, arguing such abilities are not predictable by extrapolating small-model performance and framing scale itself as unlocking qualitatively new behavior.
Peer-reviewed2021
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
It introduced a sparsely-gated mixture-of-experts layer combined with automatic sharding annotations (the GShard/XLA compiler system) so that models with hundreds of billions of parameters could be split across thousands of accelerators, removing the memory and engineering bottleneck that blocked training beyond a few billion parameters.
Peer-reviewed2022
GLaM: Efficient Scaling with Mixture-of-Experts
It applied a sparsely-gated mixture-of-experts to decoder language models so that each token activates only two experts, letting a 1.2T-parameter model match or beat a dense GPT-3 while using roughly a third of the training energy and half the inference FLOPs.
Technical report2024
DeepSeekMoE: Towards Ultimate Expert Specialization
It restructured MoE by using many finer-grained experts plus a few always-on shared experts, so routed experts can specialize on distinct knowledge while shared ones absorb common patterns, improving quality at equal activated parameters.
Peer-reviewed2020
Learning to Summarize from Human Feedback
It trained a summarization model against a reward model learned from human comparisons of candidate summaries, showing that optimizing for human preferences with reinforcement learning produces better summaries than optimizing for likelihood or standard automatic metrics like ROUGE.
Peer-reviewed2022
Reward Model Overoptimization / Sycophancy
It measured how optimizing a policy against a learned reward model eventually degrades true reward, quantifying reward-model overoptimization as a Goodhart-law failure of RLHF.
Peer-reviewed2023
Self-Consistency Improves Chain-of-Thought Reasoning
Improved chain-of-thought accuracy by sampling many reasoning paths and taking the majority-vote answer instead of trusting a single greedy decode, removing the fragility of one-shot reasoning chains.
Peer-reviewed2023
Structured Reasoning: Least-to-Most / PoT / Tree of Thoughts
This family reorganized chain-of-thought into explicit search or decomposition structures, letting a model explore, evaluate, and backtrack over intermediate steps instead of committing to one linear reasoning path.
Peer-reviewed2023
Toolformer: LMs Can Teach Themselves to Use Tools
Trains a model to decide when to call external APIs and with what arguments using a self-supervised objective, removing the need for human annotation of tool-use demonstrations.
Product release2023
Function Calling / MRKL / Model Context Protocol
This family defined structured interfaces for letting language models call external tools, routing between a reasoning core and specialized modules or APIs instead of relying on the model's parametric knowledge alone.
Peer-reviewed2023
Reflexion: Language Agents with Verbal Reinforcement Learning
Reflexion added a verbal self-reflection loop in which an agent turns feedback from a failed attempt into written critiques stored in memory and used to improve the next attempt, without updating model weights.
Preprint2023
Voyager / Generative Agents
These works built open-ended agents that set their own goals and accumulate reusable skills, and populated simulated worlds with believable character agents driven by memory and reflection.
Peer-reviewed2024
SWE-agent / OpenHands (autonomous software engineering)
They packaged a language model as an autonomous agent with a structured interface to a code repository, shell, and editor, showing that a model given the right tools and feedback loop can resolve real GitHub issues end to end rather than only generating isolated code snippets.
Peer-reviewed2021
ALIGN: Scaling Up Visual and Vision-Language Representation Learning
ALIGN showed that a dual-encoder image-text model trained with contrastive learning on a billion-scale noisy web alt-text dataset matches curated-data methods, removing the dependency on expensive human-labeled or filtered training pairs.
Peer-reviewed2023
BLIP / BLIP-2 (Q-Former)
BLIP-2 introduced a lightweight Querying Transformer (Q-Former) that connects a frozen image encoder to a frozen large language model, removing the need to train either backbone to build a vision-language model.
Peer-reviewed2023
LLaVA: Visual Instruction Tuning
Introduced LLaVA, which connects a CLIP vision encoder to an open language model through a simple projection layer and fine-tunes on GPT-generated visual instruction data, providing a low-cost recipe for building conversational vision-language assistants.
Technical report2023
Open Production VLMs (Qwen-VL/InternVL/PaLI/Kosmos)
A family of open production vision-language models (represented by Qwen-VL, with InternVL, PaLI, and Kosmos) packaged image encoders plus LLMs into openly released, instruction-tuned multimodal assistants, removing the reliance on closed APIs for capable general-purpose VLMs.
Peer-reviewed2023
SAM / Grounding DINO / ImageBind (perception foundation models)
A set of perception foundation models (represented by Segment Anything, with Grounding DINO and ImageBind) produced promptable, general-purpose perception backbones that generalize to new objects and tasks without per-task retraining.
Peer-reviewed2020
Dense Passage Retrieval (DPR)
Introduced Dense Passage Retrieval, showing that a BERT-based dual-encoder trained with contrastive learning retrieves relevant passages more accurately than keyword search like BM25, removing sparse lexical matching as the default first stage for open-domain question answering.
Peer-reviewed2020
REALM: Retrieval-Augmented LM Pre-Training
REALM introduced end-to-end pretraining of a language model jointly with a neural retriever over a text corpus, removing the need to store all world knowledge implicitly in model parameters.
Peer-reviewed2021
FiD / RETRO / Atlas (scaling retrieval)
A line of retrieval-scaling methods (represented by RETRO, with Fusion-in-Decoder and Atlas) showed that conditioning generation on large retrieved text databases lets smaller models match much larger ones, removing the need to scale parameters to hold more knowledge.
Peer-reviewed2020
ColBERT / Late Interaction / Rerankers / kNN-LM
ColBERT introduced late interaction, encoding queries and documents into token-level embeddings and scoring them with a cheap MaxSim operation, removing the trade-off between the accuracy of full cross-encoders and the speed of single-vector retrieval.
Peer-reviewed2019
Contrastive Text Embeddings (Sentence-BERT/SimCSE/E5/BGE)
Introduced Sentence-BERT, which fine-tunes BERT in a siamese setup to produce sentence embeddings comparable with cosine similarity, removing the need to run BERT on every sentence pair and making large-scale semantic similarity and search practical.
Peer-reviewed2022
AlphaCode (Competition-Level Code Generation)
AlphaCode showed that a transformer trained on public code plus massive sampling and test-based filtering could solve competitive programming problems at roughly median-human level, turning code generation from single-shot guessing into a search-and-filter pipeline.
Peer-reviewed2022
CodeT5 / CodeGen / InCoder / Fill-in-the-Middle
This family established the pretraining objectives specific to code, culminating in Fill-in-the-Middle, which showed that reordering training documents lets a left-to-right model infill arbitrary spans given both left and right context at no loss to ordinary generation.
Technical report2023
Open Code Models (StarCoder/Code Llama/DeepSeek-Coder/Qwen-Coder)
This family delivered openly released, permissively licensed code LLMs trained on large curated code corpora, with StarCoder pairing a 15B model and an 8K context with documented, opt-out, license-filtered training data to make capable code models usable and inspectable outside the big labs.
Peer-reviewed2020
Sparse Attention (Longformer / BigBird)
Sparse-attention transformers cut the quadratic cost of full attention to linear in sequence length by attending over a fixed sparse pattern, with BigBird combining local, global, and random attention while proving the pattern still preserves the transformer's expressive and theoretical properties.
Peer-reviewed2024
Ring Attention / Infini-Attention / MLA
This family scaled context length by distributing or compressing the attention computation rather than approximating it, with Ring Attention sharding the sequence across devices and passing key-value blocks in a ring so exact attention over near-arbitrarily long contexts fits in aggregate memory.
Peer-reviewed2023
Context Extension (Position Interpolation / NTK / YaRN)
A family of RoPE-rescaling methods (Position Interpolation, NTK-aware scaling, YaRN) that extend a pretrained transformer's usable context window with little or no retraining by remapping rotary position frequencies instead of pretraining longer.
Peer-reviewed2022
S4: Structured State Space Sequence Models
Introduced S4, a structured state space sequence model whose special (diagonal-plus-low-rank) parameterization made state space models trainable and efficient, giving a sub-quadratic alternative to attention that handles very long sequences and long-range dependencies.
Peer-reviewed2023
RWKV / Hyena / Attention-SSM Hybrids (Jamba)
A family of sub-quadratic sequence models (RWKV, Hyena, and attention-SSM hybrids like Jamba) that replace or interleave softmax attention with recurrent or convolutional/state-space mixing to remove attention's quadratic-cost and growing-KV-cache bottleneck at long sequence lengths.
Benchmark2021
MMLU: Massive Multitask Language Understanding
A 57-subject multiple-choice benchmark spanning STEM, humanities, social sciences, and professional exams that measures broad pretrained knowledge in zero- and few-shot settings, removing reliance on narrow single-task metrics for judging general capability.
Benchmark2023
GPQA: Graduate-Level Google-Proof QA
GPQA, a benchmark of several hundred graduate-level science multiple-choice questions written and validated by domain experts to be hard even for skilled non-experts with web access, giving a contamination-resistant measure of expert-level reasoning rather than lookup.
Benchmark2021
GSM8K / MATH (mathematical reasoning)
Released the MATH dataset of competition mathematics problems with full step-by-step solutions, providing a hard benchmark that exposed how weak language models were at multi-step mathematical reasoning.
Benchmark2022
BIG-bench / BBH / HELM (holistic eval)
A family of broad-coverage evaluation efforts (BIG-bench, BIG-bench Hard, and HELM) that shifted LLM assessment from single-task accuracy to standardized measurement across many tasks and multiple metrics, exposing where scaling helps, where it doesn't, and trade-offs beyond accuracy.
Benchmark2021
HumanEval / MBPP (function-level code)
Function-level program-synthesis benchmarks (HumanEval and MBPP) that score generated Python by executing it against hidden unit tests, removing the mismatch between surface-similarity text metrics and whether code actually runs correctly.
Benchmark2023
Agent/Web Benchmarks (AgentBench/WebArena/GAIA/tau-bench)
A family of interactive agent benchmarks (WebArena, AgentBench, GAIA, tau-bench) that evaluate models on multi-step tasks in realistic executable environments (live websites, tools, APIs, simulated customer service) rather than static prompts, measuring whether an agent actually completes a goal.
Benchmark2023
Chatbot Arena / LLM-as-Judge (MT-Bench/AlpacaEval)
Introduced Chatbot Arena and MT-Bench, establishing crowd-sourced pairwise human voting and a strong-model-as-judge protocol as scalable ways to rank chat models on open-ended quality where fixed benchmarks fall short.
Benchmark2021
Contamination-Resistant Benchmarks (TruthfulQA/LiveBench/SimpleQA)
A family of contamination-resistant benchmarks (TruthfulQA, LiveBench, SWE-bench-verified) designed so scores are not inflated by training-set memorization or leaked test data, using adversarial questions, continuously refreshed recent items, or human-verified solvable tasks.
Peer-reviewed2023
The Evaluation Crisis (contamination/saturation/judge-bias)
Analyzed the reliability problems undermining language model evaluation, showing how benchmark contamination, saturation, and biased automated judges inflate reported scores and distort model comparisons.
Vision essay2024
Situational Awareness: The Decade Ahead
Extrapolated compute, algorithmic-efficiency, and investment trendlines to argue that broadly human-level AI could arrive within a few years, and drew out the national-security implications.
Preprint2021
WebGPT: Browser-assisted Question-answering with Human Feedback
Fine-tuned a language model to answer questions by browsing the web and citing sources, using human feedback to reward well-supported answers and reducing unsupported claims.
Preprint2022
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan)
Grounded a language model’s plans in what a robot can actually do by combining the model’s task knowledge with learned affordance values, letting an LLM direct real-world robotic behavior.
Preprint2022
Solving Quantitative Reasoning Problems with Language Models (Minerva)
Showed that a large model trained on mathematical and scientific text, with chain-of-thought and majority voting, could solve quantitative reasoning problems far better than prior models without external tools.
Preprint2022
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen)
Showed that conditioning an image-diffusion model on a large frozen text encoder yields high-fidelity, well-aligned text-to-image generation, highlighting the text encoder as the key lever.
Technical report2024
Video Generation Models as World Simulators (Sora)
Framed video generation as training a diffusion Transformer on patches of video and image data at scale, producing minute-long coherent videos and evidence of emergent world-simulation behavior.
Peer-reviewed2017
Overcoming Catastrophic Forgetting in Neural Networks
Introduced elastic weight consolidation, which slows learning on the parameters most important to previously learned tasks, letting one network learn new tasks without catastrophically forgetting the old ones.
Preprint2025
Titans: Learning to Memorize at Test Time
Introduced a neural long-term memory module that learns what to memorize at test time, letting a model write new information into parameters during inference rather than staying frozen after training.
Peer-reviewed2014
GloVe: Global Vectors for Word Representation
GloVe introduced a word-embedding method that learns vectors by factorizing a corpus-wide word co-occurrence matrix, solving the problem of combining global corpus statistics with the useful linear structure of local-context methods.
Peer-reviewed2015
Effective Approaches to Attention-based NMT (Luong attention)
Luong et al. proposed simpler and more general attention mechanisms for neural machine translation, including global and local variants and a cheaper dot-product scoring function, solving the cost and rigidity of the earlier Bahdanau attention design.
Peer-reviewed1989
Universal Approximation Theorem
The Universal Approximation Theorem proved that a feedforward network with a single hidden layer and a suitable nonlinear activation can approximate any continuous function on a bounded domain to arbitrary accuracy, establishing that neural networks are in principle expressive enough to represent essentially any target mapping.
Peer-reviewed1990
Finding Structure in Time (Elman RNN)
Elman introduced the simple recurrent network, which feeds a hidden layer's previous activations back as additional input through a context layer, giving neural networks a working memory of past inputs and solving how to process sequences whose structure unfolds over time.
Peer-reviewed1997
Bidirectional Recurrent Neural Networks
Schuster and Paliwal introduced the bidirectional recurrent network, which runs two recurrent networks over a sequence in opposite directions and combines their states, so each position's representation can use both past and future context, solving the limitation that a forward-only network cannot see later inputs.
Peer-reviewed2014
Gated Recurrent Unit (GRU) / RNN Encoder-Decoder
Cho et al. introduced the RNN encoder-decoder framework for translating variable-length sequences and the Gated Recurrent Unit, a simplified gated cell, solving how to map one sequence to another while mitigating the vanishing-gradient problem with fewer parameters than an LSTM.
Peer-reviewed2006
Connectionist Temporal Classification (CTC)
Graves et al. introduced Connectionist Temporal Classification, a loss function that lets a recurrent network be trained to map an input sequence to a shorter output label sequence without knowing the alignment between them, solving the need for pre-segmented, frame-by-frame labeled training data in tasks like speech recognition.
Peer-reviewed2011
Natural Language Processing (almost) from Scratch
Collobert et al. showed that a single convolutional neural network trained on large unlabeled text could handle multiple NLP tagging tasks well using learned word features instead of hand-engineered ones, arguing for a unified, feature-engineering-free approach to language processing.
Peer-reviewed1995
TD-Gammon
Tesauro's TD-Gammon showed that a neural network trained by temporal-difference reinforcement learning, largely through self-play, could reach expert-level backgammon, demonstrating that a value function for a complex game could be learned from experience rather than programmed.
Preprint2014
Neural Turing Machines / Memory Networks
Graves et al. introduced the Neural Turing Machine, a neural network coupled to an external memory matrix it can read from and write to using differentiable attention, so the whole system can be trained by gradient descent to learn simple algorithms, addressing standard networks' limited ability to store and manipulate structured data over time.
Peer-reviewed2012
Deep Neural Networks for Acoustic Modeling in Speech
This overview paper reported that deep neural networks trained as acoustic models consistently outperformed the long-dominant Gaussian mixture models in speech recognition across four research groups, marking a practical shift to deep learning for acoustic modeling.
Peer-reviewed2019
ALBERT: A Lite BERT
ALBERT cuts BERT's parameter count through factorized embedding parameterization and cross-layer parameter sharing, and swaps next-sentence prediction for a sentence-order objective, letting larger-capacity configurations train within the same memory budget.
Peer-reviewed2023
UL2: Unifying Language Learning Paradigms
UL2 introduces Mixture-of-Denoisers, a pretraining scheme that blends several span-corruption and prefix-language-modeling objectives with mode-switching tokens, producing a single model that works across both fine-tuning and in-context few-shot settings regardless of architecture.
Peer-reviewed2023
Are Emergent Abilities of LLMs a Mirage?
This paper argues apparent emergent abilities are largely an artifact of discontinuous or nonlinear evaluation metrics rather than fundamental jumps in model capability, showing that switching to smooth, continuous metrics turns the sharp jumps into gradual, predictable improvement.
Peer-reviewed2023
Scaling Data-Constrained Language Models
It measured how repeating a fixed corpus trades off against adding parameters, showing that under a data budget you can reuse tokens for several epochs at near-fresh-data value and derive scaling laws that account for repetition.
Peer-reviewed2022
Mixture-of-Experts with Expert Choice Routing
It replaced token-chooses-expert routing with expert-chooses-token routing, letting each expert select a fixed number of tokens so load is balanced by construction and no auxiliary balancing loss or token dropping is needed.
Preprint2022
ST-MoE: Designing Stable and Transferable Sparse Expert Models
It diagnosed why sparse expert models are unstable and transfer poorly, and introduced the router z-loss plus fine-tuning fixes that let a 269B-parameter sparse model train reliably and reach then-state-of-the-art results on SuperGLUE and reasoning benchmarks.
Peer-reviewed2024
Preference Optimization Family (IPO/KTO/ORPO/SimPO)
This family of methods (IPO, KTO, ORPO, SimPO) reworks Direct Preference Optimization to fix its failure modes and drop its dependencies, aligning language models from preference data with simpler or more robust objectives and often no separate reward or reference model.
Preprint2018
Scalable Oversight: Debate / Weak-to-Strong Generalization
This line of work asked how humans or weaker models can supervise systems more capable than themselves, and showed that fine-tuning a strong model on a weak model's labels can recover much of the strong model's latent capability.
Peer-reviewed2023
Reasoning Correctives: Faithfulness / Overthinking / Contamination
These studies showed that a model's stated chain-of-thought often does not reflect the actual cause of its answer, documenting unfaithful, biased, and over-length reasoning traces.
Preprint2023
Agent Memory & Multi-Agent Systems (MemGPT/AutoGen/CAMEL)
This family gave agents persistent, paged memory and frameworks for structured multi-agent collaboration, letting systems exceed the fixed context window and divide work across role-specialized agents.
Peer-reviewed2022
Knowledge Editing (ROME / MEMIT)
Knowledge editing methods (ROME and MEMIT) located where specific facts are stored in a transformer's feed-forward layers and edited them by a direct weight update, removing the need to retrain or fine-tune to change or add individual facts.
Technical report2024
Graph & Structured Retrieval (GraphRAG)
GraphRAG builds a knowledge graph and hierarchical community summaries from a corpus so retrieval can operate over structured, aggregated entities, removing standard RAG's inability to answer global questions that span an entire dataset.
Peer-reviewed2022
Execution Feedback / Unit-Test Gen / Program Repair
This line closed the loop between generation and correctness by using program execution and unit tests as a feedback signal, so models could generate tests, run candidate code, read the errors, and repair themselves instead of relying on the first output.
Peer-reviewed2020
Linearized Attention (Reformer/Performer/Linear Transformers)
Linearized-attention models replaced the softmax attention matrix with kernel feature maps or hashing so attention could be computed in linear time and memory, with Performer using random features (FAVOR+) to approximate softmax attention without ever forming the full N-by-N matrix.
Peer-reviewed2023
Lost in the Middle / Long-Context Reliability (RULER)
An empirical study showing that long-context models retrieve information reliably only when it sits near the start or end of the input and degrade sharply for content in the middle, exposing that a large context window does not guarantee uniform use of it (with RULER later giving a systematic length-vs-reliability probe).
Software release2023
AutoGPT / BabyAGI (viral agent software)
AutoGPT and BabyAGI were widely-copied open-source programs that wrapped a language model in an autonomous loop of goal decomposition, task queuing, and self-prompted execution toward a user objective.