PreprintReviewed
Efficient Estimation of Word Representations (Word2Vec)
Introduced the continuous bag-of-words and skip-gram models (Word2Vec), simple shallow architectures that learn word vectors from raw text far more cheaply than prior neural language models, removing the compute cost that had limited embedding training to small corpora.
Executive summary
The models drop the hidden nonlinear layers of earlier neural language models and instead train a shallow network to predict a word from its surrounding context (CBOW) or the surrounding context from a word (skip-gram). This makes training fast enough to run on billions of words, producing vectors where semantic and syntactic relationships appear as consistent offsets (the vector arithmetic 'king minus man plus woman' lands near 'queen'). It made high-quality word embeddings cheap to compute and widely reusable as input features for other NLP systems.
Antecedents
Challenged, corrected, or evaluated by
- enables Natural Language Processing from Scratch — Collobert-Weston precedes word2vecStrongly supported
What it enabled — descendants
- depends on Attention Is All You Need — Learned token embeddings feed the inputDirect evidence