Peer-reviewedReviewed
A Neural Probabilistic Language Model
Proposed learning a distributed vector representation for each word jointly with a neural network that predicts the next word, addressing the curse of dimensionality that made count-based n-gram language models generalize poorly to unseen word combinations.
Executive summary
Instead of treating words as discrete symbols, the model maps each word to a real-valued vector and feeds the vectors of the preceding words into a neural network that outputs a probability distribution over the next word. Words used in similar contexts end up with similar vectors, so the model can assign reasonable probability to word sequences it never saw in training. This showed that a single network could learn word features and the probability function together, and set the template later used by neural word embeddings and language models.