Foundational paperReviewed
Backpropagation (Learning representations by back-propagating errors)
Introduced the backpropagation algorithm for training multi-layer neural networks by propagating output error backward through the network to compute weight gradients, removing the barrier that networks with hidden layers had no practical learning rule.
Executive summary
The paper applies the chain rule to compute, for every weight in a layered network, how much it contributed to the output error, then adjusts weights in the direction that reduces that error. This gave hidden units a way to learn useful internal representations instead of being hand-designed. It made training networks with one or more hidden layers a routine gradient-descent procedure and became the standard method for supervised neural network learning.
Antecedents
Challenged, corrected, or evaluated by
- enables A Logical Calculus of Ideas Immanent in Nervous Activity — Artificial neuron ancestor of backprop netsStrongly supported
- enables The Perceptron — Perceptron learning ancestor of backpropStrongly supported
- challenges Perceptrons — Backprop answers the multilayer critiqueStrongly supported
What it enabled — descendants
- depends on Distilling the Knowledge in a Neural Network — Trains a student against a teacher soft output distribution via backpropagationStrongly supported
- depends on Overcoming Catastrophic Forgetting in Neural Networks — Regularizes weight updates in a backprop-trained networkStrongly supported