Neural networks

From a single perceptron to a trained multilayer network, with backpropagation derived in full and then computed by hand and by code until the two agree.

This part builds on Foundations and Machine learning.

7 of 7 topics ready, listed in reading order

Perceptrons and multilayer networksLogic gates, why XOR needs a hidden layer, and what hidden units compute.ReadyBackpropagationThe four backpropagation equations derived, a mini-batch step by hand and a gradient check.ReadyActivations and initializationSigmoid, tanh and ReLU, vanishing and exploding gradients, Xavier and He initialization.ReadyLoss functionsSquared error versus cross-entropy, the learning slowdown and softmax with log-likelihood.ReadyRegularizationL1 and L2 penalties, dropout done right, early stopping and data augmentation.ReadyOptimizersSGD, momentum, RMSProp, Adam and learning-rate schedules.ReadyMNIST from scratchA complete digit classifier in NumPy, then the same model in PyTorch.Ready