Perceptron and Multilayer Networks

An artificial neuron multiplies the input values x1, x2, ..., xn by the corresponding weights w1, w2, ..., wn, adds them up, adds a bias b and passes the result through an activation function f: y = f(w1 * x1 + ... + wn * xn + b). In the perceptron proposed by Frank Rosenblatt in 1958, f is a step function: if the sum is greater than zero the output is 1, otherwise 0. The perceptron splits the input space in two with a hyperplane, so it can separate only linearly separable classes.

The perceptron learning rule is simple: if the prediction on an example is wrong, the weights are corrected with w = w + lr * (y - yhat) * x and the bias with b = b + lr * (y - yhat). On linearly separable data (for example, the logical AND and OR functions) the algorithm finds a solution in a finite number of steps. The classes of the XOR function, however, cannot be separated by a single straight line; Marvin Minsky and Seymour Papert proved this limitation in 1969, which temporarily reduced interest in neural networks.

A multilayer perceptron (MLP) consists of an input layer, one or more hidden layers and an output layer; each neuron of a layer is connected to all neurons of the next one. Nonlinear activation functions in the hidden layers (sigmoid, tanh, ReLU = max(0, z)) let the network learn curved boundaries: a network with two hidden neurons can solve XOR. Without nonlinearity, any number of layers would be equivalent to a single linear transformation. By the universal approximation theorem, even one hidden layer with enough neurons can approximate any continuous function to any accuracy, although in practice deeper networks are more efficient.

A multilayer network is trained with the backpropagation algorithm: in the forward pass the network computes the prediction and the error, and in the backward pass the gradient of the error with respect to every weight is found using the chain rule. The weights are then updated with gradient descent or one of its improved variants (for example, Adam). This process is repeated over many epochs, and the error on a validation set is monitored to notice overfitting.

Key takeaways

A neuron takes a weighted sum of its inputs, adds a bias and passes the result through an activation function.
The perceptron separates only linearly separable classes; XOR is beyond it.
Hidden layers and nonlinear activation (ReLU, tanh, sigmoid) extend the power of the network.
Backpropagation computes the gradient for all weights using the chain rule.
Network size, learning rate and the number of epochs affect the risk of overfitting.

Practice tasks

Check by hand: for a two-input perceptron with w1 = 1, w2 = 1 and b = -1.5, what is the output for the four input combinations and which logical function does it compute?
Write a perceptron in NumPy: when it errs, update w and b with the formulas above; train it on the AND and OR functions, then try it on XOR and explain why it cannot learn it.
In scikit-learn train XOR with MLPClassifier(hidden_layer_sizes=(4,), activation="tanh", solver="lbfgs", max_iter=2000) (if the accuracy is not 1.0, change random_state), then reduce the number of hidden neurons to 1 and observe how the result changes.

Review questions

Test yourself: think first, then reveal the answer.

1Why can a single perceptron not learn the XOR function?Show answer

The two classes of XOR cannot be separated by one straight line (they are not linearly separable), while a perceptron forms only a linear boundary.

2What happens in a multilayer network if all activation functions are linear?Show answer

A composition of linear transformations is again linear, so the network becomes equivalent to a single layer.

3What does backpropagation compute?Show answer

Using the chain rule, the gradient of the loss function with respect to every weight and bias; this gradient is then used to update the weights.