A4.3.8 · Machine learning approaches
Artificial neural networks
By the end you can
Outline how neurons, weights, layers and activation functions model complex patterns, and how forward and back propagation train a network.
The brain contains billions of nerve cells that pass signals to each other. An (ANN) is a very loose imitation: a large number of simple calculating units, connected in layers, that learn by adjusting the strength of their connections. It is the idea behind almost every impressive recent result in image, speech and language software.
One neuron#
A (or node) does three things:
- Multiplies each input by its , which says how much that input matters.
- Adds the results together with a , a number that shifts how easily the neuron fires.
- Passes that sum through an to give its output.
| Activation function | Output | Notes |
|---|---|---|
| Step | 0 if the sum is below 0, otherwise 1 | The simplest; used in the perceptron. |
| Smooth S-curve from 0 to 1 | Handy for outputs that mean a probability or yes/no. | |
| Tanh | Like sigmoid but from −1 to 1 | Centred on 0. |
| max(0, sum): 0 for negative sums, the sum itself otherwise | Cheap to compute and the usual choice for hidden layers. |
sum = (1.0 × 0.8) + (0.5 × -0.4) + 0.1 = 0.70
output = sigmoid(0.70) = 0.668
The step function gives only 0 or 1. Sigmoid squeezes any sum into the range 0 to 1 smoothly. ReLU passes positive sums through and turns negative ones into 0.
Layers: from one neuron to a network#
Neurons are arranged in layers. The input layer has one neuron for each feature (or each pixel). One or more sit in between, each neuron taking the outputs of the layer before. The output layer gives the answer: one neuron for a number (regression), or one for each class (classification). A network with many hidden layers is a deep network, which is where deep learning gets its name.
Layer by layer, a network builds up its own features. In an image network, early layers respond to edges, middle layers to shapes, and late layers to whole objects. Nobody programs this: it emerges as the weights are trained. Note that a bigger network has more weights to fit, so it needs more data and is more prone to overfitting.
Forward propagation: a worked example#
means calculating the network's output from its inputs. Take a network with 2 inputs, a hidden layer of 2 ReLU neurons and one sigmoid output neuron. The inputs are 2 and 1.
- Hidden neuron 1 (weights 0.5 and −1, bias 0.5): (2 × 0.5) + (1 × −1) + 0.5 = 0.5, and ReLU(0.5) = 0.5.
- Hidden neuron 2 (weights 1 and 1, bias −1): (2 × 1) + (1 × 1) − 1 = 2, and ReLU(2) = 2.
- Output neuron (weights 1 and 0.5, bias −0.5): (0.5 × 1) + (2 × 0.5) − 0.5 = 1, and sigmoid(1) = 0.731.
The network's answer is 0.731: if this were a yes/no question it would lean towards yes. Every number is just multiplication, addition and an activation, repeated many times.
How a network learns#
The weights start random, so the first predictions are poor. Training repeats a cycle on the training data:
- Forward propagation to get a prediction.
- A compares the prediction with the correct answer and measures the error.
- passes the error backwards through the layers, working out how much each weight contributed to it.
- Each weight and bias is nudged a small step in the direction that reduces the error (gradient descent, as in A4.3.1), the step size set by the learning rate.
One full pass over the training data is an . Training takes many epochs, and the loss on separate validation data is watched, because past a certain point a network starts to overfit.
Outputs, loss and a few practical points#
- Regression (predicting a number): the output layer usually has one neuron with no squashing, and the loss is typically the mean squared error.
- Classification (predicting a category): the output layer has one neuron for each class. A softmax function turns their scores into probabilities that add up to 1, and a cross-entropy loss measures how far those probabilities are from the correct class.
- Why ReLU is common in hidden layers. Sigmoid and tanh flatten out for large inputs, so their gradients become tiny and learning stalls in deep networks (the vanishing gradient problem). ReLU keeps a steady gradient for positive sums and is cheap to calculate.
- Batches. Large data sets are usually fed in small batches, updating the weights after each batch, which is faster and steadier than one huge update.
The perceptron, and why layers matter#
The simplest network is a single neuron with a step activation, the . After each example it nudges every weight by learning rate × error × input. Try it below: it learns AND and OR within a few passes, because a straight line can separate their 1s from their 0s. But it can never learn XOR.
Weight 1
0
Weight 2
0
Bias
0
Circles are target 0, squares are target 1. A shape is hollow while the perceptron gets that input wrong. The line is where the weighted sum equals 0.
Weights and bias start at 0. Press “Show one example” to see the rule at work.
Epochs: 0.
Strengths and weaknesses#
- Strength: can learn very complex patterns from raw data, and often beats other methods on images, speech and language.
- Weakness: needs large amounts of data and computing power (A4.1.2).
- Weakness: hard to explain. The knowledge is spread across millions of weights, which is a problem when a decision must be justified (A4.4.1).
- Weakness: can overfit, and can pick up biases in its data.
Check yourself
Artificial neural networks
Quick questions, made fresh every time and marked here. Do a few until they feel easy.
Exam practice
Paper 1-style questions
6 questions · 22 marks
Write your answer as you would in the exam, then show the mark points and tick each one your answer made. These are our own questions in the style of the exam, not past-paper questions.
1.Outline the structure of an artificial neural network.
[4]Outline: A brief account: the main points, without much detail.
2.A neuron has inputs 3 and −2, weights 2 and 1.5, and a bias of 1. It uses the ReLU activation function. Calculate its output.
[3]Calculate: Work out the number, showing the working.
3.Explain the purpose of an activation function in a neural network.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
4.Explain why a single perceptron cannot learn the XOR function, but a network with a hidden layer can.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
5.Distinguish between forward propagation and back propagation.
[3]Distinguish: Make the differences between two things clear.
6.Explain how a neural network is trained to classify handwritten digits from a large set of labelled images.
[6]Explain: Give reasons: say what happens AND why, or how one thing causes another.
In a nutshell
- An ANN is layers of neurons: an input layer, hidden layers and an output layer. Each connection has a weight. “Deep” means many hidden layers.
- A neuron computes sum = Σ(input × weight) + bias, then applies an activation function: step (0 or 1), sigmoid (0 to 1), tanh (−1 to 1) or ReLU (max(0, sum)).
- Activation functions add non-linearity. Without them a deep network would be no more powerful than one linear calculation.
- A perceptron is one neuron with a step output. It can learn AND and OR but not XOR; hidden layers can.
- Training: forward propagation, loss, back propagation of the error, then a gradient-descent update of the weights; repeat over many epochs. Needs lots of data and computing power and is hard to explain.