A4.3.1 · Machine learning approaches
Linear regression
By the end you can
Explain how a straight line fitted to data predicts a number, how its fit is measured (error, R²), and how gradient descent finds the line.
How many marks will you get if you revise for six hours? How much will a 90 m² flat cost? These are questions with a number as the answer, not a category. Supervised learning that predicts a number is called , and the simplest kind is : draw the best straight line through the data and read predictions off it.
With one input, the line is y = mx + b. The input x is the , the predicted y is the , m is the gradient (how much y rises for each extra unit of x) and b is the intercept (the value of y when x is 0). With several inputs the idea is the same, with one gradient for each.
What makes a line the best one?#
No line passes through every point, so each point has an error: its (actual − predicted). Positive and negative errors would cancel if you just added them, so we square each one and take the average. This is the (MSE). Squaring also punishes big misses more than small ones. The best-fit line is the one with the smallest MSE.
Your line
y = 2.00x + 40.0
Mean squared error
155.0
best possible 2.14
R²
0.200
share of variation explained
Gradient descent steps
0
Each dashed red gap is an error. The mean squared error is the average of the squares of those gaps. Gradient descent nudges m and b downhill on that error. Try a learning rate near the top of the slider and take 50 steps: if the steps are too big the error grows instead of shrinking.
For these ten students the best line is y = 5.58x + 31.0: each extra hour of revision goes with about 5.6 more marks. Its mean squared error is 2.14, and R² is 0.989. The line predicts a score of 64.5 for 6 hours.
is the share of the variation in test scores that the line explains. An R² of 0.99 means about 99% of the differences between students' scores go with the differences in their revision time, and the rest is down to other things.
How the computer finds the line#
For one input there is a formula that gives the best line directly. For models with many inputs, and for the neural networks later in this chapter, computers use . Picture the error as a landscape, with the values of m and b as the position. At the current position, work out which way is downhill and take a small step that way. Repeat until you reach the bottom.
The size of the step is the . Too small and learning is slow. Too large and each step overshoots the valley, so the error bounces higher instead of lower. Try it in Figure 1: take 50 steps at a rate of 0.005, then reset and try 0.038: the error grows each step instead of shrinking.
Where linear regression struggles#
- Curved relationships. A straight line through a curve is always wrong somewhere. Other models, or extra features such as x², can capture curves.
- Outliers can drag the line towards them, because squaring makes a big error count heavily.
- Extrapolation. The line says a student who revises 40 hours a day scores far above 100. The pattern was only learned for the range of the data.
- Correlation is not cause. A good fit says x and y move together, not that x makes y happen.
Check yourself
Linear regression
Quick questions, made fresh every time and marked here. Do a few until they feel easy.
Exam practice
Paper 1-style questions
5 questions · 16 marks
Write your answer as you would in the exam, then show the mark points and tick each one your answer made. These are our own questions in the style of the exam, not past-paper questions.
1.Explain how linear regression can be used to predict the salary of an employee from their years of experience.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
2.Explain the relationship between the independent (predictor) and dependent (response) variables in linear regression.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
3.A regression line for revision hours (x) and test score (y) is y = 5.6x + 31. (a) Calculate the predicted score for 6.5 hours. (b) A student who revised for 6.5 hours scored 62. Calculate the residual.
[3]Calculate: Work out the number, showing the working.
4.Describe two ways of measuring how well a regression line fits the data.
[4]Describe: A detailed account: say what happens, step by step if it is a process.
5.Outline two limitations of using linear regression to predict a value.
[3]Outline: A brief account: the main points, without much detail.
In a nutshell
- Regression predicts a number. Linear regression fits a straight line y = mx + b; m is the gradient and b is the intercept.
- The independent variable (x) is the input; the dependent variable (y) is the output being predicted.
- Residual = actual − predicted. Mean squared error is the average of the squared residuals; the best-fit line makes it as small as possible.
- R² is the share of the variation in y that the line explains (0 to 1).
- Gradient descent finds the line by repeatedly stepping m and b downhill on the error. The learning rate sets the step size: too big and it overshoots.