A4.3.3 · Machine learning approaches
Evaluating and tuning a model
By the end you can
Split data, read a confusion matrix, calculate accuracy, precision, recall and F1, spot overfitting and underfitting, and explain what hyperparameter tuning does.
Training a model is only half the job. The question that matters is does it work on data it has never seen? This page is about answering that honestly: how to test a model, what the numbers mean, what goes wrong when a model learns too much or too little, and how to adjust the settings.
Keep some data back#
If you test a student on the exact questions they practised, a good score might only show a good memory. A model is the same. So the data is split. The training set (often 70% to 80%) is used to fit the model. A validation set is used while choosing settings. The is locked away and used once, at the end, to estimate how the model will do in real use. If you keep adjusting the model until the test score looks good, the test set stops being unseen.
When data is scarce, helps: split the data into, say, five parts, train five times each holding out a different part as the test set, and average the five scores. Every example is used for both training and testing, but never in the same round.
The confusion matrix and its measures#
For a classifier with two outcomes (call one “positive”, for example “ill”), every prediction falls into one of four boxes of the :
- True positive (TP): predicted positive, really positive.
- False positive (FP): predicted positive, really negative (a false alarm).
- True negative (TN): predicted negative, really negative.
- False negative (FN): predicted negative, really positive (a miss).
| Measure | Formula | Answers the question | Use it when |
|---|---|---|---|
| (TP + TN) ÷ all | How often is the model right overall? | Classes are balanced and both kinds of mistake cost about the same. | |
| TP ÷ (TP + FP) | When it says positive, how often is it right? | False alarms are costly (spam filters, convicting someone). | |
| TP ÷ (TP + FN) | Of the real positives, how many did it find? | Missing a case is costly (disease screening, fraud). | |
| 2 × P × R ÷ (P + R) | A single balance of precision and recall. | You need both, or the classes are unbalanced. |
In the lab below, 40 patients (14 of them ill) are scored by a model. At the threshold 0.50 the matrix has TP = 11, FN = 3, FP = 2 and TN = 24. So accuracy = (11 + 24) ÷ 40 = 87.5%, precision = 11 ÷ (11 + 2) = 84.6% and recall = 11 ÷ (11 + 3) = 78.6%, giving an F1 of 0.81.
A model gives each of 40 patients a score from 0 to 1. Patients at or above the threshold are predicted ill. The strip shows the 40 patients in score order: a filled mark is truly ill, a hollow one is well.
11
true positive
3
false negative
2
false positive
24
true negative
Accuracy
88%
(TP + TN) ÷ all
Precision
85%
TP ÷ (TP + FP)
Recall
79%
TP ÷ (TP + FN)
F1 score
0.81
balance of the two
Lower the threshold and the model catches more ill patients (recall rises) but raises more false alarms (precision falls). Raise it and the opposite happens. For a serious illness you would usually accept more false alarms to avoid missing a case.
Overfitting and underfitting#
A model has to be flexible enough to follow the real pattern, but not so flexible that it follows the noise.
- : the model is too simple (a straight line through a curve). It does badly on the training data and the test data.
- : the model is too flexible (a wiggly curve through every point). It does very well on the training data and badly on the test data, because it has learned the noise, which does not repeat.
- A good fit gets right: similar, good performance on both.
Overfitting is caused by a model that is too complex for the amount of data, noisy or unclean data, too many features (A4.2.3) or training too long. Underfitting is caused by a model that is too simple, too few or poor features, or too little training. Remedies for overfitting: more (and cleaner) data, fewer features, a simpler model, a limit on tree depth, or stopping training earlier. Remedies for underfitting: a more flexible model, better features, more training.
Filled squares: training data (the model sees these). Hollow triangles: test data (held back, used only to check).
Training error
0.024
Only ever falls as flexibility rises
Test error
0.072
Falls, then rises again
Underfitting: the model is too simple to capture the pattern, so it does badly on both training and test data.
| Degree | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|
| Train | 0.024 | 0.023 | 0.007 | 0.007 | 0.005 | 0.005 | 0.005 | 0.002 | 0.002 |
| Test | 0.072 | 0.064 | 0.033 | 0.033 | 0.037 | 0.037 | 0.038 | 0.083 | 0.222 |
In Figure 2 the training error keeps falling as the curve gets more flexible: 0.024 for a straight line, 0.007 for degree 3 and 0.002 for degree 9. The test error tells a different story: 0.072, 0.033, then 0.222. The degree-9 curve is the best on the data it saw and the worst on the data it did not. That gap is what overfitting looks like in numbers.
Hyperparameters and tuning#
Some numbers are learned during training: the slope of a regression line, the weights in a neural network. These are parameters. Others are chosen by the developer beforehand, and these are : k in k-NN, the maximum depth of a tree, the learning rate of gradient descent, the number of layers in a network.
Hyperparameter tuning means trying different values, training a model for each, and comparing their scores on validation data. A simple way is to try a list of values in turn. Tuning is how you find the setting between underfitting and overfitting: a tree depth of 1 underfits, a depth of 20 overfits, and somewhere between lies the best. Do the comparing on the validation data, and keep the test set for a single final check.
| Hyperparameter | Too low | Too high |
|---|---|---|
| k in k-NN | Overfits: noisy, jagged boundary | Underfits: blurs real differences |
| Depth of a decision tree | Underfits: too few questions | Overfits: memorises the training data |
| Learning rate | Learning is very slow | Overshoots, error may grow |
| Polynomial degree / number of layers | Underfits: too simple | Overfits: too flexible |
Check yourself
Evaluating and tuning a model
Quick questions, made fresh every time and marked here. Do a few until they feel easy.
Exam practice
Paper 1-style questions
6 questions · 22 marks
Write your answer as you would in the exam, then show the mark points and tick each one your answer made. These are our own questions in the style of the exam, not past-paper questions.
1.A model that detects a disease produces 40 true positives, 10 false positives, 10 false negatives and 140 true negatives. Calculate its accuracy, precision and recall.
[4]Calculate: Work out the number, showing the working.
2.A classifier has a precision of 0.8 and a recall of 0.5. Calculate its F1 score, to 2 decimal places.
[2]Calculate: Work out the number, showing the working.
3.Explain why the data is split into a training set and a test set.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
4.Distinguish between overfitting and underfitting, saying how each can be recognised.
[4]Distinguish: Make the differences between two things clear.
5.Explain the role of hyperparameter tuning when building a k-nearest neighbours classifier.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
6.A cancer screening model is reported to be 98% accurate on a data set in which 2% of patients have the disease. Evaluate whether accuracy alone is a good measure of the model’s performance.
[6]Evaluate: Weigh the strengths against the weaknesses (or the benefits against the costs) and reach a judgement that you back up.
In a nutshell
- Split the data: train on the training set, tune on a validation set, and measure the final model once on a test set it has never seen.
- Confusion matrix: TP, FP, TN, FN. Accuracy = (TP + TN) ÷ all. Precision = TP ÷ (TP + FP). Recall = TP ÷ (TP + FN). F1 = 2PR ÷ (P + R).
- Accuracy misleads when classes are unbalanced. Use recall when missing a case is costly, precision when false alarms are costly, F1 to balance them.
- Overfitting: great on training data, poor on new data. Underfitting: poor on both. Aim for a model that generalises.
- Hyperparameters (k, tree depth, learning rate) are set before training and tuned by comparing results; parameters (weights, gradients) are learned. Cross-validation uses each part of the data as the test set in turn.