A4.3.10 · Machine learning approaches
Choosing between models
By the end you can
Explain how to compare and select models using metrics, data size, speed, cost and explainability, and why different algorithms give different results.
So far this chapter has introduced many techniques. A real project must pick one. is the process of choosing the algorithm (and its settings) that suits the problem, and the comparison has to be fair. A common beginner's mistake is to chase the highest accuracy number alone.
Why the answers differ#
Give the same data to a linear regression, a k-nearest-neighbours model, a decision tree and a neural network, and you get four different models and four different scores. They differ in what they assume (a straight line; similar things are close together; yes/no splits; layered smooth functions), in how flexible they are, and in how they cope with noise, outliers, scale and the amount of data. A model that is perfect for one data set can be poor for another.
| Algorithm | Task | Strength | Weakness |
|---|---|---|---|
| Linear regression | Predict a number | Simple, fast, easy to explain. | Only straight-line patterns; sensitive to outliers. |
| k-nearest neighbours | Classify | No training; easy to set up. | Slow to predict on large data; needs scaled features; hurt by many features. |
| Decision tree | Classify or predict | Readable rules; no scaling needed. | Can overfit; unstable if the data changes. |
| Classify or predict | Combines many trees: usually more accurate and stable. | Harder to explain; slower than one tree. | |
| Neural network | Almost anything, especially images, speech and text | Learns very complex patterns. | Needs lots of data and computing power; hard to explain. |
Two other algorithms are named in the Oxford book, though not needed in detail: logistic regression (a classifier that draws a straight-line boundary and gives a probability) and support vector machines (which choose the boundary with the widest gap between classes).
The same algorithm can also behave very differently on different data. Some models cope poorly with imbalanced classes (a rare class swamped by a common one), with outliers and noise, or with features that need scaling first (k-NN and neural networks do; trees do not). Tree-based models handle correlated features well; others need help. Hence the best model depends on the data in front of you, not on a ranking in a book.
How to choose#
- Define the task and the cost of mistakes. The nature of the problem matters: classification and regression suit different algorithms. Is a false negative worse than a false positive? That decides which metric counts (A4.3.3).
- Prepare the data (A4.2) and split it, keeping a test set aside.
- Train several candidate models, from simple to complex. Start with a simple one as a baseline: if a complicated model barely beats it, the simple one wins.
- Tune each one fairly using validation data or cross-validation, giving every candidate the same effort.
- Compare on the same unseen test data using the chosen metrics, and check each for overfitting.
- Weigh the rest: how much data and hardware it needs, how long it takes to train and to predict, how it is, and whether it treats people fairly (A4.4).
- 1. Linear regression4.00 / 5
Predicts a number from a straight-line relationship. Simple and clear, but only as good as the straight-line assumption.
- 2. Decision tree4.00 / 5
Its rules can be read like a flowchart. A deep tree can overfit.
- 3. k-nearest neighbours3.40 / 5
Almost no training, but it must compare each new point with the stored data, so predictions slow down as the data grows.
- 4. Random forest3.00 / 5
Many decision trees voting together. Usually more accurate and stable than one tree, but harder to explain.
- 5. Neural network2.00 / 5
Can learn very complex patterns, but needs a lot of data and computing power and is hard to explain.
These ratings are rough, typical values (1 is poor, 5 is good), chosen to show the reasoning. On a real project you would test the candidate models on your own data and compare their measured results.
Check yourself
Choosing between models
Quick questions, made fresh every time and marked here. Do a few until they feel easy.
Exam practice
Paper 1-style questions
5 questions · 19 marks
Write your answer as you would in the exam, then show the mark points and tick each one your answer made. These are our own questions in the style of the exam, not past-paper questions.
1.Explain why different machine learning algorithms can give different results on the same problem.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
2.Outline four factors that should be considered when selecting a model.
[4]Outline: A brief account: the main points, without much detail.
3.Compare a single decision tree with a random forest.
[3]Compare: Similarities and differences, about both things, point by point.
4.Explain why two models should be compared on the same test data.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
5.A bank is choosing between a deep neural network that is 94% accurate and a decision tree that is 90% accurate for approving small loans. Evaluate which model it should select.
[6]Evaluate: Weigh the strengths against the weaknesses (or the benefits against the costs) and reach a judgement that you back up.
In a nutshell
- There is no best algorithm for every problem. Different algorithms make different assumptions, so they give different results on the same data.
- Compare candidate models on the same unseen test data using suitable metrics (error for regression; accuracy, precision, recall, F1 for classification).
- Also weigh: amount of data needed, training and prediction time, hardware and cost, explainability, fairness and risk.
- A random forest combines many decision trees: usually more accurate and stable than one tree, but harder to explain.
- Tune each candidate fairly (A4.3.3) before comparing, and do not use the final test set to choose.