A4.2.2 · Data preprocessing
Feature selection
By the end you can
Explain what a feature is and how to keep the useful ones, using filter, wrapper and embedded methods, and remove redundant and irrelevant features.
A is one measurable property used as an input: floor area, age, number of bedrooms. The thing you want to predict is the . When you gather data it is tempting to keep every column you can find, but more features is not the same as more knowledge. is the step of deciding which features earn their place.
Two kinds of feature are worth removing.
- A repeats what another one already says: floor area in square metres and in square feet, or two features with a correlation of 0.98.
- An tells the model nothing about the target: a house's front-door colour or the owner's birth month. Worse, the model may find a coincidence in the training data and rely on it.
Features used
9
Fit on training houses
0.998
R² (1 is perfect)
Fit on unseen houses
0.949
R² on the held-back half
r is the correlation with price. Adding features never lowers the fit on the training houses, but the fit on unseen houses can fall: the model starts learning noise. Square feet is the same information as square metres, so it adds nothing.
Try “Area, age, distance to school”. It has the best fit on unseen houses. Then try “Everything”: the training score creeps up but the unseen score drops, because the model has started fitting the random columns. That gap between training and unseen performance is overfitting (A4.3.3), and feature selection is one defence against it.
Three families of method#
| Method | How it works | Strength | Weakness |
|---|---|---|---|
| Score each feature on its own (correlation with the target, chi-square, mutual information) and keep the best. | Fast and independent of any model. | Ignores how features work together, and may keep two redundant ones. | |
| Train a model on a subset, score it, then add or remove a feature and repeat (forward selection, backward elimination, recursive feature elimination). | Judges features inside a real model, so usually finds a better subset. | Slow, because it trains many models. | |
| The model selects while it trains, for example a penalty (as in Lasso) that shrinks unhelpful weights to zero. | Efficient: selection and training happen together. | Tied to algorithms that have such a mechanism. |
For house prices, a filter would compute each feature's with price and keep floor area while dropping door colour. A wrapper might start with all features, remove the one whose absence hurts least, and repeat. An embedded method would train a regression with a penalty and find that the weights of age and door colour have shrunk to zero.
Check yourself
Feature selection
Quick questions, made fresh every time and marked here. Do a few until they feel easy.
Exam practice
Paper 1-style questions
5 questions · 17 marks
Write your answer as you would in the exam, then show the mark points and tick each one your answer made. These are our own questions in the style of the exam, not past-paper questions.
1.Define feature selection, and state why it is done.
[2]Define: Give the precise meaning of the term.
2.Outline the three different strategies for feature selection.
[6]Outline: A brief account: the main points, without much detail.
3.Distinguish between a redundant feature and an irrelevant feature, giving an example of each for a house-price model.
[2]Distinguish: Make the differences between two things clear.
4.A house-price data set has these correlations with price: floor area 0.92, number of bedrooms 0.85, front-door colour 0.02. Floor area and number of bedrooms have a correlation of 0.9 with each other. Explain which features you would keep.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
5.Describe four benefits of feature selection.
[4]Describe: A detailed account: say what happens, step by step if it is a process.
In a nutshell
- A feature is one input column. Feature selection keeps the informative ones and drops redundant and irrelevant ones.
- Redundant: repeats what another feature says. Irrelevant: has no link to the output. Both can be removed.
- Filter methods score features first (for example correlation). Wrapper methods test subsets with a model. Embedded methods select during training.
- Benefits: fewer features means less overfitting, faster training, simpler and more explainable models, and often better accuracy.
- The aim is to keep the most informative features, not simply to delete columns. Removing a useful one can hurt the model.