A4.2.1 · Data preprocessing
Cleaning data
By the end you can
Explain why raw data must be cleaned, and apply the standard fixes: missing values, duplicates, outliers, wrong formats, and normalisation or standardisation.
Real data is untidy. Forms are left half-filled, people type “n/a”, the same customer appears twice, one salary has an extra zero. A model cannot tell which entries are mistakes: it treats everything it is given as truth. That is the idea behind the old saying . is the work of fixing the data before the model sees it, and it often takes more of a project's time than the modelling itself.
Poor data hurts in three ways. It makes predictions less accurate. It makes the model overfit to instead of the real pattern, so it fails on new data. And if the data is unrepresentative, it makes the model unfair.
Common problems and their fixes#
| Problem | Example | Typical fixes |
|---|---|---|
| Missing data | An age or an email address is blank or says “n/a”. | (mean, median, mode, or predict it), delete the rows, or model it from other columns. |
| Duplicates | The same customer appears in two rows. | Find them by a key or a combination of fields, then remove or merge them (average numbers, keep the most common category). |
| A salary of 410 000 among values near 35 000. | Spot with the rule, z-scores or a box plot; then trim, cap or transform. | |
| Incorrect values | An age of −25 or “thirty”. | Validation rules (ranges, formats), cross-checking another source, converting text numbers. |
| Wrong format | Dates as 01-02-2020 and 2020/02/01; “leeds” and “Leeds”. | Parse and standardise to one format; regular expressions help. |
| Irrelevant data | A column that has no bearing on the question. | Remove it (this leads to feature selection, A4.2.2) and ask an expert. |
| Name | Age | Income | City |
|---|---|---|---|
| Amy | 24 | 31,000 | Leeds |
| Ben | 31 | 38,000 | leeds |
| Cara | (missing) | 36,000 | York |
| Dev | 29 | 34,000 | York |
| Dev | 29 | 34,000 | York |
| Eli | 41 | 52,000 | Leeds |
| Fay | 35 | 410,000 | Hull |
| Gus | (missing) | 33,000 | hull |
| Hana | 27 | 30,000 | Hull |
| Ivo | 38 | 45,000 | Leeds |
Rows
10
started with 10
Mean age
31.8
Mean income
74,300
was 74,300
Different city names
5
Look at the mean income in the lab. Keeping the 410 000 outlier pulls the average far above what almost every customer earns, and capping or removing it changes the answer dramatically. That is why you check why a value is extreme before deciding. If it is a typing mistake, fix or drop it. If it is a real rare customer, dropping it may throw away exactly the case you most wanted to understand.
Putting features on the same scale#
Imagine a data set with age (about 20 to 60) and income (about 20 000 to 400 000). In a distance calculation, income differences are thousands of times bigger than age differences, so income swamps age whether or not it matters. Rescaling fixes this. There are two standard ways.
- (min-max scaling): x′ = (x − min) ÷ (max − min). Every value lands between 0 and 1. Good for algorithms that are sensitive to scale, such as gradient descent and k-NN, but a single outlier squashes everything else into a narrow band.
- (z-score): z = (x − mean) ÷ standard deviation. Values are centred on 0 and measured in standard deviations. It is not bounded to a range, and it copes better with outliers and with unknown minimum and maximum.
| Age | Income | Income, min-max | Income, z-score |
|---|---|---|---|
| 24 | 31,000 | 0.00 | -0.46 |
| 31 | 38,000 | 0.02 | -0.41 |
| 29 | 34,000 | 0.01 | -0.44 |
| 41 | 52,000 | 0.06 | -0.30 |
| 35 | 410,000 | 1.00 | 2.45 |
| 27 | 30,000 | 0.00 | -0.47 |
| 38 | 45,000 | 0.04 | -0.36 |
With the outlier, min-max scaling squashes every other income into the range 0 to 0.06, so they look almost identical to the model. The outlier also inflates the standard deviation, so the z-scores of the ordinary incomes are squeezed too, but the extreme value still stands out as far above the rest.
Check yourself
Cleaning data
Quick questions, made fresh every time and marked here. Do a few until they feel easy.
Exam practice
Paper 1-style questions
6 questions · 23 marks
Write your answer as you would in the exam, then show the mark points and tick each one your answer made. These are our own questions in the style of the exam, not past-paper questions.
1.Describe the significance of data cleaning in the machine learning workflow.
[4]Describe: A detailed account: say what happens, step by step if it is a process.
2.Describe three techniques used to deal with missing values in a data set.
[3]Describe: A detailed account: say what happens, step by step if it is a process.
3.Outline how outliers can be identified and handled in a data set.
[3]Outline: A brief account: the main points, without much detail.
4.Explain why a data set with one feature measured in the range 0 to 1 and another in the range 0 to 1 000 000 should be normalised or standardised before training.
[4]Explain: Give reasons: say what happens AND why, or how one thing causes another.
5.A column of ages is 20, 30, 40 and 50. Calculate the value 40 after (a) min-max normalisation and (b) standardisation, given a standard deviation of 11.18 (to 2 d.p.).
[3]Calculate: Work out the number, showing the working.
6.A company trains a model to screen job applications using ten years of past records, which contain many missing fields, duplicate entries and mostly male applicants. Discuss the effect of this data quality on the model, and what the company should do.
[6]Discuss: Look at the question from more than one side, weigh the points against each other, and come to a conclusion.
In a nutshell
- Garbage in, garbage out: a model is only as good as its data. Cleaning fixes errors and inconsistencies before training.
- Missing values: impute (mean, median, mode or predicted), delete rows, or use predictive modelling. Duplicates: remove or merge. Wrong formats: parse and standardise. Invalid values: validate against rules.
- Outliers: find with IQR fences, z-scores or plots; then trim, cap or transform, after checking whether they are mistakes or genuine rare cases.
- Normalisation (min-max) rescales to a fixed range such as 0 to 1. Standardisation (z-score) gives mean 0 and standard deviation 1 and copes better with outliers.
- Clean data should also be representative. Cleaning does not remove bias that is already in the history.