A4.1.2 · Machine learning fundamentals
Hardware for machine learning
By the end you can
Choose hardware (laptop, workstation, GPU server, TPU, FPGA, ASIC, edge device, cloud) for development, data processing, training, deployment and edge scenarios.
Machine learning is not one workload. A student trying out a small model, a company training a model on ten million photos, and a phone recognising a face have very different needs. The syllabus asks you to match the hardware to the scenario, so this page works through five of them and the chips that make them possible.
Five scenarios#
| Scenario | What happens | Suitable hardware |
|---|---|---|
| Development and testing | Writing code and trying algorithms on small data sets to see what works. | A standard laptop or desktop: multi-core CPU, 8 to 16 GB of RAM, an SSD. |
| Data processing and feature engineering | Cleaning, joining and transforming large data, and creating new features. | A workstation or server: strong CPUs, a lot of RAM, large fast storage (SSDs, or several disks in RAID). |
| Model training and deep learning | Adjusting millions of parameters over many passes through the data. | GPU servers (several GPUs) or TPUs, with fast high-capacity storage. Often rented in the cloud. |
| Large-scale deployment and production | Serving predictions to many users, reliably, all day. | Servers or cloud platforms with scalable CPUs and GPUs or TPUs, and strong networking and storage. |
| Running the model on the device that collects the data: a phone, a car, a camera. | Compact, low-power chips, often with a small GPU or an ASIC accelerator, flash storage and a few GB of RAM. |
Notice what changes between rows. Early on, flexibility and cost matter. During training, raw parallel arithmetic matters. In production, reliability and scale matter. At the edge, energy and delay matter most. The hardware advice is also a snapshot: machines improve so fast that any list of RAM sizes is out of date within a few years, so learn the reasoning rather than the numbers.
Development and testing
A student writes and tests a model on a small data set of 5 000 rows to learn how the algorithm behaves.
The chips behind it#
Training a neural network is mostly multiplying and adding enormous grids of numbers called . A CPU does this slowly because it has only a few cores. Several other kinds of hardware are better suited.
- CPU: a general-purpose processor. It can run introductory machine learning, but it is slow when the same operation must be done on huge arrays.
- : thousands of simple cores doing the same operation on many numbers at once. Built for graphics (see A1.1.2) and now the standard machine-learning workhorse.
- : a chip designed from the start for the tensor calculations of neural networks. It is an ASIC. It is built around a large matrix-multiplication unit and works with low-precision numbers (often 16 or even 8 bits), because neural networks do not need more precision, and that makes it fast and efficient.
- : a chip made for one job. It cannot be changed, but it is fast and sips power, which makes it a good fit for phones and cameras. Designing one costs a great deal, so it only pays off when the chip is made in very large numbers for a workload that will not change.
- : a chip whose circuits can be rewired after it is made. Less efficient than an ASIC, but it can be adapted as the algorithm changes.
- : a neural processing unit built into a laptop or phone chip to run neural-network calculations quickly on very little power.
- Cloud platforms: rented CPUs, GPUs and TPUs by the hour. The suits bursty or experimental work and demand that rises and falls, because you pay only for what you use. The downside is dependence on the provider: you must trust its security, send it your data and keep paying.
- centres: thousands of processors linked together for the biggest models and simulations. Unlike the cloud they are usually not open to anyone who pays: access is often limited to researchers and institutions.
A handy way to decide, for a trained model that must run quickly: does it need to run on the device? Then choose an NPU or ASIC for power efficiency. If it can run on a server and needs real-time speed, a GPU. If it can be processed in batches, a TPU may be more efficient. If the algorithm may change in future, an FPGA, because it can be reprogrammed; if it will not change and is made in large numbers, an ASIC. If demand may jump, rent scalable cloud GPUs or TPUs.
Some laptops now put the CPU, GPU and neural engine on one chip sharing the same memory, so a model does not have to be copied between the CPU's RAM and a separate GPU's memory. That is one reason a modern laptop can do more machine learning than its price suggests. Processing on the edge also keeps data on the device, which helps privacy, though the owner then has to buy and maintain the hardware.
Cost, performance and operating needs#
Each setup balances three things: cost (to buy and to run), performance (how fast it trains or predicts) and operational needs (reliability, power, size, security and where the data is allowed to go). Cloud platforms trade a large purchase for ongoing rental and hand maintenance to the provider, but your data then sits on someone else's servers. Edge devices keep data private and responses instant, but have little power to spare. Good answers state the need first and the hardware second.
Check yourself
Hardware for machine learning
Quick questions, made fresh every time and marked here. Do a few until they feel easy.
Exam practice
Paper 1-style questions
5 questions · 17 marks
Write your answer as you would in the exam, then show the mark points and tick each one your answer made. These are our own questions in the style of the exam, not past-paper questions.
1.Describe the hardware requirements for developing and testing a machine learning model on a small data set.
[2]Describe: A detailed account: say what happens, step by step if it is a process.
2.Explain why training a deep neural network is usually done on GPUs or TPUs rather than on a CPU.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
3.Describe the hardware requirements for running a face-recognition model on a smartphone (edge computing).
[3]Describe: A detailed account: say what happens, step by step if it is a process.
4.Distinguish between an ASIC and an FPGA for machine learning.
[3]Distinguish: Make the differences between two things clear.
5.A start-up wants to train a large model once every few months. Evaluate using a cloud platform rather than buying its own GPU servers.
[6]Evaluate: Weigh the strengths against the weaknesses (or the benefits against the costs) and reach a judgement that you back up.
In a nutshell
- Different stages need different hardware: development and testing (laptop), data processing (workstation or server with plenty of RAM and storage), training (GPUs or TPUs), deployment (scalable servers or cloud), edge (small, low-power devices).
- GPUs suit machine learning because training is mostly large amounts of parallel arithmetic on tensors. TPUs are ASICs built for tensor calculations.
- ASICs are fixed-purpose chips: fast and efficient. FPGAs can be reconfigured after manufacture: flexible. Edge devices often contain one of these.
- Cloud platforms rent CPUs, GPUs and TPUs on demand, which suits bursty or experimental work. HPC centres link thousands of processors for the very largest jobs.
- Training is the expensive part. Inference (using the trained model) needs far less, which is why a phone can run a model that took a data centre weeks to train.