Primary memory

A CPU core can do billions of operations a second, but only if the data it needs is right there. Primary memory is the memory the CPU can use directly, and it comes in several kinds because no single technology is fast, big and cheap all at once. The faster a memory is, the more it costs per byte, so computers use a little of the very fast kind and a lot of the slower kind.

The CPU works down the hierarchy below. Registers at the top hold the handful of values being used this instant; each level further down is bigger, slower and cheaper per byte. The aim is to keep latency (the wait for each item) low and throughput (data delivered per second) high, by keeping what is needed next as close to the CPU as possible.

RAM (random access memory) holds whatever is in use right now: the operating system, the programs you have open and their data. When you open an app, it is copied from storage into RAM so the CPU can reach it quickly. RAM can be read and written as often as needed, and any location can be reached as quickly as any other (that is what “random access” means).

RAM is volatile: switch off the power and everything in it is gone. That is why unsaved work is lost in a power cut, and why files must be saved to secondary storage. More RAM lets more programs stay open at once without slowing down.

When RAM fills up, the operating system can move the parts of programs that are not being used out to a reserved area of secondary storage, and bring them back when they are needed. This is virtual memory. It stops the computer running out of memory, but storage is thousands of times slower than RAM, so a computer that leans on it heavily becomes sluggish, with the drive constantly busy. The real cure is more RAM.

ROM (read-only memory) holds instructions that must survive with the power off and rarely change. Its best-known job is the start-up firmware (BIOS, or its modern replacement UEFI): when you switch on, RAM is empty, so the CPU runs the firmware in ROM, which tests the hardware and then loads the operating system from storage into RAM. In phones, ROM-style storage also holds the core system software. Modern “ROM” is usually flash memory, so the firmware can still be updated occasionally.

Even RAM is slow compared with the CPU: fetching from RAM can take a couple of hundred clock cycles, during which the core would sit idle. Cache is a small amount of very fast memory on the CPU chip that keeps copies of data and instructions the CPU has used recently or is likely to need soon. It works because of locality, two habits almost every program has:

• Temporal locality: data used recently is likely to be used again soon (a loop counter, the instructions of the loop itself).
• Spatial locality: data next to what was just used is likely to be used next (the next item of an array, the next instruction). So a cache loads a whole block of neighbouring bytes at a time, not just the one asked for.

Cache comes in levels:

• L1: the smallest (tens of kilobytes per core) and fastest, inside each core; usually split into one part for instructions and one for data.
• L2: bigger (hundreds of kilobytes to a few megabytes) and a little slower, usually one per core.
• L3: the biggest (several to tens of megabytes) and slowest cache, usually shared by all the cores, but still much faster than RAM.

The CPU looks in L1 first, then L2, then L3, then RAM. Finding the data in a cache is a cache hit; not finding it is a cache miss, and the data has to come from further away. The hit rate (hits ÷ requests) is the measure of how well the cache is working.

A bigger cache holds more, so it misses less, but it costs more and can be slightly slower. Modern CPUs also prefetch: they notice patterns, such as reading through an array, and load the next blocks into cache before they are asked for.