A4.3.6 · Machine learning approaches
Reinforcement learning
By the end you can
Describe how an agent learns from rewards by trying actions in an environment, including exploration versus exploitation and the Q-learning update.
Nobody teaches a toddler to walk by showing labelled photos of walking. They try, fall, and try again, and what works feels good. (RL) trains software the same way. There is no data set of right answers. Instead there is a running experiment with these parts:
- An that makes decisions (a robot, a game-playing program, a trading bot).
- An it acts in, which can be a real room or a simulated game.
- A : what the agent can tell about the situation right now.
- An it chooses in that state.
- A (or penalty) the environment returns, and the next state.
The loop is: observe the state, choose an action, receive a reward and a new state, learn from it, repeat. The set of all possible states is the state space and the set of all possible actions is the action space. The agent is searching for a good : a rule that says which action to take in each state. A policy can be deterministic (each state maps to one action) or stochastic (each action is chosen with some probability).
What the agent maximises is not one reward but the cumulative reward (also called the return): the total it collects over a run. It is often discounted, so that a reward soon counts for more than one far away. With rewards of 2, 4 and 8 on three steps and a discount of 0.5, the return is 2 + 0.5 × 4 + 0.25 × 8 = 6.
| Agent | Environment | Reward |
|---|---|---|
| Game-playing program | The game and its opponent | +1 for a win, −1 for a loss |
| Warehouse robot | The warehouse | + for delivering an item, − for collisions and delay |
| Trading bot | The market | Profit made, minus risk taken |
| Adaptive learning app | The student | Higher when the student stays engaged and improves |
| Robot vacuum cleaner | The house | + for new floor cleaned, − for collisions |
| Delivery drone | Terrain, weather and obstacles | + for a safe, efficient delivery, − for unsafe moves or delay |
Q-learning#
A simple way to learn a policy is . The agent keeps a table with one number for each state and action: Q(state, action), its current estimate of the total future reward from taking that action in that state. After every step it updates the entry it just used:
The part in brackets is the surprise: what the agent has just learned (the reward plus the discounted value of what comes next) minus what it expected. The learning rate decides how much of that surprise to take on board. The discount shrinks the future: a γ of 0.9 means a reward one step further away is worth 90% as much, so quick routes beat slow ones.
A tiny example. In the grid world below every step costs −1 and the goal pays +10. Start with Q = 0 for everything, α = 0.5 and γ = 0.9. After a first step of reward −1 into a state whose best Q is 0: new Q = 0 + 0.5 × (−1 + 0.9 × 0 − 0) = -0.5. Later, a step that reaches the goal (reward +10, nothing after it) from a square whose Q is now -0.5: new Q = -0.5 + 0.5 × (10 − (-0.5)) = 4.75. The good news spreads backwards, one square per update, until the whole path to the goal has high values.
Numbers: the learned value (Q) of moving up, right, down or left from that square. Every ordinary step costs −1. The best action in each square is shown in green.
The agent starts at the top left. Every Q-value starts at 0.
Episodes finished
0
Greedy path length
–
steps to the goal if it always exploits
Last episode reward
–
Explore or exploit?#
If the agent always picks the action with the highest Q (), it may lock in the first route that worked, even if a much shorter one exists. If it always picks randomly (), it never makes use of what it learned. A common compromise, ε-greedy, picks a random action with probability ε and the best-known action the rest of the time, and often lowers ε as learning goes on. In the lab, set ε to 0 on a fresh start: the agent follows its first accidental path and may never find the best one.
In practice ε is often decayed: high at the start, when the agent knows nothing, and lower later, when it knows more. Other strategies exist, such as upper confidence bound (choose the action with the highest optimistic estimate) and Thompson sampling (sample a plausible reward for each action from what has been learned and pick the best sample); both textbooks name them, but ε-greedy is the one worth being able to explain.
Designing rewards, and the limits#
- The reward is the only teacher. If it is badly designed, the agent will find an unintended way to score. A cleaning robot paid for every moment it spends moving may learn to spin on the spot.
- Learning takes a lot of trials, so RL is often done in simulation first (a game or a virtual warehouse) before a physical robot is risked.
- Rewards may be delayed. A chess move may only pay off 40 moves later. The discount and the update rule are how credit is passed back along the path.
- Safety. In a real setting, exploration can be dangerous or costly, so the actions must be limited.
Check yourself
Reinforcement learning
Quick questions, made fresh every time and marked here. Do a few until they feel easy.
Exam practice
Paper 1-style questions
6 questions · 22 marks
Write your answer as you would in the exam, then show the mark points and tick each one your answer made. These are our own questions in the style of the exam, not past-paper questions.
1.Describe the principle of reinforcement learning.
[4]Describe: A detailed account: say what happens, step by step if it is a process.
2.Define the terms “state”, “action” and “policy” in reinforcement learning.
[3]Define: Give the precise meaning of the term.
3.Explain the trade-off between exploration and exploitation.
[3]Explain: Give reasons: say what happens AND why, or how one thing causes another.
4.A Q-learning agent has Q(s, a) = 2. It takes action a, gets a reward of 4 and arrives in a state whose best Q value is 10. With learning rate α = 0.5 and discount γ = 0.9, calculate the new Q(s, a).
[3]Calculate: Work out the number, showing the working.
5.Outline how reinforcement learning is used in robotics navigation.
[3]Outline: A brief account: the main points, without much detail.
6.Explain how reinforcement learning could be used to teach a robot vacuum cleaner to clean a house efficiently, including how the reward could be designed and how the robot balances exploration and exploitation.
[6]Explain: Give reasons: say what happens AND why, or how one thing causes another.
In a nutshell
- Reinforcement learning: an agent takes actions in an environment and learns from rewards and penalties, aiming to maximise total reward. No labelled examples are used.
- Key terms: agent, environment, state, action, reward, policy (the rule for choosing actions).
- Q-learning keeps a table of Q values and updates one after each step: new Q = Q + α × (reward + γ × max next Q − Q).
- Exploration (try new actions) versus exploitation (use the best known). ε-greedy picks a random action with probability ε.
- The discount γ makes later rewards count for less. Uses: game playing, robot navigation, trading bots, adaptive systems. Poorly chosen rewards teach the wrong behaviour.