1. Courses
  2. Reinforcement Learning
  3. Lecture 1
Lecture 1 of 1416 min3D modelFree lecture

The agent–environment loop

Reinforcement learning is learning by doing. An agent acts, the world responds with a new situation and a reward, and the agent tries to collect as much reward as it can over time. Meet the loop, the vocabulary and the robot you will train in this course.

In this lecture you will

  • Describe the agent–environment loop and its signals — state, action and reward
  • Distinguish reinforcement learning from supervised and unsupervised learning
  • Explain episodes, returns and the discount factor
  • Recognize real problems that fit the reinforcement-learning framing

A child learning to ride a bicycle is not shown thousands of labeled examples of correct balance. They try, wobble, fall, adjust and try again. The feedback is sparse and delayed — a bruise now and then, the thrill of riding a few metres — yet over time they get very good. Reinforcement learning (RL) is the branch of machine learning built on exactly this kind of learning by trial and error.

RL systems have beaten world champions at Go, cooled data centers more efficiently, controlled robot arms and, as you will see later in this course, shaped the behavior of the chat assistants you use every day.

The loop

Every reinforcement-learning problem has the same shape: an agent interacts with an environment in a loop.

AnimationThe agent–environment loop
Loading animation…
  1. Two playersThe agent is the learner and decision-maker. The environment is everything else — here, a small grid world with a goal worth +10 and two pits worth −10.
  2. Observe the stateAt each time step the environment tells the agent where it is. Here, the state is simply the robot’s position on the grid: row 4, column 0.
  3. Choose an actionThe agent picks one of the actions available — up, right, down or left. The environment applies it and the robot moves.
  4. Receive a rewardThe environment answers with a reward, a single number judging that step: −1 for an ordinary move. Together with the next state, it closes the loop.
  5. Repeat until the episode endsThe loop repeats. When the robot reaches the goal (or a pit) the episode is over. This route earned 7 × (−1) + 10 = +3 in total.
  6. Learn to do betterThe agent’s job is to change the way it chooses actions so that future episodes collect more reward. That is all “learning” means in reinforcement learning.
Agent
The learner: the system that observes and chooses actions. A game-playing program, a robot controller, a recommendation engine.
Environment
Everything the agent interacts with. It responds to each action with a new state and a reward.
State (s)
A description of the current situation — the robot’s position, the board in a game, a user’s recent activity.
Action (a)
A choice the agent can make in a state.
Reward (r)
A number the environment returns after each action, saying how good that step was. The designer chooses the reward to express the goal.
Episode
One complete run from a start state to an end state, such as one game or one attempt at a maze.
Policy (π)
The agent’s strategy: which action to take in each state.

Meet the grid world

Throughout this course we will use one small world, so you can focus on the ideas rather than the setting. Scroll through it in 3D.

Scroll storyThe grid world, step by stepStep 1 of 6
Loading 3D model…
  1. 1
    The environment

    A 5 × 5 grid with walls, two pits and a glowing goal. Each open tile is a state — 21 of them in total. The environment knows the rules; the agent does not, at first.

  2. 2
    The agent and its actions

    The robot can try to move up, right, down or left. Bumping into a wall or the edge leaves it where it was.

  3. 3
    Rewards express the goal

    Reaching the goal gives +10 and ends the episode. Falling into a pit gives −10 and also ends it. Every other move costs −1, so wandering is expensive.

  4. 4
    An unlucky episode

    Acting at random, the robot moves right twice — straight into a pit. Return: −1 − 10 = −11. The agent should learn that this route is a bad idea.

  5. 5
    A good episode

    Following a better strategy, the robot takes the shortest safe route: eight moves, seven costing −1 and the last earning +10, for a return of +3.

  6. 6
    The policy

    The arrows show the best action on every tile. Reinforcement learning is the process of discovering this map from experience alone — the subject of the rest of the course.

What the agent is really maximizing

The agent does not simply grab the biggest immediate reward. Moving right from the start looks harmless (−1) but leads next to a pit. What matters is the return: the total reward from now until the end of the episode.

In practice we discount rewards that arrive later, multiplying the reward tt steps away by γt\gamma^t, where γ\gamma (gamma) is a number between 0 and 1:

Gt=rt+1+γ rt+2+γ2rt+3+⋯=∑k=0∞γk rt+k+1G_t = r_{t+1} + \gamma\, r_{t+2} + \gamma^2 r_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k\, r_{t+k+1}

Discounting captures the intuition that a reward now is worth more than the same reward later — and it keeps the sum finite in tasks that never end.

InteractiveDiscounted return calculatorClick a reward to change it
Loading visual…

Try γ = 0: only the first reward matters, and the agent becomes completely short-sighted. At γ = 1 every reward counts equally, however distant.

Compare the two goal episodes at γ = 0.9: reaching the goal in 4 steps is worth far more than reaching it in 10 — even though both end with the same +10. The discount factor, together with the −1 step cost, is how we tell the agent that sooner is better.

Where reinforcement learning fits

Problems that fit the RL framing share three features: a sequence of decisions, actions that affect what happens next, and a measurable goal.

  • Games and simulation: board games, video games, strategy simulators.
  • Robotics and control: walking, grasping, drone flight, industrial process control.
  • Operations: data-center cooling, traffic-signal timing, inventory and pricing decisions.
  • Recommendation: choosing which content to show, where each choice changes what the user does next.
  • Language models: reinforcement learning from human feedback (RLHF) tunes chat assistants to give answers people prefer.

The simplest possible RL problem has only one state and a handful of actions — like a row of slot machines. It already contains the central dilemma of the whole field: should you exploit what you know, or explore to learn more? That is the next lecture.

Key takeaways

  • An agent observes a state, chooses an action, and receives a reward and a new state from the environment — over and over.
  • The goal is not the next reward but the return, the total reward collected over time.
  • A discount factor γ makes near rewards count more than distant ones and keeps returns finite.
  • A policy is the agent’s strategy — which action to take in each state. Reinforcement learning is the search for a good policy through experience.

Check your understanding

4 questions · answers explained

Q1What makes reinforcement learning different from supervised learning?

In supervised learning every example comes with the right answer. In reinforcement learning the agent only receives a reward signal, often delayed, and has to work out which of its earlier actions deserve the credit.

Q2An episode gives rewards −1, −1, −1 and +10. With a discount factor γ = 0.5, what is the return?

G = r₁ + γr₂ + γ²r₃ + γ³r₄ = −1 − 0.5 − 0.25 + 0.125 × 10 = −0.5. With such a strong discount, even a +10 reward three steps away barely outweighs the small costs of getting there.

Q3What is a policy?

A policy maps states to actions (or to probabilities over actions). Learning a good policy is the whole point of reinforcement learning.

Q4Why is every ordinary step given a small negative reward (−1) in the grid world?

A step cost means every wasted move lowers the return. The agent is therefore rewarded for reaching the goal quickly — the designer expresses “be efficient” through the reward.