Markov Decision Processes: Decisions, Made Visible

A robot’s next action changes the situation it will face afterward. A short movement can bring it closer to its goal, increase the chance of failure, or make a later action easier. Choosing well means considering both the immediate result and what follows.

A Markov decision process, or MDP, gives us a mathematical way to describe that problem. This explainer makes the pieces visible in an isometric room: choose a state, inspect an action, and watch estimates of future reward turn into a policy.

Try the interactive explainer

Start with Step once. Repeat a few times to see rewards influence states farther from the goal. Then choose Solve to convergence and Run robot. Change the slip probability or the step reward and repeat the experiment.

Open the 3D decision lab full-screen →

Select a floor tile to inspect it, switch between isometric and top-down views, or hide the policy overlay to focus on the room. The room and its controls are part of this page, so you can scroll naturally between the experiment and the explanation. All calculations run in your browser.

The room is a 3D view of the same 4 × 3 MDP: the cabinet blocks a cell, the cyan dock is the goal, and the dark opening is the pit. Geometry does not change the transition model. Arrows follow grid directions projected into the camera view; Up still means toward the first row. If 3D is unavailable, the lab falls back to an interactive grid.

Five pieces define the world

An MDP is commonly described by the tuple (S, A, P, R, γ).

Piece	Meaning	In this explainer
States, S	The possible situations the agent can be in	The accessible grid cells, including the goal and pit
Actions, A	The choices available to the agent	Up, right, down, and left
Transitions, P	The probability of the next state given a state and action	The intended direction, with a chance of slipping sideways
Rewards, R	The immediate result of a transition	Goal reward, pit penalty, or the ordinary step reward
Discount, γ	How much later rewards count today	A slider from 0 to 0.99

The Markov property says that, given the current state and action, the next-state distribution does not depend on the earlier history. In this fixed grid world, position contains the information needed to predict the next outcome.

This assumption needs care in robotics. A single camera frame may omit velocity, contact forces, or an occluded object. A useful state representation must account for the information the dynamics need; partial observations may require memory or a belief over states.

An action is a choice, not a guaranteed outcome

At the default slip probability of 20%, choosing up produces the intended movement with probability 0.8. The agent instead tries left with probability 0.1 or right with probability 0.1.

If a movement hits a wall or the boundary, the agent stays where it is. When multiple outcomes lead to the same cell, their probabilities add together. For example, choosing up from the top-left corner leaves the agent there with probability 0.9: both the intended upward move and the leftward slip are blocked.

The goal and pit end an episode. Their rewards are paid on entry. Once inside, there are no further rewards, so their stored continuation values are zero. Every other transition, including a blocked move, receives the step reward. These conventions keep a terminal reward from being counted twice.

From a single reward to the value of a state

A policy, π, specifies which action to take in each state. Its value function, Vπ(s), is the expected discounted return starting from state s and following that policy.

G_t = R_t+1 + γR_t+2 + γ²R_t+3 + …

With γ close to one, rewards many steps away still matter. With γ equal to zero, only the next reward matters. Discounting applies to penalties as well as positive rewards.

The floor colors show stored value estimates. The arrows show the action that currently looks best according to those estimates. Early arrows can change as information propagates through the grid; they are not yet a finished solution.

Look inside the Bellman update

Value iteration begins here with V₀(s) = 0 for every state. It repeatedly looks one step ahead, evaluates all available actions, and keeps the largest expected result:

Q_k(s, a) = Σ_s′ P(s′ | s, a) [R(s, a, s′) + γV_k(s′)]

V_k+1(s) = max_a Q_k(s, a)

The first equation combines three things: the chance of an outcome, its immediate reward, and the discounted value of what comes next. The second chooses the best action’s result.

For a concrete example, select cell (1, 3), immediately left of the goal, and inspect Right before any iterations. With the defaults, the intended move reaches the goal with probability 0.8; the two sideways outcomes each receive −0.04. Future values are still zero, so:

Q₀((1, 3), Right) = 0.8 × 1 + 0.1 × (−0.04) + 0.1 × (−0.04) = 0.792.

Each iteration updates all states using the previous iteration’s values. This synchronous update is why the goal’s influence expands through the grid over successive steps. The Bellman panel below the room previews the next update; the floor shows the values already stored.

For a finite MDP with bounded rewards and γ below one, these updates converge to the optimal value function. Our numerical solver stops when the largest change in an iteration falls below 0.0001. That is an approximation threshold, not a claim of exact arithmetic. Tied actions use the order up, right, down, left.

Three experiments to build intuition

1. Change uncertainty. Choose “A slippery world,” solve, and inspect cells near the pit. Lower the slip probability and solve again. Compare the expected returns for individual actions to understand any changes in the preferred direction.
2. Change urgency. Choose “Every step costs.” A larger per-step penalty makes extra movement more expensive. Compare the resulting policy with the default world, especially where avoiding the pit requires a detour.
3. Change the horizon. Choose “Short-term thinking,” then compare it with γ = 0.99. The world’s transitions stay the same, but the relative importance of distant rewards changes.

Changing a model parameter resets the values so the new calculation starts cleanly. Agent runs sample outcomes from the transition model while following the current greedy policy. An individual run can still end in the pit: maximizing expected return does not guarantee success on every episode. Runs stop at a terminal state or after 100 steps.

How this connects to robot learning

This example performs planning with a known model: the transition probabilities and rewards are supplied. Reinforcement learning also studies how an agent can learn useful behavior from experience when that model is unavailable.

The shared question is how present actions affect future outcomes. The grid lets us inspect that question exactly enough to see why rewards, uncertainty, and the choice of state representation matter. Real robot systems add continuous states, complex dynamics, partial observations, and safety constraints, but the distinction between a chosen action and its possible consequences remains central.