DECISION LAB / 01

Small room. Big decisions.

A robot, a goal, and a little uncertainty. See how a policy takes shape.

▦The room4 × 3 grid

Scenario

Preparing the room… You can already explore the grid below.

Robot Goal +1 Pit −1 Obstacle

Select a floor tile to inspect its decision

ROBOT JOURNEY

Starts at (3, 1) · follows the current greedy policy

┄ Requested action━ Sampled movement

Steps 00Total reward 0.00Discounted return 0.00

ITERATION00

MAX CHANGE—

Ready

01 / CHANGE THE WORLD

The rules of the room

⚙

Discount γ0.90How much the future mattersSlip probability20%Chance of moving sidewaysStep reward−0.04Reward for a nonterminal arrivalGoal reward+1.00Received once, on entryPit reward−1.00Received once, on entry

Changing a rule resets the values. The robot stays in place when a move meets an obstacle or boundary.

02 / LOOK INSIDE THE DECISION

A Bellman update, unpacked

Q_k(s, a) = Σ P(s′ | s, a) [R + γV_k(s′)]

ProbabilityReward nowValue later

V_k+1(s) = max_a Q_k(s, a)−0.040

↳

Start with Step once. Watch information about the goal travel through the room.

THE MODEL BEHIND THE ROOM

Five pieces.
One decision process.

S · A

Where. And what next.

A state is a floor tile. An action is a direction. The current state holds the information needed to predict the next outcome.

P · R

A little uncertainty.

A chosen action can slip sideways. Every outcome has a probability and a reward. Planning accounts for all of them.

γ · V

Think beyond this move.

Value estimates future return. Each iteration updates every state from the previous values; the best actions form a policy.