RL Essentials — From Network to Learning Agent

← Neural Foundations 2 · Course home

Time

Reading: ~25 min · Quick reward tweak: ~20 min · Training can run in the background

What you'll be able to do after this unit

Explain what reinforcement learning adds to the network you just built
Name the loop pieces: observation, action, reward, next observation
Describe episodes, return, discounting, policy, and exploration
Point to where Godot and Python each sit during training
Change one reward and predict how the learning curve should respond

Prerequisites

Unit 0 complete — Conda, Godot, and a successful BallChase run
Neural Foundations 1–2 complete — you have seen inputs, weights, loss, gradients, and inference
Basic terminal comfort

Three ways to see your AI

Godot editor (the agent moving live) · TensorBoard (rollout/ep_rew_mean changing) · Code (the reward line you edit)

You have already built a small network that turns numbers into decisions. RL adds one missing piece: the network no longer learns from correct answers. It learns from actions, consequences, and reward.

1 · What reinforcement learning adds

Reinforcement Learning (RL) teaches software to make good decisions by letting an agent act inside an environment and scoring what happens with a reward.

Supervised learning says: "this input should produce this target." RL says: "try an action, observe what happened, and use the reward to make better decisions next time."

That difference matters for games. A designer may not know the perfect action in every position, but they can often describe what good behavior earns:

get closer to the goal;
avoid hazards;
finish quickly;
stay alive;
collect useful objects.

The reward is not the final behavior. It is the training signal the policy uses to discover behavior.

The reward hypothesis

RL rests on one bold idea: goals can be expressed as maximizing expected cumulative reward. "Land softly" becomes numbers for safe speed, upright angle, leg contact, and not crashing.

2 · Observation → action → reward → next observation

At each training step, the same loop repeats:

Godot sends the current observation to Python.
The policy chooses an action.
Godot applies that action in the scene.
Godot returns a reward and the next observation.

Observation versus state

State means the full truth of the world.
Observation means the numbers the agent actually receives.

In most games the agent receives observations, not the full state. A lander may know its velocity and angle but not every internal physics value. A racer may know ray distances and heading error but not the whole track map.

Action space

Type	Meaning	Example
Discrete	Choose from a fixed list	fire left thruster, main thruster, or nothing
Continuous	Choose values from a range	steering and throttle between -1 and 1

The action space decides what shape the policy output must have.

3 · Episodes, return, and discounting

An episode is one attempt from reset to a terminal condition. A lander episode ends when it lands, crashes, or times out. A racer episode might end after a lap, a collision, or an immobility timeout.

The reward is one step of feedback. The return is the sum of future rewards the agent is trying to maximize.

Future rewards are usually discounted by a value called gamma (γ):

γ close to 1 means the agent cares strongly about later outcomes;
lower γ means the agent focuses more on immediate reward.

The mouse, cheese, and cat intuition

Cheese near the cat may be valuable, but it is risky and far away. Discounting captures the idea that a future reward you might never reach is worth less than a reward you can reliably get now.

Everything in this course starts as an episodic task. That makes learning curves easier to read because each run has a clear beginning and end.

4 · The policy is the network you built

The policy is the agent's decision function. It maps an observation to an action, or to a probability distribution over actions.

In Neural Foundations 2, your network learned:

inputs → hidden activations → outputs

In RL, those outputs become actions:

observation → policy network → action

Training changes the policy weights. Inference only runs the forward pass. That same split will matter later when a policy is trained in Python and exported to run inside Godot.

Policy in one sentence

A policy is the trained behavior of the agent, stored as network weights.

5 · Exploration in one picture

Before an agent can use a good behavior, it must discover one. That is exploration.

The trade-off is simple:

exploit actions that already look good;
explore actions that might reveal something better.

Early in training, random-looking movement is normal. The policy is collecting experience. Later, movement should become more consistent as reward pushes the network toward better actions.

Do not judge a policy from one early episode

A useful policy often looks foolish at first. Watch trends across many episodes, especially the average return curve, not one lucky or unlucky run.

You will study detailed exploration mechanisms in the deep dive after Foundations 3. For now, the practical question is: does the agent keep trying enough actions to find rewarded behavior?

6 · Godot and Python during training

Godot RL Agents runs two programs side by side:

Godot is the environment: physics, observations, actions, rewards, resets;
Python is the trainer: it runs the RL algorithm and updates the policy.

RL concept	Where it lives in Godot RL
Environment	Your Godot scene
Observation	`get_obs()` in the AIController script
Action space	`get_action_space()` in the AIController script
Reward	Reward variables updated by your game logic
Episode end	`done`, `needs_reset`, or equivalent reset flags
Policy	The neural network trained in Python

7 · Quick win — change one reward

Your first ownership moment

In Unit 0 you ran BallChase. Here you change the reward signal and watch the training behavior respond.

Clone or open BallChase in Godot.
Find the script that updates reward for the agent.
Change one term, such as doubling the reward for getting closer to the ball or adding a small penalty per step.
Predict the result before training: faster chasing, more wandering, shorter episodes, or slower learning.

Run a short visual training session:

conda activate godot_env
gdrl --experiment_name=unit1-reward-tweak --viz \
  --save_model_path=ballchase_brain \
  --onnx_export_path=ballchase_brain.onnx

Godot — open the BallChase training scene, press F6 (Play Scene).

Watch three views:

Godot: does movement match your reward change?
TensorBoard: does ep_rew_mean trend upward after enough episodes?
Code: can you explain how the edited reward changes the loop?

8 · Done when

You are ready to continue when you can:

explain observation → action → reward → next observation without notes;
describe why the policy is the same kind of network you built in Foundations 2;
point to Godot's role and Python's role during training;
make one BallChase reward edit and predict the likely symptom;
read a reward curve as a trend, not as one episode.

Training stalled?

Check in order: reward sign and scale, sparse rewards, observation bugs, resets, and whether the run simply needs more episodes.

9 · Stretch goals

Write a random-policy loop for CartPole-v1 and print the episode return.
Run two BallChase reward tweaks: one stronger reward and one weaker reward. Compare the learning curves.
Sketch the observation vector and action space for a game idea of your own.

import gymnasium as gym

env = gym.make("CartPole-v1")
obs, _ = env.reset()
total_reward = 0.0

for _ in range(500):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, _ = env.step(action)
    total_reward += reward
    if terminated or truncated:
        break

print(f"Episode return: {total_reward}")
env.close()

What's next

You now have the operational RL vocabulary: observations, actions, rewards, episodes, returns, policies, exploration, and the Godot/Python training loop. Next, Foundations 3 turns those pieces into a small reward-learning project so you can watch a policy improve from trajectories.

Self-check before you move on

What are the four parts of the RL loop?
What is the difference between an observation and a state?
What does the discount rate γ control?
What is a policy?
Why does an agent need exploration?
What does Python do during Godot RL training?
What does Godot do during Godot RL training?

Self-check answers

Observation → action → reward → next observation.
A state is the full world description; an observation is the part the agent receives.
γ controls how much future rewards count compared with immediate rewards.
A policy is the decision function, usually a neural network, that maps observations to actions.
Exploration lets the agent discover behaviors it does not already know are useful.
Python runs the training algorithm and updates the policy weights.
Godot simulates the environment, applies actions, computes rewards, and resets episodes.

← Neural Foundations 2 · Course home · → Neural Foundations 3