Godot RL Course
Learn deep reinforcement learning by building and training agents inside real Godot game environments.
Before you start
Who this course is for. Developers who want to train game AI with reinforcement learning. You need basic Python (functions, loops, running scripts) and comfort with a terminal. No prior Godot, machine-learning, or RL experience is required — neurons, networks, and the RL loop are built from scratch in Phase 1. The math stays at high-school algebra, and every formula is walked through with concrete numbers first.
What you need. A desktop or laptop running macOS, Windows, or Linux. A GPU helps but is not required — early units train in minutes on CPU, and each unit's time box lists CPU and GPU estimates. See the Hardware Setup Guide before buying anything.
Time commitment. Every unit opens with a time estimate. Unit 0 plus your first neuron fits in one evening (~2½–3 h); most units take 1–3 hours of attention, with longer training runs happening in the background.
Where to start. Complete Setup once — clone the course repo, install Godot and the Python environment. Then begin with Unit 0, your first training run.
What you'll build
| Phase | Content | What you learn |
|---|---|---|
| Phase 1 — Foundations | Setup · neural networks · RL loop · reward learning · deep dive · first custom env · reward design | Neurons to policies, how RL works, how to design rewards |
| Phase 2 — Value-Based | Q-Learning · DQN · curiosity | Bellman equation, Q-tables, DQN, sparse reward exploration |
| Phase 3 — Policy-Based | REINFORCE · Actor-Critic · PPO · SAC · Apply It · PPO from scratch (CleanRL) | Policy gradient theorem, PPO internals, continuous control |
| Phase 4 — Scale & Complexity | Parallel · 3D · Multi-agent · Memory | Real-world training at scale |
| Phase 5 — Beyond Reward | Multi-task RL · Imitation learning · RLHF · Offline RL · Decision Transformer · ONNX/WASM · Capstone project | Alternative learning signals, generalist policies, shipping |
| Phase 6 — Robotics | Robot sensors · Locomotion · Diffusion Policy · HER · sim-to-real · Safe RL | Robot observation/action design, goal-conditioned RL, deploying policies to hardware safely |
| Guides | Debugging · Advanced Evaluation · PBT · World Models | Systematic diagnosis, evaluation, hyperparameter AutoML, model-based RL |
Three ways to see your AI (every unit)
| Channel | What it shows |
|---|---|
| Godot | Agent behavior in the world |
| TensorBoard | Learning curves (tensorboard --logdir=logs) |
| AIController source | Observations, actions, and rewards you changed |
Units
Phase 1 — Foundations
- Unit 0 — Setup & First Run
- Math Foundations 1 — Vectors and Matrices (optional refresher)
- Math Foundations 2 — Probability and Expectation (optional refresher)
- Neural Foundations 1 — One Neuron
- Neural Foundations 2 — Tiny Networks
- RL Essentials
- Neural Foundations 3 — Learn from Reward
- RL Foundations Deep Dive
- Reward Engineering
- Unit 2 — Build Your First Env
Phase 2 — Value-Based Methods
Phase 3 — Policy-Based Methods
- Policy Gradients & REINFORCE
- Actor-Critic
- PPO Deep Dive
- PPO in Practice (JumperHard)
- SAC — Soft Actor-Critic
- Apply It — SAC vs PPO on JumperHard
- PPO From Scratch (CleanRL)
Phase 4 — Scale & Complexity
- Unit 5 — Parallel Training
- Unit 6 — Continuous 3D
- Visual Observations
- Unit 7 — Multi-Agent
- Unit 8 — Memory & POMDPs
- Self-Play
- Hierarchical RL
Phase 5 — Beyond Reward
- Multi-Task RL
- Unit 9 — Imitation Learning
- RLHF & Preference Learning
- Offline RL
- Decision Transformer
- Unit 10 — Ship Your Brain
- Capstone Project
Phase 6 — Robotics
- Robot Observations & Sensors
- Locomotion Agents (Walker / Crawler / Worm)
- Diffusion Policy
- Goal-Conditioned RL & HER
- Sim-to-Real Transfer
- Safe RL / Constrained MDPs
Guides
- Debugging RL Training
- Advanced Evaluation
- Experiment Tracking (W&B / MLflow)
- GPU-Accelerated Environments
- Population-Based Training
- World Models / DreamerV3
- Foundation Models for Control (VLA)
After this course
Follow-on: model alignment (separate course)
This course ends at ONNX vector policies. A planned sequel covers language-model alignment (SFT, preferences, RLHF/DPO). Unit 9 imitation learning is the on-ramp. Optional — Units 0–10 are complete on their own.
Alignment course repository: coming soon.