LunarLander RL Demo

Author: Dr. Nishant Upadhyay
Contact: nishantup@gmail.com

What this is

An interactive comparison of three reinforcement learning approaches on the classic LunarLander-v3 control task from Gymnasium:

• Random policy — the uniform-random baseline (no learning)
• Monte Carlo control — learns from complete episodes by averaging returns
• Q-Learning (TD(0)) — learns from individual transitions using bootstrapped targets

All three use a tabular representation over a discretized state space of 1,048,576 buckets. This lets us study classical RL mechanics without the black-box complexity of a neural network.

Checkpoint statistics

"Solved" = average reward ≥ 200 over 100 episodes. None of the tabular methods hit that bar on LunarLander — the continuous physics really demands function approximation (DQN). That's by design: the point is to understand where tabular methods start to break down.

Hyperparameters

• Discount factor γ = 0.99
• Q-Learning step size α = 0.1
• Exploration: ε starts at 1.0, decays by factor 0.9995 per episode, floors at 0.05
• Max steps per episode: 1000
• Discretization bins: [8, 8, 8, 8, 8, 8, 2, 2]

How to use this demo

1. Watch an Agent — pick a method and a seed; watch a full episode.
2. Compare All Three — run all three on the same starting seed and see how dramatically outcomes can diverge.
3. Statistics — run many evaluation episodes to see the reward distribution.
4. Training Curves — inspect the pre-rendered learning curves from the full training runs.
5. Inside the Q-Table — peek at what the trained policy actually learned at specific meaningful states.
6. Live Training Demo — watch Q-Learning learn in real time on a capped budget (few minutes on CPU).

The reward function

Each step returns a sum of:

• Shaping rewards (small, every step) — penalties for distance from landing pad, horizontal/vertical speed, and tilt angle
• Leg contact bonus — +10 once per leg when it touches the ground
• Fuel cost — −0.3 for main engine, −0.03 per side engine fire, every step
• Terminal — +100 for a safe landing, −100 for a crash

The "Reward breakdown" chart on the first tab decomposes these for any episode you run.

Code

The source code and training script are designed to be readable end-to-end. Key modules:

• src/environment.py — state discretization, reward components
• src/agents.py — Random / MC / Q-Learning agent wrappers
• src/evaluation.py — episode runner, video recording
• src/training.py — live training generator
• src/visualization.py — all plot helpers
• scripts/train_and_export.py — reproduce the trained Q-tables
• scripts/generate_plots.py — re-render the training-curve PNGs

Acknowledgements

Built with Gymnasium, NumPy, Matplotlib, and Gradio. Inspired by Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed.

— Dr. Nishant Upadhyay (nishantup@gmail.com)