Reward Hacking in LunarLander
A learning project after playing LunarLander: I trained an RL agent to land the rocket, rewarded it for staying afloat, and it learned to hover instead of land. That is how I met reward hacking.
- Role
- Solo learning project
- Status
- SHIPPED
- Stack
- Python, Gymnasium, Stable-Baselines3, PyTorch
Real episodes from the trained models. One training run per variant, so these are evidence, not proof.
Step 1
Baseline PPO
Plain PPO for 1M steps. It lands, but slowly and not reliably: 7 crashes in 50 tries.
mean 142.518/50 solvedStep 2
Time fee
Charge 0.1 per frame. The agent stops dawdling and lands cleanly twice as often.
mean 195.837/50 solvedStep 3
Hackable reward
Pay +1 for every upright, slow frame. The agent learns that hovering forever pays better than landing: 32 of 50 episodes time out in the air.
mean −41.70/50 solvedStep 4
Potential-based fix
Reward progress toward the pad instead of time spent near it. Hovering no longer pays, and landings come back.
mean 178.129/50 solved
The problem
If an agent is rewarded for something that only looks like progress, will it learn to game the score instead of doing the task, and can a better reward fix it?
After playing the LunarLander game I wanted to learn reinforcement learning by training an agent to land the rocket in the target zone. One of my reward ideas paid it for every frame it stayed upright and steady in the air. It learned to hover until time ran out instead of landing: reward hacking. Measuring that and fixing it became the project.
Constraints
- Gymnasium LunarLander-v3 with PPO from Stable-Baselines3, 1M training steps per version.
- Every version scored on the game’s own reward, with my reward wrapper switched off, over 50 deterministic test landings.
The system
- 01Trained a plain PPO lander as the reference.
- 02Added a small fee of 0.1 for every frame, to push it to land sooner.
- 03Added a bonus of +1 for every frame spent upright and slow: a reward that sounds sensible but can be farmed.
- 04Replaced the bonus with potential-based shaping on distance, speed and angle, which pays for getting closer to landing rather than for staying in a good state, and kept the time fee.
Engineering decisions
- Judge every model on the game’s own score
- A shaped reward can go up while the real task gets worse, so the wrapper is off for every evaluation.
- Potential-based shaping for the fix
- It rewards improvement, not being in a state, so there is nothing to collect by hovering.
- 50 test landings per model
- A single episode is mostly luck; the same 50 deterministic landings make the four versions comparable.
Evaluation
- Every version scored on the game’s own reward, with my wrapper switched off.
- 50 deterministic test landings per model: mean score, solved (200+), crashes and timeouts.
What changed
- Replaced the per-frame bonus with potential-based shaping on distance, speed and angle, and kept the time fee.
Result
- +1 per upright frame, Solved
- 0 / 50
- +1 per upright frame, Mean score
- −41.7
- +1 per upright frame, Timeouts
- 32
- Potential-based fix, Solved
- 29 / 50
- Potential-based fix, Mean score
- 178.1
- Potential-based fix, Timeouts
- 2
| Model | Mean score | Solved (200+) | Crashes | Timeouts |
|---|---|---|---|---|
| Baseline PPO | 142.5 | 18 / 50 | 7 | 2 |
| Time fee, 0.1 per frame | 195.8 | 37 / 50 | 4 | 2 |
| +1 per upright frame (hackable) | −41.7 | 0 / 50 | 15 | 32 |
| Potential-based fix | 178.1 | 29 / 50 | 5 | 2 |
- Baseline: mean 142.5, 18/50 solved.
- Time fee: mean 195.8, 37/50 solved.
- Hackable reward: mean −41.7, 0/50 solved, 32/50 episodes timed out hovering.
- Potential-based fix: mean 178.1, 29/50 solved.
Limitations
- One training run per version, so the ordering is evidence, not proof.
- The fix removed the hovering exploit but still landed less often than the plain time fee (29/50 against 37/50).
What I learned
- The agent optimised the reward exactly as written. The hovering came from the reward, not the agent.
- Pay for improvement, not for being in a state.
Links
Want to talk through how I built this?
I'm happy to walk through the decisions, what failed and what I'd change.