What reward hacking taught me about RL
What happens when a reward looks sensible but can be farmed?
Context
I trained PPO landers in Gymnasium’s LunarLander-v3 for 1M steps each, with my own reward wrappers, and scored every version on the game’s real reward over 50 deterministic landings.
What I tried
- A bonus of +1 for every frame the lander was upright and slow. It reads like a nudge towards a careful landing.
What failed
- The lander learned to hover until the 1,000-frame limit instead of landing: 0 of 50 solved, 32 timeouts, mean score −41.7.
What changed
- Replaced the bonus with potential-based shaping on distance, speed and angle, which pays for getting closer to landing rather than for staying in a good state, and kept a small time fee.
Evidence
- Baseline: 18/50 solved, mean 142.5.
- Time fee only: 37/50, mean 195.8.
- Hackable bonus: 0/50, mean −41.7.
- Potential-based fix: 29/50, mean 178.1.
- One training run per version, so the ordering is evidence, not proof.
What I learned
- The agent did exactly what the reward said. The bug was in the reward, not the agent.
- Pay for improvement, not for being in a state, and there is nothing left to farm.
- Scoring every version on the game’s own reward is what made the failure visible.