RL applications: games, robotics, autonomous driving, RLHF and DPO for language models

Reinforcement learning left the laboratories in a few years: it has beaten the champions of Go and of complex video games, it makes robots walk and fly, it controls plants, and today it is an essential part of training large language models. This lesson reviews the most important applications, says honestly where RL works and where it does not, and proves the formulas of RLHF (reinforcement learning from human feedback) and of DPO, the method that simplified its last stage.

Games: where it all began

• AlphaGo (DeepMind) beat Lee Sedol 4 to 1 in March 2016: neural networks trained first by imitation on the masters’ games, then by self-play, together with a Monte Carlo tree search. AlphaZero (2017) removed imitation: only the rules and self-play, and it surpassed the best chess, shogi and Go programs.
• OpenAI Five beat the Dota 2 world champions in 2019, and AlphaStar reached Grandmaster level in StarCraft II: games with incomplete information, thousands of actions and long matches, tackled with PPO and variants on thousands of processors.

Games are the ideal case: the simulator is the game itself, perfect and very fast, and the reward (winning) is clear.

Robotics and control

• Locomotion: quadruped robots (such as ETH Zurich’s ANYmal) and humanoids walk on difficult terrain with policies trained in simulation with thousands of parallel copies and domain randomisation, then transferred to the real robot without further training.
• Drones: in 2023 the University of Zurich’s Swift system, trained with RL in simulation, beat human drone racing champions (a result published in Nature).
• Plants: DeepMind used RL for the cooling of Google’s data centres, reporting a reduction of up to 40 % in the energy used for cooling, and in 2022, with EPFL, to control the shape of the plasma in the TCV tokamak by acting on 19 magnetic coils.

The common problem is transfer from simulation to reality, discussed in the previous lesson, and safety: a learned controller is hard to certify, and it is almost always paired with a classic controller or a safety filter that prevents dangerous actions.

Autonomous driving and ADAS

In cars RL is used with caution. Production ADAS functions (emergency braking, lane keeping, adaptive cruise control) are based on classic control, like that of the Automation Systems Design course, and on supervised neural networks for perception. RL is studied mainly for decisions: when to change lane, how to merge into a flow, how to behave at junctions with other road users. “End-to-end” driving, from camera to steering, relies mostly on imitation of millions of kilometres of human driving, with RL used to correct rare and dangerous behaviours in simulation. The reason is safety: in a car you cannot explore, and validation requires demonstrating the behaviour in every situation.

Language models: RLHF

A pre-trained language model can continue a text, but does not necessarily know how to be helpful, honest and harmless. RLHF (reinforcement learning from human feedback), made famous by InstructGPT (OpenAI, 2022), trains it in three stages:

1. Imitation (supervised fine-tuning): the model learns from example answers written by people. It is behaviour cloning.

2. Reward model: for the same question two answers are generated, and a person chooses the better one over the other . A network learns to give a score consistent with the preferences, with the Bradley-Terry model: , with the sigmoid; is minimised.

3. Reinforcement: the model (the policy , which “acts” by choosing words) is optimised with PPO to maximise the reward, but without moving too far from the starting model :

The Kullback-Leibler term prevents the model from finding strange answers that “fool” the reward model (which is only an approximation of the true preferences). In the InstructGPT work, the answers of a 1.3-billion-parameter model trained this way were preferred by people to those of the 175-billion GPT-3.

DPO: preferences without PPO

PPO on a model with billions of parameters is expensive and delicate. DPO (direct preference optimization, Rafailov et al., 2023) shows that the third stage can be done with a simple supervised loss.

Theorem. The policy that maximises is

Proof. Rewrite the objective by collecting :

does not depend on , and the KL divergence is non-negative and zero only if (Gibbs’ inequality): the maximum is . ∎

Inverting: . The reward is determined by the optimal policy up to a constant, which in the Bradley-Terry model cancels out (only the difference matters). Substituting gives a loss that trains the policy directly on preference pairs, without a reward model and without RL:

Numerical check. With four possible answers, , rewards and , the formula gives ; gradient descent on the objective reaches the same values to the fourth decimal place, with objective 2.046 against 1.3 for the starting policy, and from the reward differences are reconstructed exactly.

import numpy as np

pi_ref = np.array([0.4, 0.3, 0.2, 0.1])                    # reference policy over four answers
r = np.array([1.0, 2.0, 0.0, 3.0])                         # rewards
beta = 0.5

pi_star = pi_ref * np.exp(r / beta)                        # closed form
pi_star /= pi_star.sum()

z = np.log(pi_ref).copy()                                  # gradient descent on the logits
for _ in range(20000):
    p = np.exp(z - z.max()); p /= p.sum()
    f = r - beta * (np.log(p) - np.log(pi_ref))
    z += 0.5 * p * (f - (p * f).sum())
p = np.exp(z - z.max()); p /= p.sum()

objective = lambda q: (q * r).sum() - beta * (q * np.log(q / pi_ref)).sum()
print("closed form:", np.round(pi_star, 4), " gradient:", np.round(p, 4))
print("objective:", round(objective(pi_star), 4), "against", round(objective(pi_ref), 4))
rr = beta * np.log(pi_star / pi_ref)                       # reward recovered from the policy
print("reward differences:", np.round(rr - rr[0], 4))

Reasoning with verifiable rewards

Since 2024 RL has also been used to teach models to reason: on mathematics and programming problems the answer can be verified automatically (the result is right, the program passes the tests), and verification becomes the reward. The model generates many attempts, the correct ones are reinforced. DeepSeek-R1 (2025) showed that long chains of reasoning, with self-corrections, emerge this way, using a critic-free variant of PPO (GRPO, which estimates the advantage by comparing attempts within the same group).

When RL goes wrong: the wrong reward

An agent maximises the reward it is given, not the one intended. In a famous OpenAI example (2016), an agent trained on a boat racing game, rewarded for hitting targets, discovered that circling while hitting the same targets over and over paid more than finishing the race. In real systems this phenomenon (reward hacking, specification gaming) is the main concern: it is fought with carefully designed rewards, explicit constraints, limits on deviation from known behaviour (like the KL term of RLHF) and independent checks of the final behaviour.

Summary