Imitation learning: behaviour cloning, error accumulation, DAgger and inverse RL

Defining a reward that describes a task well is often harder than the task itself: how do you write as a number “drive like an expert”, or “fold the shirt neatly”? It is much easier to show how it is done. Imitation learning learns a policy from demonstrations by an expert: a driver, an operator remotely guiding a robot, a program that is already good. It is the basis of the first autonomous driving systems, of many manipulator robots and also of the first training stage of language models. This lesson presents behaviour cloning, proves why its errors accumulate, presents DAgger, which solves the problem, and outlines inverse reward learning. Experiments on CartPole show everything with numbers.

Behaviour cloning

The simplest idea: collect (state, expert action) pairs and train a network to predict the action from the state, as in a classification problem (discrete actions) or a regression problem (continuous actions). No reward, no exploration: it is supervised learning. In 1989 ALVINN drove a van by learning the steering angle from camera images and a driver’s manoeuvres; in 2016 NVIDIA did the same with a convolutional network (PilotNet).

Why errors accumulate

Behaviour cloning has a deep flaw: the network is trained on the states visited by the expert, but is used on the states visited by itself. A small error is enough to bring the agent to a state slightly different from those seen; there the network makes larger errors, and the state drifts further. Errors compound.

Theorem (Ross and Bagnell, 2010). If the cloned policy chooses the wrong action with probability at most on the expert’s states, and an error can take it outside the expert’s distribution for the rest of the episode, the expected cost over a horizon of steps can grow as , instead of .

Proof idea. As long as the agent has not yet made a mistake, it is in the expert’s states and errs with probability per step. The probability of having already erred by step is about (for small ), and after the first error, in the worst case, a cost is paid at every following step. The expected cost is therefore . If instead the policy were trained on the states that it itself visits, the error per step would stay and the cost . ∎

DAgger: asking the expert where things went wrong

DAgger (Dataset Aggregation, Ross, Gordon and Bagnell, 2011) cures the problem at the root: let the learned policy drive, record the states it visits, ask the expert which action it would have chosen in those states, add the new pairs to the data and retrain. Repeating, the data cover exactly the states where the agent ends up, including those where it needs to “recover”. The DAgger theorem guarantees a cost growing as . The price: the expert must be available during training (a program, a simulator with an optimal controller, or a person correcting afterwards).

The experiments on CartPole

The expert is the PPO of the previous lesson, which keeps the pole up for all 500 steps. The imitating policy is a small network (one hidden layer of 32 neurons), trained to classify the action and evaluated over 20 episodes. With so little data the result depends a lot on chance (which states appear in the demonstrations, how the network starts), so each configuration is repeated with 10 random seeds and mean and standard deviation are reported.

For the same amount of data DAgger is always better, and with 80-100 steps it achieves what takes behaviour cloning about twice as many. The reason is the one in the theorem: the expert’s demonstrations show almost only the pole close to vertical, while DAgger adds precisely the “wrong” states the agent ends up in, with the right action to recover. Standard deviations are large: with a lucky seed even behaviour cloning can reach 500, with an unlucky one stay below 100. It is a general warning: in RL and imitation a single experiment proves nothing.

import numpy as np
import torch, torch.nn as nn
import gymnasium

expert = nn.Sequential(nn.Linear(4, 64), nn.Tanh(), nn.Linear(64, 64), nn.Tanh(), nn.Linear(64, 2))
expert.load_state_dict(torch.load("expert.pt"))               # saved by the code of the PPO lesson
expert_action = lambda s: int(expert(torch.tensor(s, dtype=torch.float32)).argmax())

def evaluate(policy, episodes=20):
    env, scores = gymnasium.make("CartPole-v1"), []
    for e in range(episodes):
        o, _ = env.reset(seed=1000 + e)
        tot, done = 0.0, False
        while not done:
            o, r, te, tr, _ = env.step(policy(o))
            tot += r; done = te or tr
        scores.append(tot)
    return float(np.mean(scores))

def collect(policy, steps, seed):
    """States visited following policy, labelled with the expert's action."""
    env, S, Y = gymnasium.make("CartPole-v1"), [], []
    o, _ = env.reset(seed=seed)
    for _ in range(steps):
        S.append(o); Y.append(expert_action(o))
        o, _, te, tr, _ = env.step(policy(o))
        if te or tr:
            o, _ = env.reset()
    return S, Y

def train(S, Y, seed):
    torch.manual_seed(seed)
    net = nn.Sequential(nn.Linear(4, 32), nn.Tanh(), nn.Linear(32, 2))
    opt = torch.optim.Adam(net.parameters(), lr=1e-2)
    X, T = torch.tensor(np.array(S), dtype=torch.float32), torch.tensor(Y)
    for _ in range(300):                                       # classification of the action
        loss = nn.functional.cross_entropy(net(X), T)
        opt.zero_grad(); loss.backward(); opt.step()
    return lambda s: int(net(torch.tensor(s, dtype=torch.float32)).argmax())

print("expert:", evaluate(expert_action))
bc = {n: [] for n in (20, 40, 60, 80, 100, 200)}
dag = {n: [] for n in (20, 40, 60, 80, 100)}
for seed in range(10):
    for n in bc:                                               # behaviour cloning with n steps
        bc[n].append(evaluate(train(*collect(expert_action, n, 100 * seed), seed)))
    S, Y = collect(expert_action, 20, 100 * seed)              # DAgger: start from 20 steps
    policy = train(S, Y, seed)
    dag[20].append(evaluate(policy))
    for it in range(1, 5):
        S2, Y2 = collect(policy, 20, 100 * seed + it)          # states visited by the learned policy
        S += S2; Y += Y2
        policy = train(S, Y, seed)
        dag[20 * (it + 1)].append(evaluate(policy))
for n in bc:
    row = f"{n:4d} steps: behaviour cloning {np.mean(bc[n]):6.1f} ± {np.std(bc[n]):5.1f}"
    if n in dag:
        row += f", DAgger {np.mean(dag[n]):6.1f} ± {np.std(dag[n]):5.1f}"
    print(row)

Beyond direct imitation

• Inverse reinforcement learning (inverse RL): instead of copying actions, one looks for the reward the expert seems to maximise, and then optimises it with RL. It is more robust outside the demonstrations, because the reward is valid also in never-seen states.
• GAIL (generative adversarial imitation learning, 2016): a discriminator learns to distinguish the expert’s trajectories from the agent’s, and the agent is trained with RL to “fool” it, as in generative adversarial networks.
• Imitation, then reinforcement: in practice one often starts with behaviour cloning (fast, stable) and improves with RL. This is the scheme of AlphaGo (first the masters’ games, then self-play) and of language models (first supervised training on answers written by people, then RL with human feedback).
• Modern robotics: diffusion policies and vision-language-action models learn from thousands of teleoperated demonstrations; error accumulation is mitigated by predicting chunks of future actions instead of a single one.