From imitation to reinforcement learning: how Chinese driving systems are trained

For the last few years the big driver assistance systems in China have been trained mostly by imitation: show a neural network millions of clips of people driving well and teach it to do the same. It works surprisingly well, up to a point. In 2025 and 2026 almost every major player started talking about reinforcement learning (RL) and world models. This article explains, without heavy maths, why imitation is not enough, what RL adds, and how companies like Li Auto, XPeng, Momenta and BYD say they use it.

The limit of learning by imitation

Imitation learning (also called behaviour cloning) treats driving as a supervised problem: input, the sensor data; target, what the human did next. The network learns the average behaviour of good drivers. Two problems show up quickly.

• Small errors pile up. The network has only seen what good drivers do, which means it has rarely seen a car drifting toward the lane edge, because good drivers do not end up there. When its own small mistakes bring it into such a state, it has no example of how to recover. Researchers call this covariate shift.
• Average is not always right. At an unprotected left turn, some drivers go and some wait. Averaging the two can produce a hesitant half move that no human would make. Learning “what people did” is not the same as learning “what is safe and efficient”.

More data helps, but the rare situations, the ones that cause accidents, are by definition rare in the data.

What reinforcement learning changes

In RL the network (the policy) is not told what to do. It acts, sees the consequences, and receives a reward: positive for reaching the goal smoothly, negative for a collision, for leaving the lane, for braking hard, for running a red light. Over many attempts it learns behaviour that maximizes the total reward. Two consequences matter here:

• the policy experiences the results of its own mistakes, so it learns to recover from them;
• it can, in principle, become better than the drivers in the data, because it optimizes a goal instead of copying an average.

The obvious catch: you cannot let a car learn by crashing on real roads. RL needs an environment where millions of attempts are cheap and harmless. That environment is a simulator, and this is where world models come in.

World models: a simulator learned from video

Traditional simulators are built like video games: hand-modelled roads, cars and pedestrians. They are useful, but the gap with real camera images is large. A world model takes another route: a generative network, trained on huge amounts of real driving video, that predicts what the cameras will see next given what they saw so far and what the car decides to do. Steer left in the simulation and the generated images show the car moving left, with the other road users reacting.

XPeng published a technical report on its world model, X-World, in April 2026 (XPeng). It is a video diffusion model that generates the view of seven cameras at once, conditioned on the planned actions of the car, on other traffic participants and on the road layout. XPeng says it uses it in three ways: to evaluate its VLA 2.0 driving model in closed loop (collision rate, progress, comfort), as an environment for online reinforcement learning on hard cases such as pedestrians stepping out at intersections or hesitation during lane changes in traffic jams, and to generate rare scenarios that real data lacks. The company reports going from about 30,000 simulated scenarios to more than 500,000, with daily simulation equivalent to 30 million km of driving.

Li Auto describes a similar pipeline for its MindVLA model: a cloud world model that combines 3D reconstruction of real scenes with generative completion of the views, used for large-scale closed-loop RL after the model has been pre-trained and fine-tuned on real driving (Gasgoo). The reconstruction part often relies on 3D Gaussian splatting, a technique that turns recorded drives into scenes that can be re-rendered from new viewpoints. A paper from Horizon Robotics and university researchers, RAD, showed in 2025 that a driving policy trained with RL in this kind of reconstructed environment had a much lower collision rate than the same policy trained by imitation alone.

What the other players say

• Momenta calls its current model R6 “flywheel” and presents it as based on reinforcement learning; it is in production on models from GAC Toyota, Chery and Buick, among others (Wikipedia).
• BYD launched God’s Eye 5.0 in January 2026 with a focus on reinforcement learning and closed-loop end to end control, and says its fleet collects more than 160 million km of assisted driving data per day (CarNewsChina).
• Huawei uses a world model as the training ground for its ADS 4, but has openly criticized the VLA approach of XPeng and Li Auto, preferring a “world action” model without the language step (ChinaEVHome).

VLA stands for vision, language, action: a model built on a large language model backbone, which can reason about the scene in words (“the bus is stopping, passengers may cross”) before producing the trajectory. Supporters say this helps with unusual situations and makes the system easier to inspect. Critics, Huawei first, say the language step adds latency and computing cost without making the car drive better. Both camps agree on one thing: the final improvement comes from closed-loop training, not from more imitation.

The hard part is the reward

Anyone who has tried RL on a toy problem knows the weak spot. The policy optimizes exactly what you reward, not what you meant. Reward only progress and it learns to cut in aggressively; punish every brake and it becomes reckless; punish collisions too heavily and it may decide that the safest move is never to leave the parking space. Real systems combine many terms (safety, rule compliance, comfort, progress) and often keep the imitation loss as an anchor, so that the policy stays close to human style while it improves on the rare cases.

The other risk is the simulator itself. If the world model has a blind spot, for example it never generates a cyclist running a red light, the policy can learn to exploit that gap. That is why the companies stress validation on real roads and why the new Chinese standards require manufacturers to document simulation, test track and road testing together.

Why it matters beyond China

The combination is new: millions of production cars collecting data, generative world models turning that data into an interactive simulator, and RL training inside it. Chinese companies are moving faster than most because they have the fleet, the market pressure and a regulator that, while strict on marketing, allows wide deployment of Level 2 systems. Volkswagen has already chosen XPeng’s VLA 2.0 for its own models, with deliveries planned from 2027 (XPeng). The training methods described here are likely to reach European roads sooner than many expected.