World models are the most discussed and least understood idea in embodied AI right now. Strip away the framing and the claim is simple: a model that can predict what happens next can plan, and a robot that can plan needs less trial and error.
This guide explains what a world model is, where it genuinely helps and where it does not, why world models in robotics need action-conditioned data rather than ordinary video, and the one collection requirement that conflicts with everything else you do.
What a world model is
A world model learns to predict what happens next. Give it the current observation and a candidate action, and it produces the observation that would follow. In effect it is a learned simulator, built from recorded data rather than written by hand.
That distinction is the point. A physics engine encodes rules a person specified. A world model encodes regularities it observed, including the messy ones nobody would think to write down: how a half-full bag slumps, how a cloth catches on a corner, how a drawer sticks before it gives.
Why prediction is useful to a robot
A policy that can predict consequences can evaluate options before committing to one. Instead of reacting, it can consider several actions, imagine the outcome of each, and pick the best. It also gets a cheap source of practice, because it can rehearse against its own predictions rather than against hardware.
Where world models help, and where they do not
| Use | Value | Caveat |
|---|---|---|
| Planning several steps ahead | High | Prediction error compounds quickly |
| Generating synthetic practice | High | Model artifacts become learned behavior |
| Detecting anomalies | High | Needs a calibrated notion of surprise |
| Reducing hardware wear | Moderate | Only if predictions are trustworthy |
| Contact-rich manipulation | Low so far | Contact is exactly where prediction is worst |
| Long-horizon household tasks | Low so far | Horizons exceed reliable prediction range |
The honest summary is that world models are strong at short-horizon prediction in familiar settings and weak precisely where robotics is hardest. Treat them as a complement to real data, not a substitute.
Action-conditioned video is the core format
Ordinary video teaches a model what tends to happen. It cannot teach what happens because of a particular action, because the action is not recorded. Action-conditioned video is video with the action attached to every frame, and it is the format world models are actually built from.
The requirements are stricter than for policy training.
- Tight sync. If actions are offset from frames by even a little, the model learns a shifted cause-and-effect relationship. This is the fastest way to ruin a world model.
- Uniform, measured control rate. Variable rate makes each action mean something different, per trajectory formats.
- Coverage of unusual actions. A dataset of only sensible actions cannot predict the consequences of bad ones, which is what planning needs most.
- Failures retained. Dropped, knocked, and jammed outcomes are the interesting predictions.
- Force alongside vision. Contact outcomes are invisible in pixels until after they matter, as covered in force-torque capture.
The deliberately bad action problem
This is the requirement that conflicts with everything else you collect.
Demonstration data is, by design, full of competent behavior. Operators are selected and scored for it. But a world model trained only on competent actions has never seen what happens when you push a glass sideways or close a gripper on nothing, so it cannot warn a planner away from those choices.
Programmes that want world-model-quality data therefore add a deliberate exploration slice: scripted sub-optimal actions, random perturbations, and recovery from states no sensible operator would create. It sits awkwardly alongside a quality-scored teleoperation programme, and it needs its own labelling so it never leaks into an imitation-learning split. This is a variant of the recovery-data argument in behavior cloning training data, pushed further.
Practical guidance
- Record at uniform rate and store measured rate per episode.
- Verify sync with a physical test, not by trusting the stack. A visible clock in frame settles it.
- Tag exploration episodes separately so they can be included or excluded deliberately.
- Keep raw resolution. Prediction quality degrades with aggressive compression in ways that are hard to notice until training.
- Pair with real evaluation. A world model that predicts well and transfers badly is a common and expensive outcome, so hold it to your evaluation benchmarks.
- Do not let synthetic rollouts dominate. The ratio of real to imagined data is a tuning decision with real consequences, discussed in our real vs synthetic comparison.
For a programme that reached production quickly on real capture rather than predicted data, our mobile manipulation case study covers a 90-day path from cold-start to production.
Frequently asked questions
Is a world model the same as a simulator?
Functionally similar, differently built. A simulator encodes physics a person wrote. A world model learns regularities from data, so it captures things nobody modelled and also inherits whatever biases the data carried.
Can a world model replace real data collection?
No. It is trained on real data and its predictions degrade fastest in exactly the contact-rich, long-horizon situations that need the most data. It reduces hardware time; it does not remove the need for capture.
How far ahead can predictions be trusted?
Short horizons in familiar settings. Error compounds with each predicted step, so treat long rollouts as illustrative rather than reliable and validate against real outcomes.
Do we need world-model data if we only do imitation learning?
Not immediately, but the recording discipline overlaps heavily. Tight sync, uniform rate, and retained failures serve both, so collecting to the stricter standard keeps the option open.
World models are promising and consistently oversold. The data discipline they require is worth adopting regardless, because it is the same discipline that makes every other model easier to train. If you want your capture reviewed against that standard, tell us what you are building.
