Behavior cloning is the simplest way to turn demonstrations into a robot policy, which is why nearly every program starts there. It is also unforgiving about what goes into it. Behavior cloning training data that looks perfectly good can produce a policy that fails the moment anything moves.
This guide covers what behavior cloning specifically demands of a dataset, why recovery episodes matter more than volume, how long-horizon tasks change the requirements, and when the method stops being enough.
What behavior cloning actually asks of your data
Behavior cloning is supervised learning on a control problem. The model sees an observation and predicts the action a human took. That is the whole idea, and its simplicity is why almost every robot program starts here.
The simplicity is also the problem. Because there is no exploration and no reward signal, the model only knows states it saw in the data. The moment it drifts slightly off the demonstrated path, it is in unfamiliar territory, and its errors compound. This distribution shift failure drives every requirement below.
The minimum viable dataset
There is no universal episode count. There is a checklist, and a dataset failing any item will disappoint regardless of size.
| Requirement | Why behavior cloning specifically needs it |
|---|---|
| Exact action labels | The label is the training target; approximations become systematic error |
| Consistent action convention | Mixed conventions teach contradictory mappings from the same observation |
| Recovery episodes | The only way the model learns what to do once it has drifted |
| Object instance variety | Prevents memorization of specific objects rather than the task |
| Starting state variety | Every episode from an identical start produces a brittle policy |
| Multiple operators | Single-operator data encodes one person’s habits as the task definition |
| Clean temporal sync | Misaligned actions are label noise the model cannot detect |
| Outcome labels | Lets you filter deliberately rather than training on everything |
Recovery data is the requirement teams skip
A dataset of perfect demonstrations teaches the model the ideal trajectory and nothing about what to do when it is not on it. In deployment the model will always be slightly off that trajectory, because the world is not the dataset. If it has never seen a recovery, it has no behavior to fall back on.
Three ways to collect recovery data
- Keep natural failures. Operators miss grasps and correct. Do not discard these episodes; label them.
- Perturb deliberately. Nudge the object mid-episode and have the operator recover. Cheap and effective, and unpopular with operators unless you explain why.
- Start from off-nominal states. Begin some episodes from a partially failed configuration rather than a clean setup.
Set an explicit target, for example a defined fraction of episodes containing a recovery. Left unmeasured it trends to zero, because clean episodes are faster to collect.
Variety beats volume, with one caveat
Given a fixed budget, more object instances and more starting configurations almost always beat more repetitions. A thousand episodes across fifty objects generalizes better than five thousand across five.
The caveat is that variety without sufficient depth per variation produces a model that is uniformly mediocre. There is a floor, roughly the number of examples needed to learn the skill at all, and variety only helps above it. Find the floor with a small pilot on a single configuration, then spend everything else on breadth. Our post on how much humanoid training data you need works through the sizing.
What long-horizon tasks change
Behavior cloning on a two-second grasp is well understood. Behavior cloning on a four-minute household task is a different problem, because errors compound across every sub-step and the model must track progress it cannot directly see.
In our everyday task capture for a humanoid robotics developer building general-purpose home and service robots, three adjustments matter most:
- Label sub-step boundaries. They let you train and evaluate per phase rather than per episode, turning one hard problem into several tractable ones.
- Keep partial episodes. A run that failed at step four contains three usable steps. Discarding whole episodes on final outcome throws away most of your data.
- Collect mid-task recoveries specifically. Not just recovery from a dropped object, but recovery from being at the wrong sub-step entirely.
The full picture is in VR headset teleoperation for everyday tasks and the methodology in long-horizon data capture.
When behavior cloning is not enough
Behavior cloning plateaus. It can match demonstration quality and rarely exceeds it, and on tasks where the demonstrations are themselves inconsistent it will average them into something worse than either. The usual next step is reinforcement learning on top of a cloned policy, which changes your data requirements substantially. Our post on moving from imitation learning to RL covers what shifts.
Recovery data is usually what moves the number. Our warehouse policy case study tracks deformable-item success from 61 to 84 percent.
Frequently asked questions
How many demonstrations do I need to start?
Run a pilot on one configuration and find the point where performance stops improving with more repetitions. That floor, multiplied across your variation dimensions, is a far better estimate than any published number from different hardware.
Should we filter out low-quality episodes?
Filter on measurable defects such as sync errors, force spikes, and latency breaches. Do not filter on outcome alone, because that removes recovery behavior. Keep everything, train on subsets, and record which subset produced which model.
Does more operators mean noisier data?
It means more behavioral diversity, which helps generalization, and more variance, which needs managing through task scripts and QA. See task script design.
Can synthetic data substitute for demonstrations?
It supplements well for perception and scene variety and substitutes poorly for contact dynamics. Our real vs synthetic vs hybrid comparison covers the split.
Most behavior cloning disappointments trace back to a dataset that was large, clean, and missing recoveries. If you are sizing a first collection and want the mix reviewed before you spend, tell us what you are training.
