A policy scores well in evaluation and disappoints in deployment. This happens so reliably that it is worth treating as a property of how robot evaluation is usually designed rather than as an unlucky outcome. Embodied AI policy evaluation is where most of the optimism is manufactured.
This guide covers the four reasons the number always drops, why splitting your data on the right axis is the highest-value fix available, the metrics worth reporting alongside success rate, and how to build an evaluation set designed to break things.
Why the number always drops
A policy reports a high success rate in evaluation and a markedly lower one in deployment. This is so consistent that it is worth treating as a structural property of how robot evaluation is usually done, rather than as bad luck.
Four causes account for most of the gap.
The test set came from the training distribution
Episodes are collected in one programme and split randomly. The same objects, operators, lighting, and layouts appear on both sides. The evaluation measures interpolation within the distribution, but teams report it as generalization.
Reset conditions are cleaner than reality
Evaluation episodes begin from a tidy, staged state. Deployment begins from the state left by the previous action, which often differs from the starting state of every evaluation episode.
Success is defined narrowly
“Object in bin” ignores whether the robot crushed it, needed four attempts, or knocked something over along the way. Real acceptance is broader than the metric.
Failures were filtered from the data
If teams remove failures from the dataset, the policy never learns recovery, and clean-start evaluations never test its ability to recover. The weakness is invisible until deployment, as covered in behavior cloning training data.
Split on the axis you are claiming
The single highest-value change is to stop splitting randomly. Split along whichever axis matches the claim you want to make.
| Claim | Split by | Held out |
|---|---|---|
| Works on new instances of known objects | Object instance | Specific items never seen in training |
| Works in a new room | Scene | Entire environments |
| Works for anyone | Operator | All episodes from held-out operators |
| Works on a new phrasing | Instruction | Unseen paraphrases and referring expressions |
| Works on a new robot | Embodiment | A whole platform |
| Works next month | Time | The most recent sessions |
Each split produces a lower number than random splitting. That is the point: the lower number is the honest one, and it is the one that predicts deployment.
Metrics worth reporting alongside success rate
- Attempts per success. A policy that succeeds on the fourth try is not equivalent to one that succeeds first time.
- Peak contact force. Successes achieved through excessive force are failures that have not happened yet, per force-torque capture.
- Recovery rate. Given a perturbation mid-episode, how often does it recover rather than continue blindly?
- Collateral events. Objects displaced or knocked that were not the target.
- Time to completion, and its variance. High variance signals an unreliable policy that averages well.
- Intervention rate, once deployed. The only metric measured against reality, and the subject of closing the intervention loop.
Build an adversarial slice on purpose
Standard evaluation asks whether the policy handles what you expected. A more useful question is where it breaks. That needs a deliberately hostile evaluation set, held separate and never trained on.
- Perturbation episodes. Move the object mid-reach and see whether it adapts.
- Off-nominal starts. Begin from partially failed states, which is what deployment does constantly.
- Novel instances. Objects in the right category with unfamiliar shape, weight, or finish.
- Degraded conditions. Poor lighting, added clutter, partial occlusion.
- Ambiguous instructions. Referring expressions that require context, per language-conditioned data.
- Adversarial physics. Deformables, unbalanced loads, items that shift when lifted.
Report both numbers. The clean figure tells you the policy works; the adversarial figure tells you how much margin you have. Programmes that report only the first metric often surprise teams during deployment.
Simulation evaluation is a screening tool
Evaluating in simulation is fast, cheap, repeatable, and systematically optimistic on exactly the tasks that matter most. Contact dynamics are where simulators diverge from reality, and contact is where manipulation lives.
Use simulation to screen out clearly bad policies quickly, and require real-hardware evaluation before any deployment decision. The distinction is developed further in why synthetic data alone will not train a deployable robot.
For a programme where a precision metric moved on real hardware, our surgical robot case study reports a 67 percent reduction in tissue contact errors.
Frequently asked questions
How many evaluation episodes are enough?
Enough that the confidence interval is narrower than the difference you are trying to detect. Teams routinely compare policies on episode counts far too small to distinguish them, then act on noise.
Should evaluation data ever be reused for training?
No. Once it has been trained on, it measures memorisation. Keep a permanently held-out set, version it, and treat contamination as a serious incident rather than a shortcut.
Is a single success-rate number ever adequate?
Only for a narrow, well-defined task with a hard pass criterion. For anything open-ended, one number hides the attempts, the force, and the collateral damage.
How do lab results relate to deployment?
Treat the lab figure as an upper bound rather than a prediction. The useful measurement is intervention rate in the field, which is why fleets that instrument for it can improve far faster.
Most evaluation gaps are not model problems. They are measurement problems that teams introduced when they chose the split. If you want an evaluation protocol reviewed before you commit to a deployment decision, tell us what you are measuring.
