A policy trained only on successful demonstrations has never seen what going wrong looks like. The first time a grasp slips, an object lands in the wrong place or a part jams, it faces a state outside everything it learned. Robot failure data collection fills that gap.
The line often attributed to Thomas Edison captures the mindset: “I have not failed. I’ve just found 10,000 ways that won’t work.” For a robot, every one of those ways is training signal, provided someone captured it and labeled it correctly. This guide explains why failure and recovery data matters, how to collect it safely, and how to scope robot failure data collection services that deliver something a model can learn from.
Why success-only datasets break in deployment
Behavior cloning learns from the states demonstrators visit, and demonstrators rarely wander into trouble. The dataset therefore covers a narrow corridor of the task. Once a deployed policy drifts slightly off that corridor, a problem known as compounding error, it lands in states with no examples and no learned way back. Robot recovery demonstrations supply the missing return paths. Our posts on behavior cloning training data and imitation learning to RL data strategy explain how this fits the wider training picture.
What counts as failure data
Failure mode data for robot learning falls into recognizable families:
- Perception failures: misdetected objects, occlusion, depth errors
- Grasp failures: slips, missed grasps, double picks
- Placement and planning errors: wrong location, blocked paths
- Contact failures: jams, collisions, force spikes
- Hardware and sensor faults: dropouts, latency spikes, calibration drift
It also helps to separate three kinds of episode. A failure episode ends badly. A near-miss almost does. A recovery episode passes through a failure state and ends with a correction. Recovery episodes are usually the most valuable.
Three ways to collect failure and recovery data
- Induced failures with deliberate recovery. Skilled operators perturb the scene on purpose, by knocking an object, shifting a target or occluding a camera, and then recover. This is controlled, repeatable and safe.
- Fleet interventions. When autonomy fails and a human steps in, the resolution is a recovery demonstration. See how this works in from intervention to training data, and how the handover is designed in our guide to autonomy handover.
- Policy rollouts with human correction. Run the current policy, log where it fails and have operators demonstrate the fix, in the spirit of dataset aggregation methods. This samples the policy’s real weaknesses rather than your assumptions about them.
Label it so the model learns the right lesson
Imitation learning copies what it sees, including mistakes. A failure episode dropped into a training set without labels teaches the robot to fail. The fix is structure. Every episode should carry the failure type, onset timestamp, recovery start and end, outcome and suspected root cause. Our guide to robotics data QA covers how to enforce that consistently.
With those labels, teams can mask or down-weight the failing segment, train the recovery segment as the target, and reuse the same data for failure detection and reward modelling. Robot error recovery data becomes multi-purpose rather than a niche add-on.
Collect it safely
Staging failures means deliberately entering risky states, so guardrails come first. Define force and speed limits, keep people out of the workspace and use proper safety layers: joint limits, collision checks and e-stops. A good provider designs perturbations that are informative without being destructive.
Balance and sequence your failure data
Failure data complements a strong success base; it does not replace it. A practical sequence:
- Build a solid set of clean demonstrations first.
- Use your intervention rate and incident logs to rank failure modes.
- Collect recovery data for the costliest modes.
- Hold out failure scenarios for testing, following the split design in evaluating embodied AI policies.
- Repeat as new failure modes appear in the field.
What to ask a failure data provider
- Which failure families can you stage, and how do you keep them safe?
- Can operators recover deliberately and consistently?
- What labels and metadata accompany each episode?
- Can you ingest our fleet interventions and incident logs?
- How do you keep failure test scenarios separate from training data?
Where Roborax fits
Roborax designs perturbation protocols, trains operators to recover on purpose and delivers tagged episodes through our teleoperation and long-tail and edge-case services. We also turn live fleet interventions into training data, including for autonomous mobility programs spanning sidewalk delivery robots and passenger vehicles, so your real failures become a training input rather than a support ticket.
“Success shows a robot what to do. Recovery shows it what to do next.” — Roborax data operations team
Frequently asked questions
Q: Does training on failure data teach a robot to fail?
A: Not when it is labeled and handled correctly. The failing segment is masked or down-weighted, while the recovery segment is trained as the target behavior.
Q: How much failure data do we need?
A: There is no fixed ratio. Start with the failure modes that cost you most, collect varied recovery episodes for each, and measure improvement on a held-out failure benchmark.
Q: Can simulation generate failure data?
A: Partly. Simulation can produce many failure variants cheaply, but contact, slip and sensor faults need real-world capture to be trustworthy.
If your robot handles the happy path but struggles when things go wrong, recovery data is the missing piece. Tell us what you are training and we will map failure modes to a collection plan.
