Simulation promises unlimited data, and it delivers unlimited data of uncertain value. Sim-to-real data collection planning is about deciding which real-world measurements to take first so that everything generated afterwards is worth generating.
This guide covers what to capture in the real world before you scale simulation, why calibration data is not training data, how to decide the real-to-synthetic ratio, and how to tell whether your simulator is actually improving.
Simulation needs real data to be worth anything
The appeal of simulation is unlimited episodes at near-zero marginal cost. The catch is that a simulator is only as useful as its correspondence to reality, and correspondence is established with real measurements.
So the question is not whether to collect real data. It is which real data to collect, and in what order, so that simulation does the largest possible share of the remaining work.
Teams that skip this and generate volume first end up with a policy that performs beautifully in simulation and fails on contact, which is the argument in why synthetic data alone will not train a deployable robot.
What to capture in the real world first
| Priority | What to capture | Why simulation cannot supply it |
|---|---|---|
| 1 | Contact force traces | Contact dynamics are where simulators diverge most |
| 2 | Object physical properties | Mass, friction, and compliance are measured, not guessed |
| 3 | Robot execution error | The gap between commanded and achieved motion |
| 4 | Sensor noise characteristics | Real cameras and depth sensors fail in specific ways |
| 5 | Deformable behaviour | Cloth, food, and packaging remain poorly modelled |
| 6 | Real failure modes | Simulators fail differently from hardware |
| 7 | Scene appearance statistics | Lighting, clutter, and texture distributions |
The ordering matters. Force and object properties calibrate the simulator itself, so capturing them first raises the value of everything generated afterwards. Appearance data mostly supports domain randomisation, which is useful and lower leverage.
Calibration data is not training data
This distinction saves programmes a lot of money and is routinely missed.
Calibration data exists to make the simulator match reality. It is small, highly instrumented, and often boring: pressing a gripper against a load cell, dropping objects, measuring friction on surfaces, recording how far a commanded motion actually travels. Nobody trains a policy on it.
Training data teaches the task. It is larger, more varied, and less instrumented.
Because calibration sets are small, they are cheap. A few hundred well-instrumented measurements can improve simulator fidelity enough to make tens of thousands of synthetic episodes usable. That is the highest-leverage real data most programmes never collect, and it is the reason to instrument for force and torque even on a mostly-synthetic programme.
Setting the real-to-synthetic ratio
There is no universal split, and the ratio should differ by task property rather than being set once for the programme.
- Free-space motion and navigation. Simulation carries most of the load; real data mainly supplies appearance variety.
- Rigid pick and place. A workable blend, with real data anchoring grasp outcomes.
- Insertion and assembly. Real data share rises sharply, because success is decided by contact.
- Deformables. Predominantly real. Cloth and food simulation is improving and is not yet a substitute.
- Tool use. Predominantly real, for the same reason.
The practical method is to hold out a real evaluation set, train at several ratios, and measure. Ratios chosen by intuition are almost always too optimistic about simulation, and our real vs synthetic vs hybrid comparison works through the trade-offs.
Measuring whether the simulator is improving
Simulator fidelity gets discussed qualitatively and should be measured.
- Replay real trajectories in simulation and measure divergence. Same commands, same start state, compare outcomes.
- Compare force profiles, not just end positions. A grasp that succeeds in both with very different force histories has not been reproduced.
- Track divergence per task class, since a simulator can be excellent at transport and poor at insertion.
- Re-measure after every simulator change, because improvements in one area frequently regress another.
- Evaluate policies on hardware, always, before any deployment decision, per policy evaluation.
Divergence per task class is the number worth putting on a dashboard. It tells you where synthetic volume is safe to lean on and where it is quietly misleading you.
For a programme that reached production quickly by anchoring simulation in real capture, our mobile manipulation case study covers a 90-day path from cold-start to production.
What to record so it is reusable
- Object identifiers with measured physical properties attached, not category labels
- Commanded and achieved motion as separate fields
- Force at native high rate, unfiltered
- Full camera calibration per session
- Scene conditions, so appearance statistics can be extracted later
- Which simulator version a comparison was run against
The commanded-versus-achieved distinction is the one most often collapsed into a single field, and it is precisely the measurement that tells you how much execution error to model.
Frequently asked questions
Can we start in simulation and add real data later?
You can, and you will likely rebuild once real measurements arrive and reveal the simulator was wrong in ways that shaped the policy. A small calibration collection first is cheaper.
How much calibration data is enough?
Far less than training data. A few hundred well-instrumented measurements per object class and task type is often sufficient, because you are fitting parameters rather than learning behaviour.
Does domain randomisation remove the need for real data?
It helps with appearance and reduces overfitting to synthetic textures. It does not fix contact dynamics, which is where sim-to-real failures concentrate.
Should we build our own simulator?
Rarely worth it. Existing platforms are strong, and the differentiator is the calibration data you feed them rather than the engine itself.
Simulation scales cheaply once it is anchored, and produces confident nonsense before it is. If you want a calibration and capture plan sized before you invest in synthetic pipelines, tell us what you are building.
