Every robotics team eventually asks the same question: how much simulation is too much? The honest answer is that there is no universal real and synthetic data mix. The right ratio depends on your task, your simulator and how wide the gap between them is. What you can do is find your ratio systematically rather than guess.
This guide covers the main ways to combine simulated and real data, how to sweep the sim-to-real data ratio, and what to watch for when synthetic data stops helping.
Why the ratio is not a constant
Synthetic data is cheap, labeled and nearly unlimited. Real data is expensive, scarce and correct. The value of each depends on how closely simulation matches deployment. A well-built twin of a fixed workcell tolerates a heavy synthetic share, while a contact-rich task with deformable objects may gain little from simulation at all.
Richard Sutton’s bitter lesson explains the appeal: “general methods that leverage computation are ultimately the most effective.” Simulation is a way to buy compute-scaled data. The catch is that scale only helps when the data points toward the real target.
Four ways to combine real and synthetic data
| Strategy | How it works | Best when | Main risk |
| Pretrain, then fine-tune | Train in simulation, then fine-tune on real data | Simulation is abundant and real data is scarce | Fine-tuning overwrites useful priors |
| Co-training | Mix both sources in every batch with sampling weights | Sim and real share an action space | Wrong weights let simulation dominate |
| Augment real with synthetic | Add synthetic variations of real scenes | Real data covers the task but lacks diversity | Artifacts that do not exist in reality |
| Synthetic for the tail | Use simulation only for rare failures | Edge cases are costly to collect | Unrealistic failure dynamics |
Co-training keeps real data in every gradient step, which is why it is a common first strategy to test. Published sim-and-real co-training studies report that adding simulation to a small real dataset can lift real-world success.
How to find your ratio
Treat the ratio as a hyperparameter and tune it against real evaluation:
- Fix a real held-out set. Never train on it, and never judge by simulated success.
- Sweep sampling weights. Try synthetic shares such as 0, 25, 50, 75 and 90 percent of each batch, holding total compute constant.
- Vary real data volume too. The best ratio at 100 real episodes is rarely the best at 1,000.
- Plot real success against synthetic share and look for the knee, the point past which more synthetic data stops helping or starts hurting.
- Repeat when the simulator changes. A better twin or wider randomization moves the optimum.
Signs your mix is wrong
- Simulation success is high and real success is flat. The simulator is too easy or unrepresentative.
- Real success drops as synthetic share grows. This is negative transfer: the policy is learning simulation-specific shortcuts.
- Policies look confident but fail at contact. That points to a dynamics gap. Add real contact data, not more visual variety.
- Gains vanish on new scenes. Synthetic variation is too narrow, so revisit domain randomization.
Make real data count
When real data is the scarce ingredient, spend it where synthetic data is weakest: contact-rich steps, deployment-specific scenes and failure recoveries. Use outcome labels and diversity scoring to pick episodes that add information rather than volume.
Keep schemas compatible across both sources so pooling is a configuration change, not a migration. It is the same principle we describe for mixing demonstration and teleoperation data in human demonstration vs. teleoperation data. For the cost side of the decision, see cost per trajectory.
Roborax supplies both halves of the equation: real collection through teleoperation and human demonstration, plus synthetic and sim-to-real work, with schemas designed to pool from day one.
Frequently asked questions
Q: Is there a rule of thumb, like 90 percent synthetic?
A: Some teams run synthetic-heavy mixes, but no ratio transfers across tasks. Treat any quoted figure as one data point and test your own.
Q: Should synthetic data be labeled differently from real data?
A: Yes. Tag the source on every episode and keep it in the schema. You will need it for weighting, ablations and debugging.
Q: When is more simulation not worth it?
A: When real success stops improving as synthetic share grows. Put the next dollar into real data from the failing cases instead.
The right mix is something you measure, not something you inherit from a paper. If you are planning a co-training program and want a real evaluation set and a ratio sweep designed around it, tell us what you are training and we will size the mix against your task.
