Randomize everything and your policy learns nothing useful. Randomize nothing and it fails on contact with the real world. Domain randomization sits between those failure modes, and for manipulation the practical question is not whether to randomize but which parameters earn their place.
This guide covers the main randomization parameters, which ones tend to matter for sim-to-real transfer in manipulation, and how to find out for your own setup without burning compute. It focuses on parameter choice. For how much variation is enough overall, see our companion piece on synthetic variation.
The idea in one paragraph
Tobin and colleagues proposed that if a simulator varies enough, reality becomes one more variation: “the real world may appear to the model as just another variation.” The principle holds. The cost is that every randomized parameter widens what the policy must handle, and a wider distribution can mean slower learning and more conservative behavior.
Four families of parameters
| Family | Examples | Mainly affects | Gap it addresses |
| Visual | Textures, lighting, camera pose, color, distractors | Perception | Appearance gap |
| Dynamics | Mass, friction, damping, actuator gains, latency | Control and contact | Physics gap |
| Geometry and task | Object size, shape, initial pose, goal pose | Generalization | Narrow task distribution |
| Sensor | Noise, depth dropout, exposure, calibration error | Perception | Sensor gap |
What tends to matter for manipulation
Results vary by task, but several patterns recur across published work and practice:
- Camera pose and lighting for vision-based policies. Small camera shifts between simulation and the real rig are a leading cause of failure. Randomize extrinsics within the calibration error you actually expect.
- Friction and mass for contact-rich tasks. Pushing, sliding and grasping hinge on contact, and friction is often the least-known parameter. Peng and colleagues showed dynamics randomization helping a policy transfer to real puck pushing.
- Latency and actuator response for fast or precise motion. Unmodeled delay breaks policies that look flawless in simulation.
- Distractors and backgrounds for clutter. They help when the deployment scene is messy and do little in a fixed workcell.
- Object geometry and initial pose for generalization. Narrow object sets overfit quickly.
Heavily randomized textures can look impressive and still miss the parameter that actually breaks transfer. Randomizing wall color is cheap and rarely the bottleneck.
Calibrate ranges to reality, not imagination
Ranges matter more than parameter lists. Too narrow and reality falls outside them. Too wide and learning stalls. Anchor ranges to measurement: use system identification to estimate friction, mass and latency on your real hardware, then randomize around those estimates with margins.
OpenAI’s Rubik’s Cube work went further with automatic domain randomization, which widens ranges as the policy masters the current ones and removes some manual tuning.
A practical ablation plan
- Start small: camera pose, lighting, friction, mass and latency.
- Hold out real evaluation episodes that never feed training.
- Add one parameter group at a time and track real success rate, not simulation reward.
- Drop what does not move the real metric. Every unnecessary dimension costs training time.
- Log real-world failures and map each to a missing randomization or a missing real-data case.
That last step is where real data pays off. Failures on real hardware tell you which parameters your simulation underspecified, and a modest real dataset is the best diagnostic you can buy. Roborax pairs simulation work with real collection for exactly this loop through our synthetic and sim-to-real service.
When randomization is not enough
Randomization cannot create physics your simulator does not model. Deformables, fluids, cables and complex contacts usually need better models or real data rather than wider ranges. Our guide to sim-to-real transfer limits covers where synthetic robot data stops helping, and digital twins built from real scans narrow the gap from the other direction.
Frequently asked questions
Q: Does more randomization always improve transfer?
A: No. Excess variation can slow training and produce overly cautious policies. Randomize parameters that plausibly differ between simulation and reality, in measured ranges.
Q: Visual or dynamics randomization first?
A: It depends on the policy. Vision-based policies usually need camera and appearance randomization first. State-based control for contact tasks benefits more from dynamics randomization.
Q: How do we know it is working?
A: Evaluate on real hardware or real held-out data. Simulation success rate alone cannot tell you.
Randomization is a hypothesis about your reality gap, and real data is how you test it. If you are tuning a sim-to-real pipeline for manipulation, tell us what you are training and we will help design the real-world validation set.
Related reading
External reference
