
Mixing Real and Synthetic Data: Finding the Right Ratio for Robot Policy Training
Every robotics team eventually asks the same question: how much simulation is too much? The honest answer is that there is no universal real and
Home / Data services / Evaluation benchmarks
Held-out test sets, scenario libraries, and scoring infrastructure for model release gates.
Evaluation benchmarks are held-out test sets, scenario libraries, and automated scoring infrastructure that tell you whether a model is ready to ship. They turn “it seems to work” into a pass/fail gate with numbers you can defend.
Building a benchmark that actually predicts real-world performance requires curated scenarios, physical test setups, and statistical rigor. We maintain the library so you run evals, not build them.
Why outsource benchmarks?
Internal benchmarks drift toward what your model is already good at. We maintain independent, adversarial test sets designed to find the gaps.
50+ benchmarks built for partners.
2,000+ scenarios in the library.
100/day automated eval runs.
Where we collect
41+ delivery centers across 12 countries. Every program runs from a Roborax hub near your target time zone.
Asia Pacific
India · Philippines
Americas
USA · Canada · Colombia · Jamaica · El Salvador · Belize
EMEA
UK · Albania · Kosovo · Morocco
Four artifacts that turn a model release into a confident decision instead of a gut feel.
Isolated scenarios your model has never seen. Leakage check enforced.
Curated scenes covering production distribution and known edge cases.
Aggregation, statistical significance, and per-scenario breakdowns.
Replayable runs that catch silent regressions between model versions.
Four stages that produce a benchmark you can actually defend in a safety review.
What does success look like for this release? Pass/fail and metric thresholds.
Curated scenarios from real + synthetic. Coverage matrix locked.
Run your model against the set. Per-scenario results plus aggregate metrics.
Release-grade scorecard with regression flags. Replayable for any future model.
Isaac eval, Habitat Lab, custom runners. Open splits or your own.
Sim harness
Your stack
Embodied AI
Public splits
Real-robot eval
Stats reports
FAQ
Benchmark design is a collaborative process. Your team defines the success criteria and task requirements. We design the evaluation protocol, build the test environment, and handle execution.
Benchmark tasks are kept strictly separate from training data. We use held-out scenarios, novel object instances, and modified environmental conditions that your model has not been exposed to.
Task success rate by condition, failure mode taxonomy, per-subtask breakdown, comparison to your previous benchmark run, and recommendations for the next training iteration.
At minimum after every major training update. For active programs we recommend a rolling benchmark cadence — typically fortnightly — so you can track policy improvement in near real time.
From the blog
The QA Pipeline Every Robotics Data Team NeedsSuccess-rate measurement, regression suites, and policy scoring.
From the blog
From Imitation Learning to RL: How Your Data Strategy ChangesEvaluation requirements shift significantly between training regimes.
Tell us the deployment domain and the metrics that matter. Six weeks to scorecard.
FROM THE FIELD

Every robotics team eventually asks the same question: how much simulation is too much? The honest answer is that there is no universal real and

Most simulated environments are invented. A digital twin is the opposite: a simulation built from a real place, so the robot trains in a scene

Randomize everything and your policy learns nothing useful. Randomize nothing and it fails on contact with the real world. Domain randomization sits between those failure

Two annotators label the same grasp. One places the contact on the handle, the other on the rim. Both are confident, and either one of

A camera tells a robot what something looks like. A point cloud tells it where things are. For navigation, grasp planning and obstacle avoidance, that

Behavior cloning treats every demonstration as equally good. Reinforcement learning cannot afford to, because it needs to know what worked. Robot outcome labeling supplies that
Seven services. One synchronized pipeline.
VR and leader-follower robot control logging.
In-person task demos for imitation learning.
RGB-D, LiDAR, force, and tactile streams.
Bounding boxes, segmentation, action labels.
Domain-randomized scenes and sim transfers.
Rare scenarios your policy will face in production.