Beyond Success Rate: Better Metrics for Evaluating Manipulation Policies

Manipulation Evaluation Metrics

Success rate is the metric everyone reports and the one that hides the most. Two policies can both succeed 60 percent of the time, one failing at the last centimeter and the other failing at the first grasp. Manipulation evaluation metrics that go beyond a binary outcome tell you which policy is closer to working, where it breaks and what to collect next.

This guide covers the task progress score, partial success metrics, efficiency and safety measures, and failure mode analysis, and how to report them without drowning your team in numbers.

Table of contents

    What success rate hides

    A binary outcome discards four things: how close a failed attempt came, how the successful ones were achieved, what they cost in time and interventions, and how robust the result is across conditions. It also behaves badly on long tasks. A ten-step task with 95 percent reliability per step succeeds only about 60 percent of the time overall, and a headline number gives no hint which step is responsible.

    There is a second risk. In Marilyn Strathern’s phrasing of Goodhart’s law, “When a measure becomes a target, it ceases to be a good measure.” A single metric invites policies, and teams, to optimize it at the expense of everything else.

    A metric toolkit

    Metric What it tells you How to collect it Watch out for
    Task progress score How far the policy got Ordered milestones scored per trial Milestones must be objective
    Partial success Credit for near misses Graded rubric such as 0, 0.5, 1 Rubric drift across evaluators
    Time to completion Efficiency Timestamps on successful trials Survivorship bias
    Intervention rate Real autonomy Human takeovers per trial or per hour Defining what counts as one
    Recovery rate Resilience Success after a failed attempt Needs retries to be allowed
    Safety events Risk Collisions, force-limit trips, e-stops Under-logging
    Smoothness Motion quality Jerk and path length from joint logs Weak link to task success

    Task progress scores

    A task progress score assigns credit for each completed milestone, such as reach, grasp, lift, transport, place and release. The score is milestones achieved divided by total milestones, with weights if some steps matter more. It separates a policy that never grasps from one that fails only at release, which success rate cannot.

    Milestones must be observable and ordered, and wherever possible computed from logs rather than judged by eye. Our guide to task segmentation for long-horizon demonstrations produces exactly the boundaries you need. Report the distribution, not just the mean. A policy that reaches milestone three every time is a different animal from one that alternates between zero and five.

    Failure mode analysis

    The most actionable output of an evaluation is the failure breakdown. Define a taxonomy and label every failed trial with one primary mode:

    • Perception error: wrong object, missed detection, bad depth
    • Grasp failure: miss, slip, unstable hold
    • Motion failure: collision, joint limit, overshoot
    • Sequencing failure: wrong order, loops, stalled progress
    • Timeout or hardware fault

    Chart the distribution across checkpoints. A policy that trades grasp failures for sequencing failures is making progress even if success rate is flat. Failure modes also tell you which data to collect next, which is why we cover the labeling side in our guide to robot outcome labeling.

    Efficiency, safety and robustness

    Success means little if the robot needed three human interventions to achieve it. Report time to completion and intervention rate alongside success, and track the field-facing version through robot fleet intervention rate.

    Measure robustness as success under each condition rather than averaged across them: lighting, clutter, object set. The worst-condition score is often more informative than the mean. Add safety events, because a policy that succeeds by colliding with the shelf is not an improvement.

    Report without overload

    Pick a primary metric for each decision. A sensible dashboard shows success rate with a confidence interval as the headline, with task progress, failure breakdown and interventions as supporting views. Use intervals throughout, following our robot policy evaluation protocol.

    Avoid composite scores that bury information. If you need a single number, publish the formula and keep every component visible next to it.

    Where Roborax helps

    Roborax evaluation collection logs milestones, timestamps, interventions and failure categories on every trial, so these metrics come from the same dataset as the success rate rather than from a separate manual process. Our evaluation benchmarks service delivers that data in a schema that pools across checkpoints and sites.

    Frequently asked questions

    Q: What is the best single metric?

    A:There isn’t one. Use success rate with an interval as the headline and a task progress score as the diagnostic, then add failure breakdown for decisions about what data to collect.

    Q: How do we score partial success consistently?

    A:Write observable milestone definitions, train evaluators on example videos and measure agreement between them. See our guide to inter-annotator agreement.

    Q: Do composite scores help?

    A: Only if the formula is public and the components stay visible. Otherwise they hide the trade-offs they are meant to summarize.

    A policy is more than its success rate, and so is the data that improves it. If you want evaluation logs that capture progress, interventions and failure modes from the first trial, tell us what you are training and we will design the metrics with you.

    Related reading

    External reference

    Partha Chowdhury

    Partha Chowdhury · SVP & Country Head, India and EMEA

    Partha oversees Fusion CX operations across India and EMEA, including the security and delivery infrastructure behind every robotics data program, and writes about operator quality and scaling collection teams.

    Contact form

    Or just fill this out

    We’ll route your message to the right inbox and respond within one business day.