A five-minute demonstration of a robot clearing a table is one episode to your storage system and a dozen distinct skills to your policy. Without robot task segmentation, the policy sees one long, unlabeled stream and has to work out for itself where reaching ends, grasping begins and placing starts.
This guide covers how to segment long-horizon demonstrations into sub-skills and action primitives, which boundary rules keep labels consistent, and how to set up long-horizon task annotation so the result trains hierarchical policies rather than just filling a spreadsheet.
Why long-horizon demonstrations need structure
Short tasks like pick-and-place tolerate raw imitation. Long-horizon tasks such as loading a dishwasher, restocking a shelf or assembling a kit do not. Errors compound across dozens of steps, and a policy that cannot tell which stage it is in will repeat, skip or reorder steps.
Segmentation gives the policy that missing context. Each segment becomes a unit that can be trained, evaluated and recombined on its own. It also enables language conditioning, since “open the drawer” maps cleanly onto a labeled span, and it shows you which stage a failing policy actually fails at.
The idea is not new. In their options framework paper, Sutton, Precup and Singh described planning and learning across multiple levels of temporal abstraction as “key, longstanding challenges for AI.” Demonstration data is where that abstraction has to be labeled.
Sub-skills vs. action primitives
Two layers do two different jobs.
| Layer | What it is | Example | Typical duration |
| Task | The full goal | Clear the table | 30 seconds to several minutes |
| Sub-skill | A goal-directed phase | Pick up the mug, place it in the bin | 3 to 20 seconds |
| Action primitive | A reusable motion | Reach, grasp, lift, rotate, release | 0.5 to 3 seconds |
Sub-skills answer “what is the robot trying to achieve?” Action primitives answer “which motion is it executing?” Keep both in the schema. Sub-skill labeling drives planning and language grounding, while primitives drive low-level control and reuse across tasks.
Boundary rules that keep sub-skill labeling consistent
Most segmentation disagreement is boundary disagreement. Two annotators agree on the sequence of sub-skills and still place the cut 400 milliseconds apart. Fix this with written rules tied to observable signals:
- Anchor to events, not impressions. Cut on gripper state change, first contact or release, using proprioceptive and force signals alongside video.
- Define start and end conditions. “Grasp” starts when the fingers begin closing and ends when the gripper command stabilizes.
- Label recoveries explicitly. A re-grasp is its own segment, tagged as a retry, not silently merged into the first attempt.
- Set a boundary tolerance. Decide how many frames of disagreement count as agreement, and enforce it in QA.
- Freeze and version the ontology. Changing label names mid-project breaks pooling across batches.
Design a schema that survives scale
Useful demonstration annotation carries more than start and end timestamps:
- Hierarchy: task, sub-skill and primitive, linked by parent IDs
- Language: a short natural-language instruction per segment
- Objects and targets: what is being manipulated and where it goes
- Outcome: success, partial or failed segment
- Signal references: pointers to synchronized video, joint state and force channels
Keep segments aligned to the underlying episode records so they join cleanly to every modality. Our posts on reading a teleoperation demonstration log and trajectory data formats show how to structure that foundation.
Automate first, then verify
Manual cutting does not scale to thousands of episodes. The efficient pattern is model-assisted: heuristics or a pretrained model propose boundaries from gripper state, velocity minima and contact events, and annotators correct them. Reviewing is faster and more consistent than drawing from scratch.
Keep a human gate on hard cases: contact-rich assembly, bimanual handoffs and tasks where two sub-skills overlap. Track agreement on boundaries and labels, and route low-agreement episodes to a senior reviewer. Roborax runs this loop inside our annotation and labeling service, with dedicated teams trained on each customer’s ontology.
Common mistakes
- Segmenting by time. Equal-length chunks cut straight through sub-skills.
- Over-fine primitives. A 40-label vocabulary nobody applies consistently is worse than 12 that everyone does.
- Dropping failures. Failed and recovered segments teach a policy what to do after a mistake.
- Annotating from video alone. Video hides the contact events that define real boundaries.
Frequently asked questions
Q: How many sub-skill labels should we start with?
A: Start with 10 to 20 sub-skills per task family and expand only when annotators keep reaching for an “other” label. Consistency matters more than granularity.
Q: Can segmentation be automated?
A: Partly. Heuristics on gripper state and velocity propose good candidates, but contact-rich and overlapping segments still need human review.
Q: Does segmentation help end-to-end policies?
A: Yes. Segment labels can supply language instructions, auxiliary losses and evaluation breakpoints, even when the final policy is a single network.
Long-horizon tasks fail in the middle, not at the ends. If you are planning a segmentation ontology or a re-annotation pass on existing demonstrations, tell us what you are training and we will scope a pilot batch against it.
Related reading
External reference
