Most video datasets are built around objects: what is in the frame and where. Robot policies need something harder. First-person action recognition data is about labelling what is being done, and verbs resist annotation in ways nouns never do.
This guide covers why verbs are harder than nouns, what an action label actually has to contain, the four places labelling reliably goes wrong, and how to build a taxonomy that survives contact with real annotators.
Nouns are easy, verbs are not
Object detection is a solved-enough problem. Point a model at a kitchen and it will find the mug, the kettle, and the tap with reasonable reliability.
Ask what the person is doing and the difficulty jumps. Pouring and rinsing look nearly identical for the first second. Reaching for a cup and moving it aside share the same opening motion. Verbs are defined by intent and outcome, both of which unfold over time and neither of which is visible in a frame.
For robot learning this matters because policies are trained on actions, not on inventories. A dataset rich in object labels and thin on action labels teaches a robot what a kitchen contains and nothing about what to do in one.
What an action label actually needs
| Element | Example | Why it is required |
|---|---|---|
| Verb | pour | The action class itself |
| Target object | kettle_03 | Instance-level, not category |
| Recipient or destination | mug_07 | Many actions are three-way relations |
| Tool or hand | right hand | Distinguishes bimanual roles |
| Temporal bounds | start and end frames | Actions are intervals, not instants |
| Outcome | completed, spilled, aborted | Separates intent from result |
| Parent task | make tea | Links primitives to the goal above them |
Most public action recognition data carries the first two. The remaining five are what make a label usable for control rather than for classification, and they connect directly to the granularity argument in language-conditioned manipulation.
The four places labelling goes wrong
- Verb taxonomies that are too fine. Distinguishing “place”, “set down”, and “put” collapses annotator agreement without helping any model. Start coarse.
- Boundaries defined by appearance. Where reaching becomes grasping is a convention, so state the convention and apply it, rather than leaving it to judgement.
- Intent inferred after the fact. An annotator watching a completed episode knows the goal. The model at inference time does not. Labels written with hindsight encode information the policy will never have.
- Ignoring simultaneity. People stabilise with one hand while acting with the other. A single-verb-per-interval schema cannot represent that, and household tasks are full of it.
Building a verb taxonomy that holds up
- Start with fifteen to twenty verbs, not two hundred. Coarse categories that annotators apply consistently beat fine ones they apply differently.
- Define each by observable outcome, not by motion shape. “Pour” is defined by contents moving between containers, which is checkable.
- Split only when a decision depends on it. If no model behaviour changes between two verbs, they are one verb.
- Allow concurrent labels so a stabilising hand and an acting hand can both be represented.
- Version it, and record the version per annotation pass, per annotation workflow.
- Pilot on unseen annotators and treat divergence as a defect in the taxonomy rather than in the people.
What the labels unlock
- Sub-task segmentation for long-horizon training, so a four-minute episode becomes a sequence of learnable units.
- Language grounding, since instructions are verbs before they are anything else.
- Targeted evaluation. Success rate per action class tells you where a policy is weak, which an overall figure hides.
- Coverage auditing. Verb frequency across the dataset reveals which actions are under-collected.
- Failure analysis by action type rather than by episode, which is far more actionable, per policy evaluation.
The coverage point is underrated. Most datasets are heavily skewed toward pick and place because those episodes are quick to collect. A verb histogram makes that visible immediately, and it is usually uncomfortable reading.
For a programme built on well-structured action supervision, our humanoid foundation model case study documents a 3x sample-efficiency result.
Frequently asked questions
Can action labels be generated automatically?
A first pass, yes, particularly where force and motion traces mark boundaries. Verb classification and intent still need human judgement, and automated intent labels tend to describe outcomes rather than intentions.
How many verbs should a taxonomy have?
Fewer than instinct suggests. Fifteen to twenty covers most manipulation domains. Expand only where a real decision depends on the distinction.
Should we reuse a public action taxonomy?
Starting from one helps interoperability and makes pooling with public data easier. Expect to extend it, since public taxonomies are built for classification rather than for control.
How do we label an action that fails?
Label the attempted action and record the outcome separately. Collapsing intent and result into one label destroys exactly the distinction a policy needs to learn recovery.
Object labels tell a robot what is present. Action labels tell it what to do. Most datasets are rich in the first and thin in the second. If you want a taxonomy reviewed before annotation scales, send us the guidelines.
