A robot that can see a mug still has no idea how to pick it up. Hand-object interaction data closes that gap: the labels describing when contact begins, how the grip forms, whether the object moves in the hand, and when it is released.
This guide covers the contact lifecycle and where its boundaries sit, what to label beyond the phases, the three places annotator agreement reliably breaks down, and which parts of the job to automate.
Why hand-object interaction is its own labeling problem
Most annotation work in robotics labels objects: bounding boxes, masks, class names. Hand-object interaction labeling asks a harder question, which is not what is in the scene but what is happening between the hand and the thing it is touching.
That is a relational, temporal label. It cannot be drawn on a single frame, because contact has a beginning, a duration, and an end. A box around a mug tells you nothing about whether it is being reached for, held, or released.
The contact lifecycle
| Phase | Defined by | Boundary set at |
|---|---|---|
| Approach | Hand moving toward object, no contact | Motion begins toward the target |
| Pre-grasp | Hand shaping for the intended grip | Fingers begin to configure |
| Contact onset | First touch | First non-zero contact force |
| Grasp closure | Grip forming and stabilizing | Force rising toward steady state |
| Hold and transport | Stable grip, object moves with hand | Force plateaus |
| In-hand adjustment | Object moving relative to fingers | Optional; hardest to label |
| Release | Grip opening | Force begins sustained decline |
| Retreat | Hand withdrawing | Contact force reaches zero |
Most of these boundaries are defined by force rather than appearance. Where force data exists, much of this segmentation derives automatically. Where it does not, every boundary becomes a human judgment call with human variance attached.
What to label beyond the phases
- Grasp type. A controlled vocabulary such as power, precision, pinch, hook, lateral. Free text cannot be aggregated across annotators.
- Contact points. Which digits or gripper surfaces touch, and where on the object.
- Object instance. Instance-level, not category.
- Object state. Full, empty, open, closed, deformed. State frequently determines correct strategy.
- Intent. What the grasp was for. Picking a knife to cut is a different grasp from picking it to pass.
- Outcome. Stable, slipped, dropped, crushed, regrasped.
- Bimanual role. Which hand acts and which stabilizes.
Intent is the field teams most often omit and most often want later, because it connects manipulation data to language-conditioned policies.
Where annotator agreement breaks down
When approach becomes pre-grasp
There is no visible instant where reaching becomes shaping. Define it operationally, for example the first frame where finger configuration leaves neutral, and accept it as a convention rather than a truth.
What counts as in-hand adjustment
Small slip and deliberate repositioning look nearly identical from outside. Without force or object-pose tracking, annotators guess, and they guess differently.
When a grasp has failed
An object that shifts and is recovered may be a failure to one annotator and a successful adjustment to another. Define failure by outcome, and record the recovery separately.
Measure inter-annotator agreement on these three specifically rather than on the task overall. An average score hides exactly the disagreement that matters. Our robot data annotation guide covers the wider QA structure.
Automate what you can, review what you cannot
- Derive contact onset and release from force traces where available. Reliable, cheap, and removes the highest-volume manual task.
- Derive transport phases from motion, using object and hand velocity together.
- Reserve human annotation for semantics: grasp type, intent, object state, failure classification.
- Review a stratified sample weighted toward the ambiguous boundaries above rather than uniformly.
- Version the label taxonomy and record which version applied, per dataset structure guidance.
Programs that automate temporal labels and reserve humans for semantic ones typically cut annotation cost substantially while improving consistency, because machines do not disagree with themselves.
What a good annotation guideline document contains
Most inconsistency traced back to guidelines is not caused by annotators being careless. It is caused by guidelines that describe the ideal case and go quiet on the ambiguous one, leaving each annotator to invent their own convention.
- A worked example per phase, with the exact frame where the boundary sits and a sentence explaining why.
- Negative examples. Two or three clips showing what a wrong boundary looks like, which teaches faster than any amount of prose.
- Explicit tie-breakers. For every ambiguous case, a stated default. “If contact and shaping are indistinguishable, mark contact onset” is a rule; “use judgment” is not.
- The reason behind each convention. Annotators who understand that contact onset feeds automatic segmentation apply it consistently under time pressure.
- A version number and change log, recorded against every episode annotated under it.
Test guidelines the same way you test a task script: give them to annotators who have not seen them, compare output, and treat divergence as a defect in the document rather than in the people.
Labeling the harder cases
Bimanual interactions
When two hands are involved, label the role of each rather than treating both as actors. One hand typically stabilizes while the other manipulates, and the stabilizing role carries force but little motion. A taxonomy that only records grasps misses it entirely, which matters because a policy that ignores the stabilizing hand drops the object. See bimanual manipulation datasets.
Deformable objects
Contact points on cloth or food move as the object changes shape, so a static contact-point label is wrong within a second of being applied. Label contact regions rather than points, and record object state transitions as their own events.
Tool-mediated contact
When the hand holds a tool and the tool touches the object, there are two contact relationships to track. Conflating them makes tool-use data unusable for anything except gross motion. Label hand-to-tool and tool-to-object separately.
Contact labelling matters most on objects that change shape. Our warehouse policy case study tracks deformable-item success from 61 to 84 percent.
Frequently asked questions
Can these labels be generated from video alone?
Partially, with meaningful error. Vision-based contact detection works on clear unoccluded views and degrades exactly where manipulation data is most valuable, during close contact.
Do we need them for end-to-end behavior cloning?
Not to train the policy directly, but to evaluate it, filter data, and diagnose failures. Teams that skip them can train a model and cannot explain it.
How granular should grasp taxonomy be?
Start coarse, around five to seven types, and refine only when a real decision depends on finer distinction. Over-detailed taxonomies collapse annotator agreement.
Does this transfer between hands and grippers?
Phase structure transfers well. Grasp taxonomy does not, since a parallel gripper has a fraction of the grasp vocabulary a hand does. See dexterous manipulation data.
Hand-object labels turn a pile of manipulation video into a dataset you can query, filter, and diagnose. If you are designing a labeling taxonomy or reviewing one producing inconsistent output, send us the guidelines.
