An annotation workflow that works for a thousand clips will quietly fall apart at twenty thousand. Egocentric video data annotation at scale is less about labelling technique than about tiering, sampling, and versioning.
This guide covers why annotation breaks at volume rather than at the start, how to tier work by who actually has to do it, why sampling beats reviewing everything, and how to version labels so results stay reproducible.
Why annotation breaks at scale rather than at the start
Annotating a hundred clips is a task. Annotating fifty thousand is a system. The workflows that work at small volume fail at large volume for reasons that have nothing to do with the labels themselves.
Three things break first. Guidelines that were clear enough when four people shared an office stop being clear across forty people in three countries. Review becomes a bottleneck because every clip is reviewed the same way. And nobody can answer which annotation version produced which trained model.
Tier the work by who has to do it
The single largest efficiency gain is refusing to treat all labels as one job.
| Tier | Examples | Who or what does it |
|---|---|---|
| Derived | Contact onset and release, transport phases | Computed from force and motion traces |
| Mechanical | Object presence, clip boundaries, view checks | Model-assisted with sampled review |
| Judgement | Grasp type, intent, failure classification | Trained annotators |
| Specialist | Clinical, industrial, or safety-relevant labels | Credentialed reviewers |
| Adjudication | Disagreements and edge cases | A senior reviewer with authority to change guidelines |
Programmes that derive the temporal labels from sensor traces rather than having humans mark them typically remove the largest single block of manual work, and improve consistency at the same time, because machines do not disagree with themselves. Where force data exists, contact boundaries should never be hand-marked, as covered in hand-object interaction labelling.
Sample review, do not review everything
Reviewing every clip is the default and it does not scale. It also produces worse quality than sampling does, because uniform review spreads attention evenly across work that is not evenly risky.
- Weight by annotator. New annotators reviewed heavily, established ones sampled.
- Weight by ambiguity. Label types with known low agreement get more review than obvious ones.
- Weight by novelty. New task types, new objects, and new sites are reviewed harder.
- Use blind double-annotation on a slice to measure agreement rather than assuming it.
- Escalate systematically. Repeated disagreement on one label type is a guideline defect, not an annotator defect.
Track inter-annotator agreement per label type rather than overall. An aggregate score hides exactly the categories that are broken.
Version labels like code
Annotations change. Taxonomies get revised, boundaries get redefined, errors get corrected. If labels live inside the episode file, every revision rewrites history and you lose the ability to reproduce an earlier model.
- Store annotations separately, referencing episodes by identifier, per dataset structure.
- Version the taxonomy and record which version applied to each annotation pass.
- Never edit in place. Publish a new annotation version and keep the old one.
- Cite both in every training run: an episode set and an annotation version.
- Re-brief on change, and mark the first sessions after a taxonomy revision for review.
This costs almost nothing at setup and is the difference between “the model got worse” and “the model got worse because the label definition changed in March.”
Where throughput actually goes
Teams model annotation cost as time per clip. The real distribution is less even.
- Clip loading and navigation often rivals labelling time. Tooling that pre-buffers and jumps to candidate boundaries pays back quickly.
- Ambiguous cases consume disproportionate time. A small share of clips absorbs a large share of hours.
- Context switching between label types is expensive. Batch by label type rather than annotating each clip fully.
- Rework from unclear guidelines is usually the largest hidden cost and the least tracked.
Measure time per label type, not per clip. It is the only way to see which part of the taxonomy is expensive and whether it is worth what it costs.
For a programme where sustained labelling discipline moved a hard outcome, our warehouse policy case study tracks deformable-item success from 61 to 84 percent.
Frequently asked questions
How much can be automated?
Temporal boundaries derived from sensor traces, and a first pass on mechanical labels. Semantic judgement, intent, and failure classification still need people, and pretending otherwise produces confident wrong labels.
What agreement rate should we expect?
It varies enormously by label type. Object presence should be near-unanimous; grasp type and intent will not be. Set a target per label type rather than one number for the whole taxonomy.
Should annotators see the model’s predictions?
Model-assisted labelling is faster and biases annotators toward accepting predictions. Use it for mechanical labels and keep judgement labels blind, or you will measure agreement with the model rather than accuracy.
In-house or outsourced annotation?
Outsourcing scales better once guidelines are stable. Keep adjudication and taxonomy ownership in-house, because those decisions shape what the model learns.
Annotation quality is a systems problem long before it is a labelling problem. If you want a workflow reviewed before you scale headcount behind it, send us the guidelines.
