Ask a home robot team what they are collecting and the answer is usually “household tasks.” That phrase hides more than it explains. Building an activities of daily living taxonomy first is what turns an open-ended ambition into a dataset you can scope, budget, and audit.
This guide covers why capability grouping beats room grouping, how to define a task so two operators perform it the same way, which dimensions actually drive difficulty, and how to prioritise what to collect first.
Why a taxonomy comes before a dataset
“Household tasks” is not a specification. It is a category label covering hundreds of distinct behaviors with very different data requirements, and treating it as one thing is how home robot programs end up with large datasets that cover almost nothing well.
A task taxonomy fixes that. It names the tasks, groups them by what they demand of the robot, and turns an open-ended ambition into a countable collection target. It is also what makes coverage auditable: without one you cannot answer whether your dataset is broad or merely large.
Group by capability, not by room
The instinct is to organize by location: kitchen tasks, bathroom tasks, bedroom tasks. This is intuitive and close to useless, because two kitchen tasks can demand entirely different capabilities while a kitchen task and a bedroom task demand identical ones.
| Capability group | Representative tasks | Dominant difficulty |
|---|---|---|
| Transport | Carrying items between rooms, tidying away | Navigation with a stable grip over distance |
| Rigid pick and place | Loading a dishwasher, shelving groceries | Clutter, occlusion, placement precision |
| Deformable handling | Folding laundry, making a bed, bagging | Object state is unpredictable and changes |
| Articulated objects | Doors, drawers, taps, appliance lids | Discovering the constraint through force |
| Surface work | Wiping, sweeping, spraying | Maintaining consistent contact force |
| Tool use | Cutlery, brushes, scoops | Grip must survive external forces |
| Granular and fluid | Pouring, scooping, filling | Material behavior is not rigid-body |
| Multi-step sequences | Making a drink, unpacking a bag | Progress tracking and error propagation |
Grouped this way, gaps become obvious. A program with strong pick-and-place coverage and nothing in articulated objects will fail at every door in the building, and the taxonomy shows that before the robot does.
What a task definition has to contain
A task name is not a task definition. Two operators handed “fold a towel” will produce different demonstrations unless the definition pins down five things.
- Entry state. Where the towel starts, how it is oriented, whether it is crumpled or flat.
- Sub-step sequence. Named phases with observable boundaries, so annotation gets segments for free.
- Graded success criteria. Not “folded” but a scale covering completion, neatness, and intervention count.
- Permitted variation. Which dimensions must change between episodes, and within what bounds.
- Abort conditions. When to stop and what to record, so failed episodes remain interpretable.
This is the same discipline covered in teleop task script design, applied at the level of the task list rather than a single script.
The dimensions that actually drive difficulty
Once tasks are grouped, rate each on the dimensions that determine how much data it will need. These are what your collection budget is really a function of.
| Dimension | Low | High |
|---|---|---|
| Horizon length | Seconds, single step | Minutes, many dependent steps |
| Object openness | Fixed known set | Unbounded, unseen instances |
| Deformability | Rigid throughout | State changes as you act on it |
| Contact richness | Grasp and release only | Continuous force regulation |
| Bimanual coupling | One hand suffices | Both hands on one object at once |
| Outcome ambiguity | Clearly pass or fail | Graded and partly subjective |
| Recovery complexity | Restart from scratch | Must adapt mid-task |
Tasks scoring high across several dimensions are not slightly harder, they are categorically harder, and they need the long-horizon treatment described in our long-horizon capture guide.
What we see in practice
In our everyday task capture for a humanoid robotics developer building general-purpose home and service robots, the taxonomy does more operational work than expected.
- It makes coverage a dashboard. Collection targets are set per capability group, so under-collected groups surface weekly rather than at the end of a quarter.
- It exposes the articulated-object gap early. Doors, drawers, and appliance lids are ubiquitous in homes and rare in benchmark datasets. Almost every program under-collects them.
- It reframes deformables as their own program. Laundry and food handling need different scoring, different session lengths, and different scene reset procedures from rigid pick-and-place.
- It makes partial episodes salvageable. Sub-step definitions mean a run that failed at step four still yields three usable segments.
The capture method behind this is covered in VR headset teleoperation for everyday tasks.
How to prioritise what to collect first
- Frequency in the deployment environment. Count how often the task genuinely occurs, not how impressive it looks in a demo.
- Capability reuse. Favour tasks whose skills transfer across many others. Articulated-object handling unlocks a great deal; a single novelty task unlocks nothing.
- Failure cost. Dropping laundry is inconvenient; dropping a full mug is not. Weight collection toward tasks where failure matters.
- Data availability. Some tasks have usable public coverage for pretraining. Spend your own budget where nothing exists.
- Collection feasibility. A task requiring a real appliance in a real kitchen is slower to stage than one that does not. Sequence accordingly.
Publish the resulting priority order and revisit it quarterly. Task lists that live in someone’s head drift toward whatever is easiest to collect.
Deformable handling is the capability group that most often lags. Our warehouse policy case study tracks success on deformable items from 61 to 84 percent.
Frequently asked questions
How many tasks should a taxonomy contain?
Enough to cover every capability group with several instances each, rather than a target number. A taxonomy with sixty tasks and one articulated-object entry is worse than one with twenty that is balanced.
Should the taxonomy match a clinical ADL scale?
Only if you are working in assistive or care settings where those scales are the customer’s language. For general home robotics, capability grouping serves data planning better.
Does this apply outside the home?
Directly. Hotel, retail, and eldercare environments are unstructured and cluttered in the same ways. The capability groups transfer; the specific task list does not.
How often should the taxonomy change?
Rarely, once settled. Frequent restructuring makes historical batches incomparable. Add tasks freely; regroup capabilities almost never.
A taxonomy costs a few days to build and determines what your entire collection budget buys. If you are scoping a home or service robot program and want the task list pressure-tested before you start collecting, send it to us.
