If you want to know whether a home robot actually works, put it in a kitchen. Kitchen task robotics data is the hardest category to collect well because a kitchen combines deformables, liquids, heat, clutter, long horizons, and success criteria that resist definition.
This guide covers why kitchens beat every standard benchmark for difficulty, where policies actually fail, what has to be captured, and how to define success for tasks where “done” is a judgement call.
Why kitchens beat every benchmark for difficulty
Robotics benchmarks tend to feature blocks, cubes, and rigid household objects on clean tables. A kitchen has almost none of those properties, and it combines several hard problems at once rather than isolating one.
- Materials change state. Dough, batter, and vegetables are different objects before and after you act on them.
- Liquids and granulars. Pouring and scooping are not rigid-body problems and are poorly served by most simulation.
- Deformables everywhere. Bags, cloths, wrapping, food itself.
- Transparent and reflective surfaces. Glass and chrome defeat most depth sensors, per RGB-D guidance.
- Heat and mess. Consequences of failure are real, and cleaning is part of the task.
- Long horizons with dependencies. A missed step early ruins everything after it.
- Subjective success. “Chopped finely enough” is a judgement, not a boolean.
Any one of these is a research problem. A kitchen presents all of them simultaneously, which is why it is the honest test of a general-purpose home robot.
Where policies actually fail
| Task | Failure mode |
|---|---|
| Loading a dishwasher | Placement collisions in dense clutter |
| Pouring | Flow rate and stopping point; no rigid-body analogue |
| Opening packaging | Requires force discovery and often two hands |
| Handling produce | Grip force varies by ripeness, invisible to vision |
| Wiping a surface | Sustained contact force over a changing area |
| Using cutlery and tools | Grip must survive external forces from the task |
| Putting shopping away | Open-ended objects and household-specific destinations |
Notice how many are force problems rather than perception problems. Kitchen data without force and torque capture will train a robot that looks competent and crushes things.
Grading success when “done” is subjective
Binary success labels break immediately in a kitchen. A dishwasher can be loaded badly but functionally. Vegetables can be chopped unevenly and still usable.
Graded criteria work better, and they need to be defined per task before collection begins.
- Completion. Was the intended end state reached at all?
- Quality. Against a stated standard, with reference photographs rather than adjectives.
- Damage. Anything crushed, spilled, or broken, scored independently of completion.
- Interventions. How many times a human had to help.
- Duration, relative to a human baseline.
Reference photographs are the single most useful artifact here. “Neatly stacked” means five different things to five annotators; a photograph means one. This is the task-script discipline described in task script design.
What we see in household capture
We run everyday task capture for a humanoid robotics developer building general-purpose home and service robots, and kitchen work is consistently the most demanding part of the task list.
- Scene reset is expensive. Unlike a bench task, a kitchen episode changes the world. Resetting takes longer than the demonstration in some cases, and that belongs in the cost model, per cost per trajectory.
- Variation is free if you let it be. Kitchens do not reset to identical states naturally. Rather than fighting that, instruct operators to vary placement and quantity within bounds, and diversity stops being a cost.
- Partial episodes are worth keeping. A run that failed at step four still contains three usable sub-tasks, provided sub-step boundaries were labelled.
- Wrist cameras recover episodes. Kitchen clutter causes constant arm occlusion; the head view alone loses the contact moment.
The wider programme is covered in VR headset teleoperation for everyday tasks and the task structure in activities of daily living.
For a comparable programme that reached production quickly, our mobile manipulation case study covers a 90-day path from cold-start to production.
Frequently asked questions
Can kitchen tasks be learned in simulation?
Partially. Navigation and rigid pick-and-place transfer reasonably. Liquids, granulars, and deformables are where simulators diverge most, so those need real capture, per sim-to-real planning.
How many kitchens do we need?
More than one, and the number matters less than the variety. Layout, appliance type, lighting, and clutter level all shift the distribution, and a single well-instrumented kitchen produces a policy that works in that kitchen.
Do we need real food?
For anything involving deformation, ripeness, or mess, yes. Props behave differently, and the difference is exactly the property the policy needs to learn.
Is kitchen data relevant to commercial deployments?
Strongly. Hotel, catering, and eldercare settings share the clutter, deformables, and open-ended object sets, so kitchen data transfers better than most verticals.
Kitchens are where general-purpose robots will either work or quietly fail, and they are the hardest environment to collect properly. If you are scoping a home or service programme, tell us what you are building.
