Most robot manipulation data is collected on a bench, with a fixed camera, a clean table, and a small set of known objects. Real homes are none of those things. VR headset teleoperation has become the practical way to capture the messy, multi-step tasks a general-purpose robot will actually face.
This guide covers how headset-driven capture works, why everyday household tasks are harder to collect than lab benchmarks, what we changed operationally to make it work at scale, and where the method breaks down.
How headset teleoperation works
The operator wears a VR headset. Inside it they see the robot’s own camera feed, usually a head-mounted stereo pair, rendered live. Their hand and finger motion is tracked and retargeted onto the robot’s arms and grippers. When they reach for a cup, the robot reaches for the cup.
Two properties make this valuable for data collection.
The recording is egocentric by construction
The operator is literally looking through the robot. Every frame recorded is a frame the policy could plausibly see at deployment, with no viewpoint mismatch to correct for. This is the cleanest form of egocentric data collection available.
The learning curve is short
Leader-follower rigs require training on the rig. A headset requires training on the task. New operators reach useful output far faster, which changes the economics of scaling a dedicated team.
Why everyday tasks are the hard ones
We run household task capture for a humanoid robotics developer building general-purpose home and service robots. The task list looks unglamorous: loading a dishwasher, folding laundry, wiping a counter, putting groceries away, tidying a table.
These are far harder to collect well than the tabletop benchmarks that dominate published work, for four reasons.
| Property | Lab benchmark task | Everyday household task |
|---|---|---|
| Duration | Seconds | Minutes, with many sub-steps |
| Object set | Fixed, known | Open-ended, unseen instances constantly |
| Scene state | Reset identically | Never the same twice |
| Success definition | Binary | Graded and often subjective |
| Failure recovery | Restart | Adapt mid-task, which is the actual skill |
| Deformables | Rare | Constant: cloth, food, packaging |
The long duration is the biggest operational difference. A folding sequence is not one demonstration, it is a chain of dependent sub-tasks where an early error propagates. That makes it a long-horizon capture problem, not a manipulation problem.
What we changed to make it work
Session design around comfort, not throughput
Headset sessions cannot run like keyboard shifts. Visual fatigue arrives before the operator reports it, and quality degrades before either party notices. Shorter blocks with mandatory breaks produced more usable trajectories per shift than longer blocks, even though raw episode count fell. This mirrors what we see across teleoperation generally: fatigue is a data quality variable, not an HR one.
Deliberate scene variation as a standing rule
Household scenes reset naturally, which is an advantage nobody exploits. Instead of restoring a fixed layout between episodes, operators are instructed to vary object placement, quantity, and orientation within defined bounds. Diversity becomes free rather than expensive, which is the core argument in long-tail and edge-case capture.
Graded outcomes instead of pass or fail
“Did the laundry get folded” is not answerable with a boolean. Episodes are scored on completion, tidiness, and intervention count, so partial successes remain trainable rather than being discarded.
Wrist cameras recording throughout
The operator works from the head view, but wrist streams record silently alongside. Household clutter causes constant arm occlusion, and the wrist view recovers episodes that would otherwise be unusable. The trade-off is covered in wrist-cam vs head-cam.
Where headset control struggles
It is not the right tool for everything, and pretending otherwise costs you a re-collection.
- Force regulation. Hand tracking gives no resistance feedback at all. Operators over-grip. Instrumenting the follower with force-torque sensing is mandatory, not optional.
- Fine alignment. Sub-centimeter insertion is easier on a leader-follower rig where the operator’s hand is mechanically constrained.
- Latency sensitivity. A headset makes delay far more noticeable than a screen does, and jitter causes discomfort as well as bad data. See latency budgets.
- Retargeting error. Human hands are not robot grippers. The mapping is approximate, and the approximation is recorded as if it were intent.
Most mature programs run both: headset control for long-horizon and mobile tasks, a leader-follower cell such as ALOHA for precision tabletop work.
Setting up a household capture program
- Write the task taxonomy first. Not “kitchen tasks” but a defined list with entry conditions, sub-steps, and graded success criteria.
- Build realistic environments. Real kitchens and laundry areas, not a table with props. The clutter is the point.
- Fix the log schema before episode one, including sub-task boundaries, per our demonstration log guidance.
- Instrument force from the start. Retrofitting means re-collecting.
- Design shifts around visual fatigue, with rotation and mandatory breaks.
- Score every episode, keep every episode. Failures and partial completions carry the recovery behavior you need most.
[Insert programme metrics here: trajectories per week, usable-episode rate, task coverage.]
For what this kind of capture produces downstream, our humanoid foundation model case study documents a 3x sample-efficiency result.
Frequently asked questions
Is VR teleoperation better than a leader-follower rig?
Not better, different. Headsets win on long-horizon, mobile, and whole-body tasks and on operator ramp time. Leader-follower rigs win on precision and force fidelity.
Do operators need robotics experience?
No, and that is much of the appeal. They need task competence and consistency. Ramp time is measured in days rather than weeks, which is why this model scales geographically.
How long can an operator work in a headset?
Shorter than you expect. Plan in blocks with breaks and measure quality by block rather than by shift, because degradation shows up in the data before it shows up in the operator’s own report.
Does household data transfer to commercial service robots?
Substantially. Household environments are unstructured, cluttered, and full of deformables, which makes them a strong proxy for hotel, retail, and eldercare settings. Task-specific fine-tuning is still required.
Everyday tasks are where general-purpose robots will either succeed or quietly fail, and they are the hardest thing to collect properly. If you are standing up a household or service capture program, tell us what you are building and we will map the operating model against your task list.
