Every humanoid programme eventually asks what the leading teams are actually collecting. The honest answer is that humanoid robot training datasets look less like bimanual manipulation data than most teams expect, because the whole body is part of the action.
This guide covers why humanoids are not simply arms on legs, what the streams have to include, where collection volume actually goes, and the gaps that show up in deployment.
Humanoids are not just arms on legs
A bimanual manipulation dataset covers two arms working on a table. A humanoid dataset has to cover a body that walks to the table, adjusts its stance, leans, reaches past its own torso, and looks around while doing all of it.
That changes the data requirement in three ways. The camera moves independently of the hands. The base position becomes part of the action. And whole-body posture affects what is reachable, so the same task from a slightly different stance is a genuinely different problem.
Programmes that scope a humanoid effort as bimanual manipulation plus locomotion consistently under-collect the part in between.
What the leading programmes actually record
| Stream | Why humanoids specifically need it |
|---|---|
| Head camera | Moves with gaze; the policy’s primary view |
| Wrist cameras, both arms | Torso occlusion is constant, per viewpoint trade-offs |
| Whole-body joint state | Legs and torso change what the arms can reach |
| Base pose and locomotion state | Where the robot stood is part of the demonstration |
| Centre of mass and balance | Reaching affects stability; stability constrains reaching |
| Force and torque at both wrists | Contact quality, per force capture |
| Head pose | Grounds a moving camera in space |
| Task and sub-task labels | Humanoid tasks are long-horizon by nature |
The balance and base-pose rows are what separate a humanoid dataset from a bimanual one. Without them you cannot tell why a reach failed: wrong arm trajectory, or wrong place to stand.
Where the volume goes
Humanoid programmes tend to distribute effort roughly like this once they are past the first year.
- Manipulation primitives. Grasping, placing, opening, pouring, across many object instances. The largest single block.
- Long-horizon sequences. Multi-step tasks with dependencies, per long-horizon capture.
- Mobile manipulation. Approach, position, act, which is where base pose earns its place.
- Recovery and failure. Dropped objects, missed grasps, and mid-task adaptation.
- Whole-body reaching. Low shelves, high shelves, awkward angles.
- Human interaction. Handovers and shared spaces, for service deployments.
The common gap is the fifth. Tabletop-height data is easy to collect and does not teach a robot to reach into a bottom drawer.
Teleoperating a humanoid
Leader-follower rigs map cleanly onto arms and not onto legs, torso, or head. That is why humanoid programmes lean on headset-driven control, where the operator sees through the robot’s head and hand tracking drives the arms.
It solves whole-body capture and introduces its own problems: no force feedback, so operators over-grip, and retargeting error between human and robot hands. Both are covered in VR headset teleoperation for everyday tasks.
Most mature programmes run both, using a leader-follower cell for precision tabletop work and headset control for anything mobile or long-horizon. The rig comparison is in GELLO vs ALOHA.
The four gaps that show up later
- Height variation. Everything collected at bench height, then the robot meets a floor-level task.
- Stance diversity. The same task always performed from the same standing position, so the policy never learns to position itself.
- Failure and recovery. Discarded during collection, then needed at deployment, per behavior cloning training data.
- Deformables. Cloth, packaging, and food are the bulk of real service work and a minority of most datasets.
For a programme that got the mix right, our humanoid foundation model case study documents a 3x sample-efficiency result.
How much is enough
Volume follows from task variety, not the reverse. Find the learning floor on one fixed configuration in a pilot, then multiply across your variation dimensions, which for a humanoid include object instance, height, stance, and scene layout.
That multiplication is why humanoid programmes are expensive. Four dimensions compound quickly and each is legitimate. Our volume guide and cost model work through the arithmetic.
Frequently asked questions
Can we use bimanual arm data for a humanoid?
For manipulation primitives, largely yes. It teaches nothing about stance, balance, or positioning, which is where humanoid-specific failures concentrate.
Does data transfer between humanoid platforms?
Visual and task-structure learning transfers reasonably. Joint-space actions do not, because kinematics differ, per cross-embodiment data.
Should we collect locomotion and manipulation separately?
Collect them together wherever the task involves both. Splitting them produces a robot that walks well and manipulates well and cannot do the transition, which is the part that matters.
How many operators should a humanoid programme use?
Enough for behavioural diversity, tracked per operator so variance is measurable rather than assumed, per operator quality.
Humanoid data is the most expensive category to collect and the least forgiving of narrow scoping. If you want a programme sized against a realistic variation model, tell us what you are building.
