First-person video shows what someone saw. It does not say where they were standing or which way they were facing, and without that a model cannot tell whether the world moved or the camera did. Ego-pose estimation data is what supplies the missing half.
This guide explains what six-degree-of-freedom pose is, what it unlocks that video alone cannot, the five ways it is captured, why drift is the failure mode that matters, and how to record it so it stays usable.
The problem ego-pose solves
A head-mounted camera produces video where everything moves. When the wearer turns, the whole frame sweeps. A model looking only at pixels cannot easily tell whether the mug moved or the head did.
Ego-pose is the answer: a continuous record of where the camera was and which way it faced, in six degrees of freedom, at every moment. Once you have it, camera motion can be separated from scene motion, and frames captured seconds apart can be placed in the same 3D space.
Six degrees of freedom, plainly
Three for position, where the camera is. Three for orientation, which way it points and how it is tilted. Together they place the camera precisely in a room. Without orientation, position alone tells you almost nothing useful about what was visible.
What ego-pose unlocks
| Capability | Why pose is required |
|---|---|
| Separating self-motion from scene motion | Otherwise every head turn looks like the world moving |
| 3D reconstruction of the workspace | Frames must be registered to a common frame |
| Persistent object memory | Knowing an object is still there when you look away |
| Retargeting human capture to a robot | Pose is the bridge between two bodies |
| Reach and distance reasoning | Pixel size alone is ambiguous about distance |
| Pooling sessions recorded weeks apart | A shared frame makes them comparable |
The persistent memory point matters most for long-horizon work. A robot tidying a room needs to remember what was on the counter after it has turned to the sink, and that is a pose problem before it is a memory problem.
How pose is actually captured
- Headset tracking. VR and AR headsets compute their own pose continuously and to a good standard. If you are already using headset teleoperation, you have ego-pose for free and should be logging it.
- Visual-inertial odometry. Camera plus IMU, estimating pose from motion. Cheap and prone to drift over long sessions.
- External motion capture. Highest accuracy, confined to an instrumented space, and unusable in a real home or warehouse.
- Fiducial markers. Printed tags at known positions, giving accurate correction where they are visible. Practical for periodic re-anchoring.
- Robot kinematics. On a robot, forward kinematics gives camera pose directly from joint angles, provided calibration is current.
The last one is the reason robot-embodied capture is cleaner than wearable capture: pose comes from the same system that produced the actions, on the same clock.
Drift is the failure mode that matters
Every pose estimate that integrates motion accumulates error. Small per-frame errors compound into a position that is metres wrong by the end of a long session, and the trace looks perfectly smooth throughout. Nothing in the data announces the problem.
Three defences are worth building in from the start.
- Re-anchor periodically. A marker or known landmark visible at intervals lets you correct rather than accumulate.
- Return to a start pose. End each session where it began and measure the closing error. That single number tells you the quality of the whole session.
- Log confidence, not just pose. Most trackers expose an uncertainty estimate. Recording it lets you filter later instead of discarding whole sessions.
Store the closing error per session in your episode metadata, as with the other quality signals in demonstration log design.
Recording it properly
- Sample above video rate. Pose at 60 to 200 Hz interpolates cleanly onto frames; pose at frame rate does not.
- State the convention. Which axis is up, whether rotations are quaternions or Euler angles, and in which order. Assumed conventions are a leading cause of unusable pooled data.
- Record the reference frame origin per session, and the transform to any shared frame.
- Timestamp on the same clock as video and actions. Pose offset from frames is worse than no pose, because it looks valid.
- Keep raw tracker output alongside any smoothed version. Filtering is a modelling decision that belongs downstream.
Paired viewpoints benefit from the same rigour. Our aerial perception case study reports a 3x detection rate from paired drone-and-ground capture, which only works when both sources are registered to a common frame.
Frequently asked questions
Do we need ego-pose if we only train a vision policy?
Not strictly. A vision-only policy can be trained without it. You lose 3D reasoning, cross-session pooling, and the ability to diagnose whether a failure was perception or positioning.
Is IMU data enough on its own?
No. An IMU gives you acceleration and rotation rate, which drift badly when integrated into position. It is an excellent complement to visual tracking and a poor substitute for it.
How accurate does pose need to be?
It depends on the task scale. Reaching and navigation tolerate centimetre-level error. Fine manipulation needs better, which is usually why wrist cameras carry the final approach instead, as covered in wrist-cam vs head-cam.
Can pose be estimated after the fact from video?
Yes, with structure-from-motion, and it is slow, imperfect, and fails on textureless scenes. Capturing pose live is far cheaper than reconstructing it.
Ego-pose is the layer that turns first-person video from a picture into a measurement. It is nearly free to record while capturing and expensive to reconstruct afterwards. If you are specifying an egocentric programme, tell us what you are building and we will check the pose stack against it.
