Two years ago cross-embodiment data was a niche research topic. Now every robotics roadmap depends on it. The reason is that foundation robot model data has to be broad, and no single company owns enough robots to supply that breadth alone.
This guide covers what actually makes a model a foundation model, why breadth forces the cross-embodiment problem, exactly which layers transfer between robots and which do not, and what all of it demands of the way you record data.
What makes a model a foundation model
Not size. The defining property is that it is trained broadly once and adapted many times. You pretrain on a wide corpus, then fine-tune cheaply for each specific task, robot, or environment.
For robotics that changes the economics. Under the old model, every new task meant a new dataset and a new policy, and the cost per task stayed roughly flat. Under a foundation model, the first task is expensive and each subsequent one gets cheaper. That is the entire commercial argument, and it only holds if the pretraining corpus is broad enough.
Why breadth has to mean cross-embodiment
In language, breadth is easy. Text is text. In robotics, breadth runs into a wall: a trajectory is tied to the robot that produced it.
Every organisation has a limited number of robots, so a single-embodiment corpus hits a ceiling set by how many hours that hardware can run. Pooling across robot types is the only way past it. That is why cross-embodiment data moved from an academic curiosity to the central practical problem in the field.
What transfers and what does not
| Layer | Transfers across robots | Why |
|---|---|---|
| Visual representation | Very well | A mug looks the same to any camera |
| Object affordances | Well | Handles are for grasping regardless of gripper |
| Task structure | Well | Sequence and sub-steps are embodiment-agnostic |
| End-effector trajectories | Partially | Needs a shared frame and normalization |
| Joint-space actions | Poorly | Kinematics differ; the numbers mean different things |
| Contact and force strategy | Poorly | Depends on gripper geometry and compliance |
The practical read: pool aggressively at the top of that table, cautiously in the middle, and not at all at the bottom without deliberate retargeting.
What this demands of your data
Cross-embodiment pooling is mostly a metadata discipline. Six things have to be true.
- Embodiment is recorded per episode. Robot model, gripper, degrees of freedom, payload. Without it, pooling is guesswork.
- One canonical action representation with per-source transforms, as covered in trajectory formats.
- A shared coordinate frame convention, with the transform from each robot’s frame recorded.
- Units stated, never assumed. Mixed radians and degrees across sources is the quietest and most common corruption.
- Camera geometry per source. Intrinsics, extrinsics, and mounting position, so viewpoint differences can be modelled rather than absorbed as noise.
- Source tags on every episode, so you can weight, ablate, and exclude by origin.
None of this is expensive at capture time. All of it is close to impossible to reconstruct afterwards, which is the recurring theme in dataset structure.
The uncomfortable strategic question
If breadth is what makes foundation models work, and no single company has enough robots to supply it, then either you pool with others or you buy breadth from someone who has already assembled it.
Public cross-robot corpora exist and are genuinely useful for pretraining. They also contain almost nothing resembling your specific embodiment, objects, or environment. The realistic pattern is layered: public data for representation, purchased or partner data for breadth, your own capture for grounding.
Deciding that split is a budget question rather than a research one, and our cost-per-trajectory model is the right frame for it. Our humanoid foundation model case study documents a programme built on that layered approach reaching 3x sample efficiency.
Where the tail comes from
Broad pretraining gives a model competence across many situations. It does not give it the specific failures your deployment will produce, because those are not in anyone else’s dataset.
The cheapest source of that tail is a fleet already running. We generate it continuously for an autonomous mobility company operating sidewalk delivery robots and self-driving passenger vehicles, where every remote intervention records a situation the policy could not handle. That is a sample from the true failure distribution rather than a guess at it, and it is covered in closing the loop from intervention to training data.
Frequently asked questions
Do we need a foundation model at all?
If you run one task on one robot in one environment, almost certainly not. A narrow policy is cheaper and easier to debug. Foundation models earn their cost when the task list keeps growing.
How much data does pretraining need?
More than any single team collects alone, which is the point. Most organisations adopt an existing pretrained model and spend their budget on fine-tuning rather than attempting pretraining themselves.
Does pooling data from a different robot hurt performance?
It can, if joint-space actions are pooled naively across incompatible kinematics. Pooled at the representation and task-structure level with embodiment tags retained, it generally helps.
What is the minimum metadata for future pooling?
Embodiment identifiers, action convention, coordinate frame, units, control rate, and camera calibration. Record those from episode one even if you have no pooling plans yet.
The bet behind foundation models is that breadth bought once pays off across every task afterwards. That only works if your data was recorded so it can be pooled. If you want your schema reviewed against that requirement, send us the spec.
