A language model learns from text. A robot brain has to learn from text, images, and the physical consequences of its own movements. Understanding VLA data requirements starts with what a chatbot never needed and a robot cannot do without.
This guide covers the three requirements unique to embodied models, the four data layers a VLA is built from, why language annotation is the layer teams underfund, and a minimum spec you can hold a programme to.
Three things a chatbot never needed
1. Actions attached to observations
Text models learn from text alone. A robot policy has to learn what to do, which means every observation needs the action that followed it. No amount of video without action labels produces a controller. This is the single hardest and most expensive requirement, and it is why first-person video is useful for pretraining and insufficient on its own.
2. Physical consequence
A language model can produce a wrong sentence and nothing breaks. A robot that grips too hard destroys the object. Your dataset therefore needs the signals that reveal consequence: contact forces, success and failure outcomes, and what happened after a mistake.
3. Embodiment specificity
Text is embodiment-free. Actions are not. A trajectory recorded on a seven-axis arm with a parallel gripper describes something a five-finger hand cannot execute, and vice versa. This is why cross-embodiment data is a research area rather than a solved problem.
The four data layers a VLA needs
| Layer | What it supplies | Where it comes from |
|---|---|---|
| Web vision and language | Object knowledge, common sense, semantics | Public pretraining corpora |
| Egocentric human video | How tasks look from a first-person view | Public sets plus wearable capture |
| Robot demonstrations | Grounded action labels on your embodiment | Teleoperation; your budget lives here |
| Language annotations | What each episode and sub-step was for | Task scripts plus annotation |
Teams routinely fund layer three and skip layer four, then wonder why the model does not follow instructions reliably.
Language annotation is the underfunded layer
An episode labelled “pick and place” is nearly useless for instruction following. What a VLA needs is closer to how a person would actually describe the task, at several levels at once.
- High level. “Clear the breakfast table.”
- Sub-task. “Put the cereal bowl in the dishwasher.”
- Primitive. “Grasp the bowl by the rim.”
- Paraphrases. Several natural phrasings of each of the above.
- Referring expressions. “The blue one”, “the one nearest you”, “the empty mug.”
- Negative and corrective phrasing. “Not that one, the other one.”
The referring expressions matter more than they look. Real instructions rarely name objects unambiguously, and a model trained only on clean names cannot resolve “the one behind the kettle.” Much of this can be generated at collection time from a well-built task script, which is why task script design and VLA data quality are the same problem.
Diversity requirements, ranked
Given a fixed budget, spend in roughly this order.
- Object instances. Many examples per category, not many categories with one example each.
- Instruction phrasing. Cheap to vary, disproportionately valuable.
- Scene configuration. Layout, clutter, lighting, surface.
- Task compositions. Sequences that combine known primitives in new orders.
- Operators. Behavioural diversity, and a check on script ambiguity.
- Embodiments. Valuable if you can afford it, and the hardest to arrange.
Repetitions of an identical episode sit below all of these once you are past the learning floor described in behavior cloning training data.
What we see in household capture
In our everyday task work for a humanoid robotics developer building general-purpose home and service robots, the language layer is where most of the recoverable value sits.
Household instructions are ambiguous by nature. “Put that away” depends on what “that” is and where it belongs, and both vary by household. Capturing the instruction as it was actually given, alongside the resolved target, gives the model paired ambiguity and resolution rather than a sanitised label. That pairing is difficult to reconstruct after the fact and nearly free to record at the time. The wider programme is covered in VR headset teleoperation for everyday tasks.
For downstream results from a programme built this way, our humanoid foundation model case study reports a 3x sample-efficiency gain.
A minimum specification
If you are writing a data spec for a VLA programme, it should require all of the following per episode.
- Synchronized multi-view video, including at least one wrist view
- Action labels in one stated convention, with measured control rate
- Proprioceptive state and gripper state
- Force and torque on any contact-rich task
- A natural-language instruction, plus at least two paraphrases
- Sub-step boundaries with their own short descriptions
- Outcome, failure mode, and retry count
- Object instance identifiers and scene condition metadata
Anything less and you will be able to train something, just not the thing you budgeted for.
Frequently asked questions
How many episodes does a VLA fine-tune need?
Far fewer than training from scratch, and more than teams hope. The driver is task and object variety rather than raw count, so a spec with fifty objects and modest depth usually beats one with five objects and heavy repetition.
Can we generate language annotations automatically?
Partly. A captioning model can produce a first pass, and it will describe what it sees rather than what was intended. Instructions given before the episode are more valuable than descriptions written after it.
Do we need multiple robots to train a VLA?
Not to train one. Pooling across embodiments improves generalization, but a single-embodiment dataset with good breadth is a perfectly reasonable starting point.
Is public robot data enough to skip collection?
It is enough to start and not to finish. Public corpora rarely match your embodiment, your objects, or your environment, so most teams pretrain broadly and fine-tune on their own capture.
The architecture is increasingly something you can download. The grounded, language-annotated demonstrations for your robot are not. If you want a data specification pressure-tested against a VLA target before you start collecting, send us the plan.
