“Pick up the red mug” is the canonical robotics demo, and it hides most of the problem. Real instructions are vaguer, shorter, and far more dependent on context. Language-conditioned manipulation data is what determines whether a robot handles the real version or only the demo.
This guide covers the four kinds of reference people actually use, why instructions must be captured rather than labelled afterwards, what household capture reveals about ambiguity, and how to judge whether your language layer is any good.
Why instructions are harder than they look
“Pick up the red mug” contains at least four problems. Which object is a mug. Which of them is red. Whether “the” implies there is exactly one. And what “pick up” means for this gripper on this object.
A model gets the first two largely free from language pretraining. The last two only come from grounded demonstrations. And the third, reference resolution, is the one that quietly breaks systems in the field, because real instructions are far more ambiguous than benchmark ones.
The four kinds of reference people actually use
| Reference type | Example | What it requires |
|---|---|---|
| Direct naming | “the mug” | Object recognition alone |
| Attribute | “the blue mug” | Attribute grounding and comparison |
| Spatial relation | “the mug behind the kettle” | Scene geometry and viewpoint reasoning |
| Contextual or deictic | “that one”, “the empty one” | State tracking and shared context |
Benchmarks are dominated by the first two rows. Real households and warehouses run heavily on the bottom two. A dataset that only contains clean naming produces a robot that works in demos and stalls in kitchens.
Collect the instruction, not a label
The most common mistake is writing task labels after the fact. Someone reviews a recorded episode and types “pick up mug, place in sink.” That is a description, not an instruction, and it differs from real language in three ways: it is unambiguous, it is written in a consistent style, and it describes what happened rather than what was wanted.
The fix is to capture the instruction before the episode and record it verbatim. It costs nothing extra if the task script is built for it, which is one more reason task script design determines dataset quality.
What to record per episode
- The instruction exactly as given, including any vagueness
- The resolved target, recorded as an object instance identifier
- Two or more natural paraphrases of the same instruction
- Sub-step instructions with their own boundaries
- Any clarification exchanged before or during the episode
- Whether the instruction was ambiguous, flagged explicitly
The last two are the valuable ones and the ones nobody collects. A model that has never seen a clarification cannot learn to ask for one.
What household capture reveals
In our everyday task work for a humanoid robotics developer building general-purpose home and service robots, instruction ambiguity is not an edge case. It is the normal condition.
“Put that away” requires knowing what “that” is and where it belongs, and where it belongs differs per household. “Clear the table” has no defined stopping point. “Be careful with that one” modifies how rather than what. None of these resolve from the instruction alone; they resolve from context the robot has to carry.
Two practices help. Record the pre-episode instruction verbatim alongside the resolved target, so the model gets ambiguity and resolution as a pair. And vary who gives the instruction, because phrasing habits are strongly personal and a dataset written by one person teaches that person’s idiolect. The wider programme sits in VR headset teleoperation for everyday tasks.
Granularity: the level that trains best
Instructions can be pitched at very different levels, and the level changes what the model learns.
- Goal level. “Tidy the kitchen.” Excellent for planning, useless for control on its own.
- Task level. “Put the bowl in the dishwasher.” The most useful single level for most policies.
- Primitive level. “Close the gripper.” Precise, and it teaches nothing about intent.
Collect at all three where you can, linked by sub-step boundaries. That structure lets a model decompose a goal into tasks and tasks into motion, which is what long-horizon behavior requires. The methodology is in long-horizon data capture.
How to tell if your language data is any good
- Count unique phrasings per task. If it is close to one, the model is learning strings rather than meaning.
- Measure the share of non-naming references. Low share means the dataset is easier than reality.
- Check who wrote the instructions. One author is a bias you can see in the model.
- Look for negatives. “Not the big one” should appear somewhere.
- Confirm instructions predate episodes. If they were written during review, they are descriptions.
For an outcome from a programme with this structure, our warehouse policy case study tracks deformable-item success from 61 to 84 percent.
Frequently asked questions
Can we add language annotations to an existing dataset?
Partly. You can describe what happened, and you cannot recover the instruction that was intended or the ambiguity that was present. Retrofitted language is usable and weaker than captured language.
Should instructions be in one language?
Start with one and record the language per episode. Multilingual instruction data is valuable for deployment across markets, and mixing it in without tagging makes evaluation confusing.
How many paraphrases per task are enough?
Two or three natural variants per task usually captures most of the benefit. Beyond that, effort is better spent on more objects and scenes.
Do we need a language model in the loop at collection time?
No, and it can hurt. Generated instructions are cleaner and more uniform than human ones, which is precisely the distribution you are trying to avoid.
Language is the cheapest layer in a robot dataset and the one most often added last, when it can no longer be captured properly. If you are specifying a programme and want the instruction layer designed in from the start, tell us what you are building.
