Ask ten operators to “pick up the cup and put it on the shelf” and you will get ten different demonstrations. Some grasp the handle, some the body. Some approach from the front, some the side. Good teleoperation task script design is what turns that variance into something a model can learn from instead of something it has to average away.
This guide covers the seven parts of a workable script, how to write for operators rather than engineers, how to specify variation instead of merely permitting it, and how to test a script before you scale headcount behind it.
What a task script is for
A task script is the document that tells an operator exactly what to do, under what conditions, and what counts as done. It exists to make demonstrations comparable. Without it, twenty operators produce twenty subtly different tasks and the model has to average across all of them.
It is also the artifact that makes a program auditable. When an enterprise buyer asks how you guarantee consistency, the honest answer is the script and the QA that enforces it.
The seven parts of a workable script
| Section | What it specifies | Failure if omitted |
|---|---|---|
| Task ID and variant | Stable identifier and version | Batches cannot be compared across weeks |
| Entry conditions | Exact starting scene state | Episodes start from incomparable states |
| Object set | Instances, not categories | Diversity cannot be audited afterwards |
| Required variation | What must change between episodes | Model memorizes one staging |
| Sub-step sequence | Named phases with boundaries | No segmentation labels; annotation cost balloons |
| Success grading | Graded criteria, not a boolean | Partial successes get discarded |
| Stop and escalate rules | When to abort and what to record | Operators improvise; data becomes unexplainable |
Notice how many of these exist to serve annotation and QA rather than the operator. That is deliberate. A script written only for the operator produces demonstrations nobody can label.
Write for the operator, not for the engineer
The most common failure is a script written in the vocabulary of the ML team. Operators are task experts, not roboticists. If the script says “achieve stable grasp with minimal wrench,” it will be interpreted seven different ways.
Rules that work
- One instruction per line, in the imperative. “Pick up the mug by the handle” beats a paragraph describing grasp strategy.
- Specify observable conditions. “The drawer is fully closed” is checkable. “The drawer is properly closed” is not.
- Give the reason for anything counter-intuitive. Operators who understand why a rule exists follow it under time pressure. Operators who do not, will not.
- Show the failure cases. Two or three photographs of what a bad episode looks like beat a page of prose.
- Keep it to one screen where possible. A script that must be scrolled during a session will not be read during a session.
Specify variation, do not just permit it
This is the single highest-value section and the one most often left vague. “Vary the setup between episodes” produces almost no variation, because under throughput pressure operators default to whatever is fastest.
Instead, enumerate the variation dimensions and require coverage:
- Object instance – rotate through a named set, not a category
- Position and orientation – defined zones and rotation ranges
- Quantity and clutter – a specified distribution, not “some”
- Lighting – named conditions the operator can actually set
- Starting distance and approach angle – often forgotten, quietly important
- Distractor objects – present in a defined fraction of episodes
Then track coverage as a metric, not an aspiration. A dashboard showing which combinations are under-collected turns diversity into an operational target. This is what separates real long-tail capture from wishful thinking.
Long-horizon tasks need sub-step scripts
A five-minute household task cannot be scripted as one instruction. Break it into named sub-steps, each with its own entry condition and success criterion, and have the recorder mark the boundaries.
In our household capture work for a humanoid robotics developer building general-purpose home and service robots, this structure does three things at once. It gives annotation free segment boundaries. It lets QA identify which sub-step a failure occurred in. And it makes partial episodes salvageable, because a run that failed at step four still contains three good steps. The wider methodology is in long-horizon data capture and the program detail in VR headset teleoperation for everyday tasks.
Version the script like code
Scripts change. A rule gets clarified, a variation dimension gets added, a success criterion gets tightened. If the version is not recorded in the episode log, you end up with a dataset where two batches followed different rules and nothing in the data says so.
- Give every script a version number and change log.
- Write the version into every episode record, as covered in demonstration log design.
- Never edit a live version in place. Publish a new one.
- Re-brief operators on change, and mark the first session after a change for review.
- Keep retired versions. You will need them to explain an old batch.
How to test a script before you scale it
Run a pilot with three or four operators who have not seen it before. Do not explain it verbally, because verbal explanation is exactly what will not exist at scale.
Then compare their episodes. Where they diverge, the script is ambiguous. Where they all made the same mistake, the script is wrong. Fix both before adding operators, because ambiguity multiplies with headcount. Our operator quality guide covers how to measure the divergence.
Script discipline shows up directly in outcomes. Our warehouse policy case study tracks deformable-item success from 61 to 84 percent.
Frequently asked questions
How detailed should a task script be?
Detailed enough that two operators who have never met produce comparable episodes, and no more. Over-specification suppresses the natural variation that helps a policy generalize.
Should operators be told the ML objective?
Broadly, yes. Operators who understand that failures are valuable stop hiding them. Operators who think they are being scored on success rate quietly delete the most useful data you have.
Who should write the script?
A collaboration. The ML team owns success criteria and variation requirements; the operations team owns feasibility and phrasing. Scripts written by either alone tend to fail in predictable ways.
How often should scripts be revised?
Frequently early, rarely once stable. Expect several revisions during pilot and near-zero during production. A script still changing weekly at scale is a sign the task definition is not settled.
The task script is the cheapest quality control instrument in a data program and the one most often improvised. If you want yours reviewed before you scale operator headcount, send us the task list.
