Ask a teleoperation team what their latency is and you will usually get one number. Ask what their teleoperation latency budget is per task class, and the room goes quiet. That gap is where a lot of unusable demonstration data comes from.
This guide breaks the loop into its stages, explains why jitter damages data more than delay does, sets out tolerance by task type, and gives you a method for building a latency budget you can actually defend.
What “latency” means in a teleoperation loop
Latency is not one number. It is a chain, and each link adds delay before the operator sees the consequence of their own motion.
| Stage | What happens | Usual culprit |
|---|---|---|
| Sensor capture | Camera exposes and reads out a frame | Low frame rate, long exposure in dim rooms |
| Encode | Frame is compressed for transport | Quality settings tuned for file size, not delay |
| Transport out | Frame travels to the operator | Network hops, congestion, wireless contention |
| Decode and display | Operator’s screen or headset renders it | Buffering, display refresh, compositor delay |
| Human reaction | Operator perceives and responds | Fixed biology; roughly a fifth of a second |
| Command transport | Input travels back to the robot | Usually small, but jitter here is brutal |
| Actuation | Robot executes | Controller loop rate, mechanical compliance |
Add these together and you have glass-to-glass latency, the number that actually matters. Measure it directly with a clock in frame rather than summing component specs, because component specs consistently understate reality.
Jitter hurts more than latency
This is the finding that surprises most teams. Operators adapt remarkably well to consistent delay. Within a session or two, they learn to lead their inputs and performance recovers.
What they cannot adapt to is delay that varies. If the loop runs at one figure for a minute and then doubles for three seconds, the operator’s internal model breaks exactly when the task is hardest. A stable higher latency will usually beat an unstable lower one for demonstration quality.
Practical consequence: report your latency as a distribution, not an average. The 95th percentile and the worst-case spike tell you more than the mean ever will. This belongs in your demonstration log schema so you can correlate bad episodes with network events after the fact.
Latency tolerance by task class
There is no universal threshold. Tolerance depends on how tightly the task couples perception to force.
| Task class | Sensitivity | What breaks first |
|---|---|---|
| Free-space reaching and transport | Low | Little; operators compensate easily |
| Pick and place, rigid objects | Moderate | Grasp timing; objects nudged before closing |
| Insertion and alignment | High | Overshoot and repeated correction cycles |
| Deformable handling | High | Grip force overshoot, tearing, dropping |
| Bimanual coordination | Very high | Arms desynchronize; the task simply fails |
| Contact-rich tool use | Very high | Force spikes the operator cannot feel in time |
| Remote fleet intervention | Moderate | Situational awareness rather than fine control |
Set a budget per task class rather than one figure for the whole program. A cell that is fine for tote transfer will produce unusable insertion data.
Controlled cell versus public network
A teleoperation cell on a local wired network is a solved problem. Remote operation over public infrastructure is not.
Inside a controlled cell
Wired links, one switch, predictable load. Latency is dominated by capture and display, both of which you control. This is the environment behind most ALOHA-style collection.
Over public cellular networks
We run remote monitoring and intervention for an autonomous mobility company operating sidewalk delivery robots and self-driving passenger vehicles. Here the robot is outdoors, moving between cell towers, competing with everyone else on the same network. Latency is not a specification, it is a weather report.
Three things make that workable:
- Design for supervision, not manual driving. Assisting a robot with a route decision tolerates far more delay than steering it. Splitting these two modes is the single biggest lever, and we cover it in remote assistance vs remote driving.
- Degrade deliberately. When the link worsens, drop resolution and frame rate on secondary views before the primary view suffers. Never let the system silently buffer.
- Make latency visible to the operator. A live indicator changes operator behavior immediately. Hidden latency produces confident inputs into a loop that cannot honor them.
How to set a budget you can defend
- Measure glass-to-glass with a physical clock in frame. Do not sum spec sheets.
- Record the distribution per session: median, 95th percentile, worst spike, and spike count.
- Run a degradation study. Inject known delay and measure task success against it for each task class. You now have evidence instead of opinion.
- Set a hard ceiling per class and have the recorder flag episodes that breach it.
- Quarantine, do not delete. High-latency episodes are useful for studying recovery behavior even when unfit for imitation training.
- Re-measure after every stack change. A codec or driver update can move the number more than a hardware upgrade.
Tie this into your evaluation benchmarks so latency-flagged episodes can be excluded from a training run without being lost.
For a timeline where latency was budgeted from day one, see our mobile manipulation case study, which covers a 90-day path from cold-start to production.
Frequently asked questions
What is an acceptable teleoperation latency?
It depends entirely on task class. Free-space motion tolerates far more than insertion or bimanual work. Run a degradation study on your own tasks rather than adopting a number from a paper about different hardware.
Does high latency ruin the data or just the experience?
It corrupts the data. The operator’s compensating behavior, leading inputs, pausing, over-correcting, gets baked into the action labels, and the policy learns those artifacts as if they were the task.
Can we fix latency in post-processing?
Partially. You can re-align streams if timestamps are trustworthy. You cannot recover the demonstration the operator would have produced under a responsive loop.
Is wireless ever acceptable inside a collection cell?
For secondary context cameras, often yes. For the primary view and the control channel, wired is worth the inconvenience because wireless jitter is unpredictable in shared industrial space.
Latency is the cheapest thing to measure and the most expensive thing to discover late. If you are scoping a program, or running one and seeing quality problems you cannot explain, describe your setup to us and we will tell you where the delay is hiding.
