Every robot learning team stores trajectories. Very few can state precisely what is in one. Trajectory data for robotics looks like a solved problem until you try to pool two sources and discover they disagree about what an action even is.
This guide covers the six ways an action can be represented, the control-rate trap hiding inside delta actions, how to choose sampling rates, a storage layout that survives scale, and the fields that make a trajectory reusable by someone who was not there.
What a trajectory is, precisely
A trajectory is a time-ordered sequence of states and the actions that produced the transitions between them. In robot learning terms: at each timestep you have an observation, an action, and a timestamp. Everything else is metadata.
That sounds unambiguous. It is not, because “action” can mean at least six different things and teams frequently mean different ones within the same organization.
Six ways to represent an action
| Representation | What it stores | Best for | Weakness |
|---|---|---|---|
| Absolute joint positions | Target angle per joint | Single-embodiment fine-tuning | Does not transfer between robots |
| Joint velocities | Rate per joint | Smooth continuous control | Integrates error over time |
| Absolute end-effector pose | Position and orientation of the tool | Cross-embodiment pooling | Loses arm configuration and elbow choice |
| Delta end-effector pose | Change since last step | Most policy architectures | Highly sensitive to control rate |
| Joint torques | Commanded effort | Contact-rich and compliant tasks | Hardware-specific; rarely portable |
| Discretized action tokens | Binned actions as symbols | Transformer-style policies | Binning choice is baked in permanently |
Pick one as canonical, store it, and derive the rest. Storing a derived representation as if it were primary is how datasets become unpoolable, which matters most for cross-embodiment work.
Delta actions and the control-rate trap
Delta representations are popular because they suit most policy architectures. They also carry a hazard that catches teams out repeatedly.
A delta is meaningless without the control rate it was recorded at. A five-millimeter step at 10 Hz and a five-millimeter step at 50 Hz describe completely different motions. If you record deltas without recording the actual measured rate per episode, and your rate varies under load, you have introduced noise the model reads as intent.
Two rules follow. Record measured control rate per episode, not the configured one. And store absolute poses as the canonical form, deriving deltas at training time where the rate is known and explicit.
Sampling rate: what to record versus what to train on
These are different questions and conflating them is expensive.
- Record at the highest rate your hardware sustains. Downsampling later is free. Upsampling is fabrication.
- Train at whatever rate the policy needs. Often much lower than capture rate.
- Keep force and torque at native high rate regardless. Contact transients live in the milliseconds and disappear under video-rate sampling. See force-torque capture.
- Never interpolate into the stored record. If a sample is missing, mark it missing. Interpolated values that look like measurements are the hardest class of bug to find later.
Practical storage layout
Multi-camera trajectory data grows faster than most first forecasts. A layout that survives contact with scale looks roughly like this.
- One directory per episode, containing a manifest that describes everything inside it. Self-describing episodes survive migrations that break central indexes.
- Video in a compressed container, one file per camera, with a frame-index-to-timestamp table stored separately. Do not rely on container timestamps.
- Numeric streams in a columnar format such as Parquet, one file per stream. They compress well and support partial reads.
- Metadata as plain JSON in the manifest. It should be readable without any tooling in five years.
- Annotations outside the episode directory, versioned and referencing episode IDs, as covered in dataset structure.
- A dataset-level index that is fully regenerable by scanning manifests. If the index is the only source of truth, one corruption loses everything.
The fields that make a trajectory reusable
Beyond the sequence itself, a trajectory needs enough context to be interpreted by someone who was not there.
- Action representation and coordinate frame, stated explicitly
- Measured control rate and per-stream timestamps
- Robot model, gripper, and firmware version
- Calibration reference in force at capture
- Units, stated rather than assumed
- Gripper state as a continuous value where the hardware allows
- Outcome and failure mode from a controlled vocabulary
- Source and embodiment tags for pooling and ablation
The unit question sounds trivial and is not. Mixed radians and degrees, or metres and millimetres, across sources is one of the most common and most silent defects in pooled datasets.
For a programme where the format was settled before episode one, see our mobile manipulation case study and its 90-day path to production.
Frequently asked questions
Should we store absolute or delta actions?
Store absolute as canonical and derive deltas at training time. Deltas depend on control rate, and rate varies more than teams expect under real load.
What sampling rate should we record at?
The highest your hardware sustains reliably, with force and torque well above video rate. You can always downsample; you can never recover a transient you did not capture.
Is a standard format worth adopting?
Yes for the raw and episode layers, because it makes pooling with public datasets straightforward. Most teams extend the metadata layer to fit their own operation.
How do we handle variable-length episodes?
Store them at native length with explicit start and end markers and phase boundaries. Padding and truncation are training-time decisions that should never be baked into storage.
Trajectory format decisions are cheap to make and expensive to revisit, because every episode collected under the old choice has to be migrated or discarded. If you are about to lock yours in, send us the schema and we will tell you what will hurt at scale.
