Depth sensors are on almost every robot sensor list and switched off in more programmes than teams admit. RGB-D data collection costs real bandwidth, real storage, and real calibration effort, and it is not always worth it.
This guide covers what depth genuinely adds over colour alone, the situations where it earns its cost and where it actively misleads, the storage reality nobody forecasts correctly, and how to record it so it is still usable a year later.
What depth actually adds
A colour image tells you what is in front of the camera. It does not reliably tell you how far away. Humans infer distance from experience, shading, and context; a model can learn to do the same, but only for objects and scenes resembling its training data.
A depth stream measures distance directly, per pixel. That converts an inference problem into a measurement, and the difference shows up most in three places: reach planning, object geometry for grasping, and separating an object from a background of similar colour.
The catch
Depth sensors are not cameras that also measure distance. They are separate devices with their own failure modes, their own noise, and their own opinions about surfaces they cannot handle.
When depth earns its cost
| Situation | Depth value | Reason |
|---|---|---|
| Mobile robots navigating | High | Obstacle distance is the core question |
| Grasping novel objects | High | Geometry cannot be memorised in advance |
| Cluttered bins | High | Separating touching objects by colour alone fails |
| Fixed-position pick and place | Low | Geometry is constant and can be learned |
| Wrist-camera final approach | Low | Close-range monocular carries enough cue |
| Transparent or shiny objects | Negative | Sensors return noise; it can mislead |
| Outdoor bright sunlight | Low to negative | Many sensor types degrade badly |
That last group is the one worth internalising. On glass, chrome, and dark matte surfaces, consumer depth sensors do not fail cleanly. They return confident, wrong values, which is worse than returning nothing.
The bandwidth and storage reality
Depth is expensive in ways that surprise teams mid-programme. It is high-rate, poorly compressible with standard video codecs, and lossy compression destroys precisely the precision you paid for.
Three practical consequences:
- Storage grows faster than your forecast. Depth frequently exceeds the colour streams it accompanies.
- Write throughput becomes the frame-rate limit. Not the sensor, the disk.
- Lossy compression is not an option for the depth channel, though it is fine for the paired colour frame.
Capturing depth at a lower rate than colour is a reasonable compromise, since geometry changes more slowly than appearance. Record the actual rate rather than assuming it, per multi-modal sync guidance.
Recording it so it stays usable
- Store depth aligned to colour, and keep the raw too. Alignment is a processing choice you may want to redo.
- Record intrinsics and the depth-to-colour extrinsic per session. Without them the streams cannot be registered later.
- Record units and the invalid-value convention. Whether zero means “no reading” or “zero distance” has ruined more datasets than sensor noise.
- Keep the confidence or validity mask if the sensor provides one. It is how you tell measurement from guess.
- Never interpolate holes into the stored data. Fill them downstream if you want; a filled hole that looks like a measurement is a silent error.
- Log the sensor model and firmware. Depth characteristics differ enough between devices that pooling without this is unwise.
Depth versus other ways to get geometry
Depth sensors are not the only route, and sometimes not the best one.
- Stereo cameras. Two colour cameras plus computation. Robust in sunlight where many active sensors fail, and weak on textureless surfaces.
- LiDAR. Excellent range and accuracy, sparse compared with a depth image, and better suited to navigation than to grasping.
- Monocular depth estimation. A model predicting depth from a single image. Free at capture time, improving quickly, and an estimate rather than a measurement.
- Wrist-camera proximity. Get the camera close enough and monocular cue is sufficient, which is why wrist views often reduce the need for depth on manipulation tasks.
For paired-viewpoint capture where geometry mattered, our aerial perception case study reports a 3x detection rate from paired drone-and-ground capture.
Frequently asked questions
Do modern policies still need depth?
Less than they used to. Vision-only policies have closed much of the gap on familiar objects. Depth still helps on novel geometry, heavy clutter, and anything where reach distance must be judged rather than learned.
Can we add depth to an existing colour dataset?
Only as an estimate, using monocular prediction. That is useful and it is not a measurement, so label it clearly and never mix estimated and measured depth in one field.
What about transparent objects?
Most depth sensors fail on glass and clear plastic, often returning plausible wrong values. If your task involves transparent items, plan for depth to be unreliable and lean on colour, tactile, and force signals instead.
Should depth run at the same rate as colour?
Not necessarily. Geometry changes more slowly than appearance, so a lower depth rate is a sensible saving. Record the real rate per stream so alignment stays correct.
Depth is worth its cost on navigation, novel geometry, and clutter, and is frequently an expensive habit everywhere else. If you are specifying a sensor stack and want the depth decision tested against your actual task list, tell us what you are building.
