Figure 1. D585 Pro prototype in the RealSense Viewer: depth stream (left) and RGB stream with on-camera detections and measured distance, 2.2m and 2.4m (right). warehouse scene, ambient light.

Discover how RealSense on-camera people detection combines AI-powered recognition, precise depth measurements and wide-field coverage to help robots operate more safely around people.

The D585 Pro detects people on its own V5 SoC and reports a box, a confidence and a measured distance to center of mass for each one, 30 times a second, as a few hundred bytes of JSON. No host GPU. This post goes under the hood: why the distance is accurate, what the 120° field of view buys you, what the API looks like, where the detector is strong and where it is not, and how to place it in a safety architecture as the rules for AI in safety functions take shape.

A compact person detector, based on DAMO-YOLO and trained with real data collected by the RealSense D585 optical module, runs on the available Gen5 Vision SoC supporting D555 (Already in production) and D585 pro (sampling to customers these days).

 D585 Pro’s Gen 5 SoC. For every frame it outputs a list of detections: class, confidence (0 to 100), bounding box in pixels, and distance in millimeters. Results are published at 30 fps on a dedicated ROS 2 topic or through the RealSense SDK, independent of the RGB and depth streams. You can subscribe to the detections and never pull a single video frame from the camera.

Three design choices make this useful for safety-related work, and this post is about those three: the distance comes from the depth engine and not from a neural network; the depth the distance comes from is aligned to the RGB the detector sees on the camera itself, from one factory-calibrated module; and the output is small metadata with a predictable cadence.

1. Why the distance is right: one module, one calibration

Any RGB-D camera with a color sensor next to its depth sensor needs extrinsic calibration between them and a re-projection step to put a depth value on an RGB pixel. The baseline between the two sensors creates parallax, so at the edge of a person the RGB pixel and the “aligned” depth pixel can belong to different objects. A detector box drawn on RGB then collects some background depth along its edges. How much this hurts depends on three things: how short and how rigid the baseline is, how good and how stable the calibration is, and how the distance is read out of the box.

The worst case is an external RGB camera: a separate module that is not part of the depth unit at all and sits wherever the mechanical design allows, often tens of centimeters from the depth sensor. The two now share no housing, no rigid mount and no thermal path. A bracket that flexes under vibration, a mount that expands as the electronics warm up, a knock during maintenance: any of these shifts the extrinsic by a few pixels, and the depth values behind a detection box quietly stop belonging to the person. Nothing in the data tells you it happened. The only way to hold day-one accuracy is to re-calibrate, and to keep re-calibrating.

The D585 Pro does not have this problem because it does not leave the alignment to the host or the mechanical design. It has a dedicated RGB imager mounted next to the left imager of the stereo pair, at a short baseline, on the same rigid optical module, in one housing with one thermal path. Depth is computed by the stereo engine between the left and right imagers, and the camera re-projects it into the RGB pixel grid using the module’s factory calibration, frame by frame, before anything reaches the host. The detector runs on the RGB, so its box lands on an aligned depth map that was produced on the camera from a calibration done once, on the module, in the factory; nothing in the robot’s mechanical design can move the sensors relative to each other. The residual parallax at the box edges is handled in the distance read-out, which is the subject of the next section: the camera reports the nearest depth cluster around the center of the box and drops outliers, so background pixels along the edges do not enter the measurement.

Figure 2. Top: a two-sensor RGB-D camera needs extrinsic calibration and re-projection, and picks up background depth at the edges of a box. Bottom: on the D585 Pro the RGB imager sits on the same rigid, factory-calibrated module as the stereo pair, and depth is re-projected into the RGB grid on the camera, so the box coordinates are valid in the aligned depth map.

2. Why the distance is trustworthy: no neural network in the measurement

The detector is AI. The distance is not. Once the box exists, the camera takes the depth pixels inside it, finds the nearest cluster around the box’s centre of mass, drops outliers with non-maximum suppression, and reports that cluster’s range. Stereo depth on the D585 Pro comes from a dedicated engine with dual projectors (Active IR Stereo); it is geometry, not inference.

This split matters for safety engineering. A detector can miss a person or mislabel a coat rack. It cannot make a wall look 1.5 m closer than it is. Keeping the measurement path free of learned components lets you reason about worst-case error the way you would for any range sensor, and it is exactly the pattern the emerging guidance on AI in safety functions describes: AI proposes, deterministic logic measures and decides.

Figure 3. The measurement chain. Only the first block is a neural network. Distance method per librealsense 2.58.3; firmware-side distance option added in 2.58.4. [CONFIRM (b)]

3. What 120° × 100° and 15cm minimal detection range buys you

Field of view is the spec people underrate until they mount the camera. Put a D585 Pro on an AMR at 1.4 m and the lower edge of the 100° vertical FoV hits the floor 1.2 m ahead. A typical 58° vertical camera at the same height sees the floor from 2.5 m. That difference is a person standing right next to the robot with their feet in frame, or cut off at the knees.

Horizontally, 120° gives a 6.9 m wide strip at 2 m and 10.4 m at 3 m. Three cameras cover 360° around a vehicle. A person stepping out of an aisle is in view before they are in the path, which is what gives the navigation stack time to slow down instead of stop. The trade-off is pixels per person at range, which is the next section.

Figure 4. Coverage geometry at 1.4 m mounting height. Side view: 100° vertical (blue) vs a typical 58° camera (grey). Top view: strip width at 1, 2 and 3 m for 120° vs 87° horizontal.

4. Know your detection envelope

A wide lens spreads the scene over the same number of pixels, so a person shrinks faster with distance. The detector runs at a 416×416 input. For a 1.75 m adult standing and facing the camera, the box crosses the COCO “large” threshold (96² = 9,216 px²) at about 1.3 m and the “small” threshold (32² = 1,024 px²) at about 3.8 m. Box area falls with the square of distance, so the numbers move fast. 

Detection quality follows the box size. On our warehouse benchmark (IoU 0.50:0.95) the tuned model reaches AP 0.91 on large boxes, 0.70 on medium and 0.06 on small. On COCO person the same model gives 0.75, 0.55 and 0.04. Inside 1.3m detection is strong and distance comes from thousands of depth pixels; from 1.3 to 3.8m it is solid and good for slow-down and zone logic; beyond 3.8m a standing person is a few hundred pixels and you should treat detections as awareness only.

Figure 5. Box area against distance for a standing adult, D585 Pro geometry, 416×416 detector input. Zones follow the COCO small/medium/large definitions; AP values from the RealSense warehouse benchmark.

Posture changes the envelope. A crouching person or someone lying on the floor presents a smaller box at the same distance, so the ranges move closer. This is the table to design against:

Distance ranges per box-size class for a 1.75 m adult, standard BMI, Drillis & Contini (1966) body proportions, camera at 1.4 m, 416×416 detector input. Internal RealSense analysis.

Figure 6. Postures used for the table: standing, arms spread, sideways, crouching, lying.

RealSense-tuned detector. Original model values: COCO 0.168 / 0.517 / 0.737 AP; warehouse 0.041 / 0.559 / 0.862 AP. You can see the performance leap on the RealSense datasets.

5. Metadata, not video

The traditional way to add people detection to a robot is to stream RGB to the host and run a model there. That costs the GPU you wanted for navigation, adds copy and pre-processing overhead, and puts a full video stream on the bus. On the D585 Pro the whole chain runs on the camera and what comes out is a JSON message per frame. Latency measured at the host on the D555 is 53ms end to end.

Figure 7. What leaves the camera. Detection JSON at 30 fps, a few KB/s, against roughly 110 MB/s for raw 1280×960 RGB at 30 fps. The RGB and depth streams stay available and independent.

The interface is deliberately plain. Topic name /realsense/D585_{serial}_ObjectDetection, message type std_msgs/String carrying JSON, QoS BEST_EFFORT to match image topics, 30 fps. Subscribe to enable detection, unsubscribe to disable it. The payload:

{
  "schema_version": 1,
  "message_type": "object_detection",
  "timestamp_us": 123456789,
  "frame_id": 42,
  "enabled": true,
  "has_result": true,
  "number_of_detections": 2,
  "detections": [
    { "class_id": 0, "confidence": 85, "x1": 100, "y1": 50, "x2": 300, "y2": 250, "distance_mm": 1500 },
    { "class_id": 0, "confidence": 72, "x1": 400, "y1": 100, "x2": 550, "y2": 300, "distance_mm": 2200 }
  ]
}

{
  "schema_version": 1,
  "message_type": "object_detection",
  "timestamp_us": 123456789,
  "frame_id": 42,
  "enabled": true,
  "has_result": true,
  "number_of_detections": 2,
  "detections": [
    { "class_id": 0, "confidence": 85, "x1": 100, "y1": 50, "x2": 300, "y2": 250, "distance_mm": 1500 },
    { "class_id": 0, "confidence": 72, "x1": 400, "y1": 100, "x2": 550, "y2": 300, "distance_mm": 2200 }
  ]
}

Two details are there for people who build supervision logic. When nobody is in view the camera still publishes a message with has_result true and zero detections, every frame. Silence means the link or the camera is down, not that the floor is clear. And has_result false means the model did not run on this frame. You never have to guess between “no people”, “no data” and “no inference”.

6. How to use it in a safety architecture

Say it plainly: on-camera people detection is a perception feature. It is not a safety-rated function, yet, and it is not certified to IEC 61496 or to a performance level under ISO 13849-1. Your protective stop still comes from your safety-rated devices now. What this feature adds is a second, diverse layer that knows the difference between a person and a pallet, sees a wider field than a scanner plane, and costs no host compute. Used that way, here is what works:

1. Speed and separation, advisory. Use distance_mm to scale speed down before the person reaches the protective field. Fewer protective stops, smoother fleet throughput, and the safety device still does its job.
2. Zones beyond the scanner plane. A 2D scanner sees ankles at one height. The camera sees a person leaning over a conveyor or a hand reaching into a cell. Feed that to the machine controller as a slow-down or a warning.
3. Classification for a fused system. Pair the detection with your existing range sensor. The scanner measures, the camera says “that is a person, not a box”. Diversity of technology is what the sensor standards ask for when you go beyond the classic ESPE.
4. Design against the envelope. Put your zone boundaries inside 3.8 m for standing people, closer for crouching or lying. Mount at 1.3 to 1.5 m. Use the posture table above to size the zones, then test with the postures.
5. Watch the cadence. Treat the 30 fps heartbeat as a liveness signal. Missing messages for more than N frames is a fault; act on it.
6. Log the metadata. A few KB/s of detections is cheap to keep for weeks. Incident review, near-miss statistics and zone tuning come almost for free.

7. Where the standards are going

The reason to design this carefully now is that the rules are moving toward exactly this kind of sensor. A short map, status as of September 2026:

• ISO 3691-4:2023 is the current standard for driverless industrial trucks and defines personnel detection means and how to test them. It is technology-neutral about the sensor, and it expects measured, testable detection.
• IEC TS 61496-4-3:2022 covers stereo-vision protective devices. It is a technical specification, not yet a full standard, and it excludes object classification. It tells you how the committee thinks about 3D cameras as protective devices.
• IEC 62998-1 is the coming full standard for safety-related sensors that do not fit the classic ESPE mould, including sensor fusion and cameras in the visible range. It passed the committee vote and is heading to final draft in 2027. This is the door through which fused camera-plus-scanner systems will walk.
• ISO 10218-1 and -2:2025 pulled speed and separation monitoring into the main robot standards. A camera that measures distance to a person at 30 fps is a natural input to that logic.
• ISO/IEC TR 5469:2024 and the ISO/IEC TS 22440 drafts are the first international texts on AI inside safety functions. Their common recommendation is the architecture in Figure 3: keep the learned component to classification, keep measurement and decision deterministic, and supervise the AI with non-AI logic.
• EU Machinery Regulation 2023/1230 applies from 20 January 2027. Safety components with self-evolving behaviour based on machine learning go through notified-body assessment. If you plan to put any learned model into a protective function in Europe, the paperwork starts now.

Figure 8. Standards touching camera-based people detection and AI in safety functions. Blue published, orange in progress. Not legal advice.

None of this makes a beta detector a safety device. All of it says the same thing: the sensor that wins is the one that measures distance deterministically, reports it in a testable form, and keeps its AI where it can be supervised. That is how the D585 Pro is built.

Get started

• D585 Pro: 120° × 100° FoV, 12 cm to 10 m+ range, 1280×960 depth at 60 fps, dual RGB mode, GMSL2 and USB, IMU, IP65. On-camera person detection ships in beta at launch. Shipping expected Q1 2027; evaluation units are with customers now.
• Today: the same feature runs on the D555 with the Early Access firmware. Perception Studio members get the topic, the schema and the setup guide.
• SDK: librealsense 2.58.4 or later. The “Perception” sensor exposes the “Object Detection” stream; RS2_OPTION_DETECTION_DISTANCE enables per-detection distance.
• Tell us your zone geometry and mounting height. We will tell you where your envelope is.