From Perception to Autonomy
As robots move from controlled environments into factories, warehouses, sidewalks and eventually our homes, AI alone isn’t enough. A robot must understand where it is, what surrounds it and how the world is changing around it. That starts with perception.
At RealSense, we’re building the Visual Cortex of Physical AI™. Pixels show AI what the world looks like. Depth gives machines the spatial understanding required to navigate, manipulate objects and operate safely alongside people.
NVIDIA Isaac ROS provides another critical part of that stack: GPU-accelerated perception, localization, mapping and manipulation running on NVIDIA Jetson.
With the release of NVIDIA Isaac ROS 5.0, RealSense and NVIDIA are continuing to make it easier for robotics developers to connect high-performance 3D perception with accelerated robotics software and AI compute.
The goal is to shorten the path between seeing the world, understanding it and acting on it.
Building Cameras for the ROS and NVIDIA Robotics Ecosystems
RealSense’s newest AI-native stereo depth cameras are being designed not simply as sensors, but as intelligent participants in the robotics compute architecture.
The RealSense D555 PoE introduced the company’s fifth-generation RealSense Vision SoC, Power over Ethernet and native ROS 2 communication through DDS. Rather than requiring every perception stream to first pass through a traditional host-side camera architecture, the D555 can publish ROS 2 data directly onto the robot’s network for consumption by ROS-based applications and Isaac ROS pipelines.
The RealSense D585 extends that architecture substantially. Built around the RealSense V5 SoC, the D585 combines wide-field-of-view stereo depth, RGB, IMU and on-camera processing in a platform designed specifically for Physical AI.
Both platforms are designed to fit naturally into modern NVIDIA robotics architectures, including NVIDIA Holoscan for low-latency sensor streaming and ROS-based perception pipelines.
The objective is straightforward: reduce the amount of work required between seeing the world and acting on it.
Moving RealSense Data Faster on Jetson
As perception workloads become more sophisticated, moving camera data efficiently becomes almost as important as generating it.
Traditional robotics pipelines can involve multiple memory copies as image and depth data move between the camera driver, CPU memory, ROS nodes and GPU processing. Those copies consume something robots have in limited supply: latency, memory bandwidth and power.
RealSense has been optimizing its open-source SDK and ROS integration for NVIDIA Isaac ROS, including GPU zero-copy paths designed to move RealSense perception data more efficiently into GPU-accelerated pipelines on NVIDIA Jetson.
For developers, that means more of the Jetson platform’s compute budget can go toward the work that actually makes the robot intelligent — localization, mapping, inference, planning and manipulation — instead of moving pixels between memory buffers.
That becomes increasingly important as robots incorporate multiple cameras and run more sophisticated AI models at the edge.
RealSense + cuVSLAM: Giving Humanoids Spatial Awareness
The benefits are already visible in real robots.
RealSense, NVIDIA and LimX Dynamics have demonstrated autonomous humanoid navigation combining RealSense dense 3D perception with NVIDIA’s GPU-accelerated cuVSLAM pipeline.
LimX’s Oli humanoid uses RealSense depth perception as part of a multi-sensor architecture combining RGB-D vision, IMU data and joint sensing. NVIDIA cuVSLAM adds accelerated visual localization and odometry, helping the humanoid understand not only what surrounds it, but also its own movement through that environment.
That distinction is critical for legged robots.
A humanoid navigating a building must continuously understand free space, obstacles, terrain, elevation changes and its own position — all while its cameras are moving with the robot’s body.
Dense depth combined with visual-inertial localization provides the spatial foundation required for autonomous navigation.
The result is an architecture capable of moving beyond teleoperation toward robots that can independently perceive, localize and navigate through dynamic environments.
From Depth to 3D Maps with nvblox
That same depth information can feed NVIDIA Isaac ROS libraries such as nvblox, transforming streams of depth data into continuously updated 3D representations of the environment.
RealSense provides dense spatial observations. cuVSLAM helps determine where the robot is and how it is moving. nvblox can turn depth and pose into GPU-accelerated 3D reconstruction and navigation costmaps that downstream planning systems can use.
RealSense Depth → Isaac ROS → cuVSLAM → nvblox → Navigation
For AMRs and humanoids, this combination transforms a camera from something that simply captures images into part of a continuously operating spatial intelligence system.
Physical AI Manipulation
The same principle applies when a robot isn’t moving through an environment but instead needs to interact with it.
RealSense customer Inbolt demonstrates what perception-first automation looks like in live production.
Its robot-mounted guidance platform uses the RealSense D435 depth camera with AI-driven software to continuously understand where parts are and adjust robot trajectories in real time.
RealSense depth data feeds directly into Inbolt’s AI stack, enabling continuous 6-DoF pose estimation and trajectory correction at high frequency, with perception-to-motion latency under 80 milliseconds.
Instead of detecting a part once and assuming nothing changes, the system continuously re-localizes parts during execution. If a bin shifts, a part overlaps or a conveyor keeps moving, the robot adapts in real time.
Perception and motion are no longer separate systems. They become one closed loop.
Inbolt also integrates NVIDIA Jetson and edge AI capabilities into its 3D vision and real-time robot guidance architecture. Using NVIDIA accelerated computing, Inbolt has reported reducing depth-image computation from roughly 100 milliseconds to just a few milliseconds. The company is also a member of the NVIDIA Inception program.
Together, high-quality 3D perception and accelerated edge computing help enable the responsiveness required for robots to operate in real-world manufacturing environments.
That changes the traditional automation model.
Industrial robot stations have historically depended on precise fixtures, repeated calibration and predictable part placement. Vision-guided robotics allows a robot to perceive where an object actually is and adjust its motion accordingly.
Across the NVIDIA Isaac ROS ecosystem, GPU-accelerated perception, localization, 3D reconstruction and manipulation capabilities provide additional building blocks for transforming that spatial perception into robotic action.
Instead of programming a robot around a perfectly predictable world, we can build robots capable of perceiving and adapting to an imperfect one.
The Next Layer: Agentic Robotics
There is another transformation beginning above the traditional robotics stack.
Robots have historically been commanded through applications, scripts and predefined state machines. The emergence of AI agents and reasoning models creates the possibility of commanding robots at a much higher level.
Instead of telling a robot, “Move forward 0.5 meters,” we can tell it: “Find the pallet blocking aisle three and inspect it.”
The AI agent can determine the sequence of perception, navigation and manipulation capabilities necessary to accomplish that mission.
AgenticROS, an open-source project sponsored by RealSense, is exploring this new layer by exposing ROS 2 capabilities to AI agents.
Through AgenticROS, NVIDIA technologies including Isaac ROS can become tools available to agentic systems running NVIDIA NemoClaw and Nemotron.
This creates a different model for robot intelligence.
AI agents can provide reasoning and mission-level intelligence. RealSense provides perception and spatial awareness. NVIDIA Jetson provides accelerated AI compute, while Isaac ROS delivers robotics capabilities for perception, localization, navigation and manipulation.
AgenticROS provides a layer for connecting those capabilities to AI agents that can reason about which tools the robot should use to accomplish a higher-level goal.
The result is a progression from robotic commands to robotic intent: Perceive → Understand → Plan → Act
Instead of requiring a developer to explicitly program every step of a mission, an agentic system can begin to reason about the goal, select the appropriate robotic capabilities and orchestrate how they are used.
That is an important step toward more general-purpose Physical AI systems.
Building the Physical AI Stack Together
No single technology creates an autonomous robot.
It requires an entire stack working together: sensors that reliably perceive the environment, accelerated computing that transforms sensor data into spatial understanding, robotics software that connects capabilities and, increasingly, AI models capable of reasoning about what the robot should do next.
That is why the collaboration between the RealSense and NVIDIA robotics ecosystems is so important.
RealSense provides the Visual Cortex of Physical AI™.
NVIDIA Jetson provides accelerated compute.
NVIDIA Isaac ROS transforms perception data into actionable robotics capabilities across localization, mapping, navigation and manipulation.
And emerging agentic frameworks such as AgenticROS can connect higher-level AI reasoning to ROS-based robots.
Together, these technologies are helping developers move from robots that simply execute commands toward machines that can perceive, understand, reason and act in the physical world.
That is the foundation of Physical AI.
