Vision-Language Grounded Semantic Navigation for Autonomous Mobile Robots

vision-language-grounded-semantic-navigation-for-autonomous-mobile-robots

Vision-Language Grounded Semantic Navigation for Autonomous Mobile Robots

[Submitted on 16 Nov 2026]

View PDF HTML (experimental)

Abstract:Language-guided navigation enables mobile robots to interpret natural language commands and navigate toward user-specified targets in unstructured environments. However, integrating open-vocabulary visual perception with classical robotics pipelines for mapping and navigation remains a challenging problem. In this work, we present a complete ROS2-based system that combines frontier-based exploration, open-vocabulary object detection using Grounding DINO, depth-based 3D localization, and semantic mapping to enable end-to-end vision-language grounded navigation in simulation. The system accepts natural language commands such as “go to the yellow box,” detects candidate objects from RGB images using text prompts, estimates their positions in the world frame, and generates navigation goals executed by the Nav2 stack. In a Gazebo-based indoor environment with a TurtleBot3 robot, the system autonomously explored the room, detected three of the five target objects, and successfully executed both natural language navigation commands tested, without task-specific training. The results indicate that open-vocabulary grounding models can be integrated into classical navigation pipelines, though performance is limited by detection uncertainty, depth-based localization error, and incomplete exploration coverage. These findings highlight both the feasibility and current limitations of deploying vision-language grounded navigation systems using lightweight, CPU-only architectures.

Submission history

From: Ravindran Vasoor Saravanan [view email]
[v1] Mon, 16 Nov 2026 05:38:50 UTC (10,135 KB)