Dual-Policy Robot Navigation for Mapless Obstacle Avoidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robot navigation systems face challenges in deploying reinforcement learning-based policy models on real robots due to high sample complexity, leading to poor performance in dynamic environments, as they are often trained in simulation environments with different sensory data.
Innovation Solution
The implementation of a dual policy model approach, where a high-level recurrent neural network (RNN) model performs long-range planning and a low-level feed-forward neural network (CNN) model generates granular control actions, allowing for efficient and safe navigation without relying on environmental maps, using supervised training for the high-level model and reinforcement training for the low-level model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning is used to train policy models for robot navigation, then the robot can learn to associate raw observations with actions without mapping or explicit path planning, but the high sample complexity causes the models to fail when deployed on real robots due to visual differences between simulation and real sensor data
Solution Approach 1:
The navigation task is segmented into two distinct levels: high-level policy model for long-range path planning and low-level policy model for immediate obstacle avoidance and precise control. This segmentation allows each model to be trained with appropriate data - high-level with simulated data for general navigation strategies, and low-level with real sensor data for reliable execution.
Solution Approach 2:
The high-level policy model acts as an intermediary that processes simulated navigation experiences and generates target poses, which then guide the low-level policy model. This intermediary structure allows the system to benefit from both simulated training efficiency and real-world reliability.
2Device complexity
If a single policy model is used for both long-range planning and immediate control, then the system structure is simpler, but the model cannot simultaneously handle both strategic navigation and reactive obstacle avoidance effectively
Solution Approach 1:
The navigation system is divided into two specialized policy models: high-level for strategic long-range planning and low-level for tactical immediate control. This segmentation enables each model to optimize for its specific function, improving overall navigation efficiency despite increased system complexity.
Solution Approach 2:
The navigation problem is solved by adding a temporal and functional dimension - high-level policy operates at longer time scales for strategic planning, while low-level policy operates at shorter time scales for reactive control. This multi-dimensional approach allows simultaneous handling of both strategic and tactical requirements.
3Adaptability or versatility
If reinforcement learning with high sample complexity is used, then the robot can learn complex navigation behaviors, but the training requires extensive computational resources and time
Solution Approach 1:
Training is segmented into two phases: high-level policy trained on simulated data for complex navigation strategies, and low-level policy trained on real sensor data for precise control. This segmentation reduces overall training time by allowing parallel training of both models with appropriately scaled data requirements.
Solution Approach 2:
The high-level policy model is trained using simulated environment copies rather than requiring extensive real-world data collection. This copying approach maintains learning effectiveness while dramatically reducing training time and resource requirements for the strategic planning component.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Training and/or using both a high-level policy model and a low-level policy model for mobile robot navigation. High-level output generated using the high-level policy model at each iteration indicates a corresponding high-level action for robot movement in navigating to the navigation target. The low-level output generated at each iteration is based on the determined corresponding high-level action for that iteration, and is based on observation(s) for that iteration. The low-level policy model is trained to generate low-level output that defines low-level action(s) that define robot movement more granularly than the high-level action – and to generate low-level action(s) that avoid obstacles and/or that are efficient (e.g., distance and/or time efficiency).