Hierarchical Robot Control for Data-Efficient Autonomous Navigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement Learning (RL) algorithms for robot control are data-inefficient and computationally intensive due to the need for extensive training with sparse reward signals and end-to-end learning of value iteration procedures.
Innovation Solution
A hierarchical approach is employed, where a transition probability model is learned using supervised learning for high-level planning, and a low-level control policy is trained via reinforcement learning, decoupling the complexity and reducing the need for backpropagation through continuous control policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If end-to-end reinforcement learning is used to learn value iteration procedures, then the robot can achieve autonomous navigation and control, but the training data requirement increases substantially and training time becomes very long
Solution Approach 1:
The patent segments the learning process into two distinct parts: (1) supervised learning of transition probability models from demonstration data, and (2) reinforcement learning of control policies using these pre-learned models. This segmentation allows the complex end-to-end learning to be broken down into manageable stages, reducing the overall training time and data requirements while maintaining autonomous navigation capability.
Solution Approach 2:
The patent applies preliminary action by pre-learning transition probability models from demonstration data before performing reinforcement learning. This preliminary supervised learning phase prepares the system with prior knowledge about environment dynamics, so that the subsequent reinforcement learning phase requires less training data and time to achieve effective autonomous control.
2Extent of automation
If end-to-end reinforcement learning is used to learn value iteration procedures, then the robot can achieve autonomous navigation and control, but the amount of training data required increases substantially
Solution Approach 1:
The patent segments the learning process into supervised learning of transition models and reinforcement learning of control policies. This allows the system to leverage demonstration data efficiently for model learning, reducing the amount of interactive training data needed during the reinforcement learning phase while still achieving autonomous navigation.
Solution Approach 2:
The patent uses demonstration data as copies of expert behavior to pre-train transition probability models. By copying knowledge from demonstrated trajectories, the system acquires prior understanding of environment dynamics without requiring extensive interactive training data, thus reducing the overall training data volume needed.
3Adaptability or versatility
If value iteration procedures are learned end-to-end with continuous control policy, then the robot can achieve adaptive planning, but computational complexity and training difficulty increase due to backpropagation through planners
Solution Approach 1:
The patent segments the learning architecture into distinct components: transition probability models learned via supervised learning, and control policies learned via reinforcement learning. This segmentation eliminates the need for backpropagation through value iteration planners, reducing training system complexity while preserving adaptive planning capability through the use of pre-learned transition models.
Data Source
Figure 1
Figure 2~3
AI summary
According to various embodiments, a method for controlling a robot is described comprising receiving an indication of a target configuration to be reached from an initial configuration of the robot, determining a coarse-scale value map by value iteration, wherein the transition probabilities are determined using a transition probability model mapping coarse-scale states and coarse-scale actions to transition probabilities for coarse-scale states and for each coarse-scale state of a sequence of coarse-scale states of the robot, starting from an initial coarse-scale state determined from the initial configuration of the robot and until the robot reaches the target configuration or a maximum number of fine-scale states has been reached, determining a fine-scale sub-goal from the coarse-scale value map, performing, by an actuator of the robot, fine-scale control actions to reach the determined fine-scale sub-goal and obtaining sensor data to determine the fine-scale states reached as a result of performing the fine-scale control actions for each fine-scale state of the resulting sequence of fine-scale states of the robot, starting from a current fine-scale state of the robot and until the robot reaches the determined fine-scale sub-goal, the robot transitions to a different coarse-scale state, or a maximum sequence length of the sequence of fine-scale states has been reached and determining the next coarse-scale state of the sequence of coarse-scale states from the last fine-scale state of the sequence of fine-scale states.