Robot Control with Coarse-to-Fine Planning and Data-Efficient Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current reinforcement learning methods for robot control are data-inefficient and computationally costly, particularly when learning value iteration procedures end-to-end, requiring substantial training data and effort.
Innovation Solution
A method that splits the learning process into two parts: using supervised learning to train a transition probability model for high-level planning, and reinforcement learning for low-level control policy training, eliminating the need for backpropagation through the planner and reducing data requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If end-to-end reinforcement learning is used to learn the value iteration procedure, then the robot can achieve autonomous navigation, but the training requires substantial training data and computational effort
Solution Approach 1:
The patent segments the learning process into two distinct parts: (1) supervised learning of transition probability models for high-level planning, and (2) reinforcement learning of low-level control policies. This segmentation allows each component to be trained independently with appropriate learning methods, reducing the overall data requirements compared to end-to-end reinforcement learning while maintaining autonomous navigation capability.
Solution Approach 2:
The patent introduces transition probability models as an intermediary between the high-level planner and low-level controller. These models, learned via supervised learning, provide a data-efficient representation of environment dynamics that guides the reinforcement learning process, thereby reducing the amount of training data needed compared to direct end-to-end learning.
2Extent of automation
If end-to-end reinforcement learning is used to learn the value iteration procedure, then the robot can achieve autonomous navigation, but the training process is computationally costly
Solution Approach 1:
The patent segments the learning process into two distinct parts: (1) supervised learning of transition probability models for high-level planning, and (2) reinforcement learning of low-level control policies. This segmentation allows each component to be trained independently with appropriate learning methods, reducing the overall data requirements compared to end-to-end reinforcement learning while maintaining autonomous navigation capability.
Solution Approach 2:
The patent introduces transition probability models as an intermediary between the high-level planner and low-level controller. These models, learned via supervised learning, provide a data-efficient representation of environment dynamics that guides the reinforcement learning process, thereby reducing the amount of training data needed compared to direct end-to-end learning.
3Quantity of substance
If supervised learning is used for transition probability model and reinforcement learning for control policy, then data efficiency is improved, but the system complexity increases
Solution Approach 1:
The patent segments the learning process into two distinct parts: (1) supervised learning of transition probability models for high-level planning, and (2) reinforcement learning of low-level control policies. This segmentation allows each component to be trained independently with appropriate learning methods, reducing the overall data requirements compared to end-to-end reinforcement learning while maintaining autonomous navigation capability.
Data Source
AI summary
A method for controlling a robot. The method includes receiving an indication of a target configuration to be reached from an initial configuration of the robot, determining a coarse-scale value map by value iteration, starting from an initial coarse-scale state and until the robot reaches the target configuration or a maximum number of fine-scale states has been reached, determining a fine-scale sub-goal from the coarse-scale value map, performing, by an actuator of the robot, fine-scale control actions to reach the determined fine-scale sub-goal and obtaining sensor data to determine the fine-scale states reached, starting from a current fine-scale state of the robot and until the robot reaches the determined fine-scale sub-goal, the robot transitions to a different coarse-scale state, or a maximum sequence length of the sequence of fine-scale states has been reached and determining the next coarse-scale state.