Hierarchical Robot Control With Off-Policy Goal Relabeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotic control tasks using deep reinforcement learning are limited to atomic, episodic, or cyclic skills, lacking complex multi-level reasoning and discrete decision-making, which restricts their ability to perform complex goals involving movement and object interaction.
Innovation Solution
A multi-level hierarchical reinforcement learning (HRL) model is employed, comprising a higher-level policy model and a lower-level policy model, where the higher-level model generates goals and the lower-level model produces direct control actions, enabling temporally extended planning and off-policy training with an off-policy correction mechanism to adapt to changing lower-level policy behaviors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deep reinforcement learning is used for robotic control tasks, then the robot can perform atomic skills episodically or cyclically, but it cannot perform complex multi-level reasoning or discrete decision-making
Solution Approach 1:
The control system is segmented into multiple hierarchical levels: a high-level policy model that performs discrete decision-making and temporally extended planning, and a low-level policy model that executes atomic control actions. This segmentation enables complex multi-level reasoning by dividing the control architecture into specialized components that operate at different temporal and functional levels.
2Speed
If the higher-level policy model generates goals at every control step, then the robot can respond quickly to changes, but the training requires extensive experience data and computational resources
Solution Approach 1:
The higher-level policy model generates goals periodically rather than at every control step. Specifically, it generates a goal every c control steps (where c > 1), and a fixed goal transition function interpolates between these periodic goals. This periodic operation reduces the computational burden and experience data requirements during training while maintaining responsive control through the low-level policy model that operates at every step.
3Reliability
If the lower-level policy model is trained using task-based environment reward, then the training aligns with the ultimate goal, but the training requires extensive experience data and more iterations
Solution Approach 1:
The lower-level policy model is trained preliminarily using an intrinsic reward signal based on goal achievement rather than the final task-based environment reward. This preliminary training with goal-oriented intrinsic rewards allows the low-level model to learn effective control policies faster, before being fine-tuned with the ultimate task rewards. This approach accelerates training while maintaining alignment with the ultimate goal through the hierarchical structure.
4Productivity
If off-policy training is used for the lower-level policy model, then training requires fewer iterations and less experience data, but the training becomes more complex to coordinate with the higher-level model
Solution Approach 1:
A goal transition function serves as an intermediary between the higher-level and low-level policy models during off-policy training. This function generates intermediate goals that bridge the temporal gap between high-level decisions and low-level executions, enabling the low-level model to be trained off-policy with fewer iterations while maintaining coordination with the high-level model through the mediating goal structure.
Data Source
AI summary
Training and/or utilizing a hierarchical reinforcement learning (HRL) model for robotic control. The HRL model can include at least a higher-level policy model and a lower-level policy model. Some implementations relate to technique(s) that enable more efficient off-policy training to be utilized in training of the higher-level policy model and/or the lower-level policy model. Some of those implementations utilize off-policy correction, which re-labels higher-level actions of experience data, generated in the past utilizing a previously trained version of the HRL model, with modified higher-level actions. The modified higher-level actions are then utilized to off-policy train the higher-level policy model. This can enable effective off-policy training despite the lower-level policy model being a different version at training time (relative to the version when the experience data was collected).