Hierarchical Reinforcement Learning With Off-Policy Action Relabeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing robotic control systems lack the ability to perform complex multi-level reasoning tasks involving locomotion behaviors, interaction with objects, and discrete decision-making, and off-policy training in hierarchical reinforcement learning poses challenges due to changing lower-level policy models.
Innovation Solution
A multi-level hierarchical reinforcement learning model with a higher-level policy model and a lower-level policy model is trained using off-policy correction techniques, where past experiences are re-labeled with modified higher-level actions to align with the current lower-level policy model, enabling efficient training and control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If off-policy training is used to train hierarchical reinforcement learning models, then training efficiency and data efficiency are improved, but the training becomes challenging due to changing lower-level policy models
Solution Approach 1:
The training process is segmented into two distinct phases: first training the lower-level policy model independently using off-policy data, then training the higher-level policy model while keeping the lower-level model fixed. This segmentation resolves the instability caused by simultaneous changes in both models during off-policy training.
Solution Approach 2:
The lower-level policy model is trained in advance before training the higher-level policy model. This preliminary action ensures that when the higher-level model is trained, the lower-level model provides stable, predictable behavior, eliminating the reliability issues caused by concurrent model changes.
2Adaptability or versatility
If a hierarchical reinforcement learning model is trained with complex multi-level reasoning tasks, then the robot's ability to perform diverse behaviors is improved, but the amount of training data and computational resources required increases
Solution Approach 1:
The complex reinforcement learning task is segmented into two hierarchical levels: lower-level policy for basic actions and higher-level policy for strategic decision-making. This segmentation allows each level to be trained independently with smaller, more manageable datasets, reducing the total training data volume while maintaining behavioral diversity.
Solution Approach 2:
The lower-level policy model is trained in advance with basic tasks before the higher-level model is trained with complex reasoning tasks. This preliminary training of foundational skills reduces the computational resources and data needed for the higher-level model to learn complex behaviors from scratch.
3Speed
If the higher-level policy model generates actions at every control step, then the responsiveness to current state is improved, but the computational overhead and training complexity increase
Solution Approach 1:
The higher-level policy model generates actions periodically at selected control steps rather than at every step. This periodic action reduces computational overhead and model complexity while maintaining responsiveness through the lower-level policy model, which handles intermediate control steps using the higher-level model's periodic outputs.
Data Source
AI summary
Training and/or utilizing a hierarchical reinforcement learning (HRL) model for robotic control. The HRL model can include at least a higher-level policy model and a lower-level policy model. Some implementations relate to technique(s) that enable more efficient off-policy training to be utilized in training of the higher-level policy model and/or the lower-level policy model. Some of those implementations utilize off-policy correction, which re-labels higher-level actions of experience data, generated in the past utilizing a previously trained version of the HRL model, with modified higher-level actions. The modified higher-level actions are then utilized to off-policy train the higher-level policy model. This can enable effective off-policy training despite the lower-level policy model being a different version at training time (relative to the version when the experience data was collected).