Hierarchical Reinforcement Learning Constraint Sequencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
State-of-the-art Reinforcement Learning (RL) algorithms fail in hierarchical tasks, necessitating a technique for Hierarchical Reinforcement Learning (HRL) that can effectively learn and prioritize tasks within a hierarchical structure.
Innovation Solution
A method for Hierarchical Reinforcement Learning that involves learning a sequence of constraints by repeating reinforcement learning and supervised learning with good and bad samples, and applying these constraints to subsequent tasks, allowing for reduced training time and task prioritization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If state-of-the-art RL algorithms are used for hierarchical tasks, then the algorithms can learn optimal actions in complex environments, but they completely fail in hierarchical tasks
Solution Approach 1:
The patent segments the hierarchical task into multiple levels, where each level learns constraints that guide the next level. The hierarchy is divided into parent tasks and child tasks, with each level having its own constraint learning process. This segmentation allows standard RL algorithms to succeed at each level while collectively solving the hierarchical problem.
Solution Approach 2:
The patent applies preliminary action by learning constraints at each hierarchical level before proceeding to the next level. Constraints are learned from good and bad samples, then applied to guide subsequent learning at deeper levels. This preliminary constraint learning prevents failure in hierarchical tasks by establishing proper guidance structures in advance.
2Productivity
If reinforcement learning is repeated with good and bad samples to learn constraints, then task prioritization and reduced training time are achieved, but the complexity of the learning process increases
Solution Approach 1:
The patent introduces constraints as intermediary elements that mediate between good and bad samples and the final policy. Instead of directly learning from samples, the system learns constraints that act as intermediaries to guide learning at each hierarchical level. This intermediary constraint layer simplifies the overall learning process while maintaining efficiency.
Solution Approach 2:
The system performs self-service by automatically learning constraints from its own good and bad samples without external intervention. The constraint learning process is self-contained, using the agent's own experiences to generate training data and derive constraints, thereby reducing the need for external guidance or complex training procedures.
Data Source
AI summary
A computer-implemented method, computer program product, and computer processing system are provided for Hierarchical Reinforcement Learning (HRL) with a target task. The method includes obtaining, by a processor device, a sequence of tasks based on hierarchical relations between the tasks, the tasks constituting the target task. The method further includes learning, by a processor device, a sequence of constraints corresponding to the sequence of tasks by repeating, for each of the tasks in the sequence, reinforcement learning and supervised learning with a set of good samples and a set of bad samples and by applying an obtained constraint for a current task to a next task.