Hierarchical Reinforcement Learning Subtask Graph Interpretability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks used in hierarchical reinforcement learning (HRL) for robot tasks generate non-interpretable sub-goals, making it difficult for human domain experts to understand the policy of the Manager agent.
Innovation Solution
Employing a feudal network architecture combined with graph neural networks to create a sub-task graph that provides an intuitive understanding of the Manager policy, using a graph convolutional neural network for the Manager agent and a deep neural network for the Worker agent, allowing for human interaction and interpretation, and using Generative Adversarial Networks or graph variational Autoencoders to generate and adapt the sub-task graph.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deep neural networks are used in hierarchical reinforcement learning for the Manager agent, then the device can learn complex policies and perform multi-task reinforcement learning, but the learned sub-goals become non-interpretable for humans
Solution Approach 1:
The patent segments the Manager agent's decision-making process into two distinct components: a Graph Convolutional Network (GCN) that generates human-interpretable subtask graphs, and a Deep Neural Network (DNN) that processes these graphs into actionable sub-goals. This segmentation allows the system to maintain both interpretability (through the GCN's explicit graph structure) and learning capability (through the DNN's processing), resolving the contradiction between adaptability and interpretability.
Solution Approach 2:
The patent introduces subtask graphs as an intermediary representation between the human-interpretable policy space and the DNN's latent space. These graphs serve as a bridge that preserves interpretability while enabling complex learning, as they can be visualized and understood by humans while still providing rich input for the DNN to learn from.
2Difficulty of detecting and measuring
If the Manager agent operates in high dimensional vector space to encode larger picture information, then the policy can consider more comprehensive factors, but the output sub-goals become less interpretable
Solution Approach 1:
The patent segments the information processing into two stages: first, the GCN processes comprehensive environmental information and maintains it in an interpretable graph format; second, the DNN processes this structured information in its high-dimensional space. This segmentation allows comprehensive information processing without sacrificing interpretability at the policy level.
Solution Approach 2:
The patent transitions from high-dimensional vector space to graph structure space, providing another dimensional representation that is inherently more interpretable. The graph structure preserves the relationships and dependencies between subtasks in a visually and conceptually understandable format, while still capturing comprehensive information.
3Measurement precision
If sparse rewards are used in reinforcement learning, then the learning signal reflects the true task completion accuracy, but the learning process becomes inefficient for complex long-term tasks
Solution Approach 1:
The patent segments the long-term task into multiple intermediate subtasks with their own sub-goals. The Worker agent receives denser rewards for achieving these sub-goals along the path to the final goal, improving learning efficiency. The GCN ensures these subtasks are properly segmented and ordered, maintaining the connection to the true task completion objective.
Solution Approach 2:
The patent performs preliminary decomposition of the complete task into subtasks before the learning process begins. This preliminary action creates a structured pathway with intermediate milestones, allowing the Worker agent to receive timely feedback and learn more efficiently without compromising the accuracy of the final reward signal.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
The invention relates to a method for operating a device by using hierarchical reinforcement learning, whereby a complete operation task is to perform by said device, said method having the following steps: a) Decomposing the complete operation task into a set of subtasks, b) Representing dependencies among all subtasks of the complete task by a subtask graph (ggm) being direct acyclic and having nodes connected by edges, said nodes representing said subtasks and said edges representing relations among these subtasks, c) using the subtask graph as input for a graph convolutional neural network in which from the input data by use of a convolutional operator for each network node a representation is obtained containing information about the relation of one network node and at least some neighbored network nodes and delivering encoded data in order to create a subtask graph vector (gv) in which a relation between nodes that are directly connected by an edge is encoded, d) using the subtask vector (gv) obtained in step c) as input data for a deep neural network by which further encoded data are obtained; e) Using said further encoded data to set up an action of the device (a) in order to perform at least one subtasks of the set of subtasks.