Support Policy Learning for Robot Adaptability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning (RL) agents face challenges in generalizing solutions to broader tasks due to large or infinite state and action spaces, sample inefficiency, sparse rewards, credit assignment issues, and catastrophic forgetting, especially when transferring policies from simpler to more complex tasks.
Innovation Solution
The method involves a support policy learning (SPL) system where a main policy and a support policy are used within the same action and state spaces, with a master policy deciding between executing the main policy or learning a support policy based on a general value function's predicted accumulated success value, allowing for efficient reuse of existing solutions and minimizing catastrophic forgetting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning agents are trained to solve complex tasks with large or infinite state and action spaces, then the ability to solve specific tasks is improved, but sample efficiency deteriorates and training becomes time-consuming
Solution Approach 1:
The patent applies preliminary action by pre-training RL agents on simpler tasks before transferring knowledge to complex tasks. The system breaks down complex tasks into simpler sub-tasks, trains agents on these sub-tasks first, and then transfers the learned policies to solve the full complex tasks, significantly reducing training time while maintaining adaptability
Solution Approach 2:
The patent segments complex tasks into simpler sub-tasks with smaller state and action spaces. By dividing the overall task into manageable components, the system can train agents more efficiently on each segment and then compose the solutions to solve the complete complex task
2Productivity
If existing solutions are reused through transfer learning to speed up training on new tasks, then training efficiency is improved, but catastrophic forgetting occurs where the agent loses ability to solve original tasks
Solution Approach 1:
The patent applies dynamics by implementing a dynamic policy selection mechanism that chooses between different trained policies based on the current task requirements. The system maintains multiple policies for different tasks and dynamically selects the appropriate policy, allowing efficient transfer learning while preventing catastrophic forgetting through selective policy application
Solution Approach 2:
The patent uses an intermediary task representation layer that bridges simple and complex tasks. This intermediary representation allows knowledge transfer while maintaining task-specific characteristics, enabling the agent to apply learned patterns to new tasks without losing the ability to perform original tasks
3Adaptability or versatility
If function approximation with deep learning is applied to handle large state and action spaces, then the capability to represent complex states is improved, but sample efficiency further deteriorates
Solution Approach 1:
The patent applies local quality by using different function approximation methods for different parts of the state space. Instead of uniformly applying deep learning across all states, the system uses simpler approximation methods where sufficient and reserves deep learning for regions requiring higher representational capacity, reducing overall training time while maintaining necessary adaptability
Data Source
AI summary
Methods and systems are described for support policy learning in an agent of a robot. A general value function (GVF) is learned for a main policy, where the GVF represents future performance of the agent executing the main policy for a given state of the environment. A master policy selects an action based on the predicted accumulated success value received from the general value function. When the predicted accumulated success value is an acceptable value, the action selected by the master policy is execution of the main policy. When the predicted accumulated success value is not an acceptable value, the master action causes a support policy to be learned. The support policy generates a support action to be performed which causes the robot to transition from to a new state where the predicted accumulated success value has an acceptable value.