Intermediate RL Policy Transitions With Shrinking State Boundaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing reinforcement learning systems for quadruped locomotion face challenges in seamlessly transitioning between multiple independent policies due to the need for retraining when new policies are added, and the complexity of high-level policies increases with the number of low-level policies.
Innovation Solution
An intermediate policy training method using reinforcement learning with an annealing function to set shrinking tolerance boundaries, allowing smooth transitions between any two policies by training an agent to stay within these boundaries during episodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a hierarchical approach is used to connect multiple low-level policies, then the system can coordinate multiple policies, but retraining is required when new policies are added and the high-level policy complexity increases
Solution Approach 1:
The patent introduces an intermediate policy as a mediator between source and target policies. This intermediate policy learns transition behaviors through reinforcement learning with shrinking tolerance boundaries, enabling seamless policy switching without requiring the high-level policy to directly manage all low-level policies. The intermediate policy acts as a buffer that simplifies the hierarchical structure and eliminates the need for retraining when new policies are added.
Solution Approach 2:
The patent segments the policy transition problem into independent source-policy, intermediate-policy, and target-policy components. Each policy can be trained and updated independently, with the intermediate policy handling the transition logic. This segmentation allows new policies to be added without affecting existing ones, reducing overall system complexity while maintaining adaptability.
2Ease of manufacture
If independent RL policies are used for each low-level task, then each policy can be trained separately, but transitioning between policies is not straightforward
Solution Approach 1:
The intermediate policy serves as a bridge that facilitates smooth transitions between independently trained policies. It learns the transition dynamics through reinforcement learning with annealing functions that gradually shrink tolerance boundaries, ensuring continuous and stable policy switching. This mediator approach maintains the independence benefit for training while solving the transition difficulty.
3Reliability
If the high-level policy is conditioned on low-level policies, then coordination is achieved, but adding new policies requires retraining the whole system
Solution Approach 1:
The patent prepares transition policies in advance through the intermediate policy that learns general transition behaviors. Instead of retraining the entire hierarchical system when adding new policies, the intermediate policy can handle transitions to new policies using pre-learned strategies and shrinking boundary mechanisms, significantly reducing retraining time while maintaining coordination reliability.
Data Source
AI summary
An intermediate policy training method for agent is provided. This method includes: selecting a source and a target policy from a plurality of policies. Each policy is configured to drive an agent to perform a plurality of actions to be in a plurality of states. Each state includes a plurality of physical properties. The method further includes: respectively selecting one from the source policy and the target policy as a source state and a target state; training an intermediate policy by a reinforcement learning to transition the agent from the source state to the target state over an episode, where the reinforcement learning includes an annealing function for setting a plurality of tolerance boundaries of the plurality of physical properties, and the tolerance boundaries gradually shrink during the episode.