Multi-Critic Policy Learning for Safe Robotic Skill Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robots with high degrees of freedom operating in continuous state spaces face challenges in efficiently exploring manipulation strategies, planning trajectories, and ensuring safety in shared workspaces, which complicates their integration into real-world applications.
Innovation Solution
A multi-critic architecture is employed to learn a policy using multiple critics with respective objective functions that balance accuracy, exploration, and safety, minimizing interference between reward functions and enabling stable policy learning for robotic devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple reward functions are used to guide skill discovery, then the robot can explore diverse manipulation strategies, but interference between reward functions destabilizes policy learning
Solution Approach 1:
The patent divides the single policy learning process into multiple independent critic networks, each responsible for evaluating one specific reward function. This segmentation allows each critic to provide specialized feedback without interference from other reward functions, resolving the contradiction between exploring diverse skills and maintaining stable learning.
Solution Approach 2:
The patent introduces critic networks as intermediary components between the policy and multiple reward functions. These critics act as mediators that translate different reward signals into standardized value estimates, enabling the policy to learn from multiple objectives simultaneously without direct conflict between reward functions.
2Adaptability or versatility
If the state space is expanded to cover continuous positions and orientations, then the robot can operate in more complex environments, but the search space becomes too large for efficient exploration
Solution Approach 1:
The patent applies preliminary action by using curiosity-driven exploration to proactively discover important states and transitions before formal skill learning begins. This preliminary exploration phase identifies key regions in the continuous state space, allowing subsequent skill learning to focus on these pre-identified areas rather than searching the entire space.
Solution Approach 2:
The patent transforms the continuous state space problem by introducing a discrete skill representation dimension. Instead of directly learning continuous control policies, the system learns discrete skill transitions that map continuous states to skill identifiers, effectively reducing the computational complexity of exploring continuous spaces.
3Adaptability or versatility
If exploration is encouraged to discover new skills, then the robot finds more manipulation strategies, but safety constraints may be violated during exploration
Solution Approach 1:
The patent applies beforehand cushioning by using safety critics that are trained in advance to recognize unsafe states and transitions. These safety critics provide protective feedback during exploration, cushioning against harmful actions before they can cause damage, while still allowing beneficial exploration to proceed.
Solution Approach 2:
The patent converts the potential harm of unsafe exploration into benefit by using safety violations as learning signals. The safety critics identify violated constraints and use this information to guide the policy toward safer regions of the state space, transforming harmful exploration attempts into constructive learning opportunities.
Data Source
AI summary
Systems and methods are disclosed for determining a policy to recommend transition in a position-representing space for a robotic device using a multi-critic architecture. To learn policy in a multi-critic architecture, a set of critics is defined pertaining to a position-representing space where each critic corresponds to a different objective function such as reach-reward, discovery-reward, and safety-reward. For each one of the critics of the set of critics, a learned value function in position-representing space is determined. The policy is learned based on the weighted feedback of the learned value functions to recommend transitions that are safe in the position-representing space. The multi-critic architecture minimizes interference between multiple reward functions and learns a safe and stable policy for the robotic device.