Domain-Split Skill Embeddings for Cross-Domain Policy Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing skill-based reinforcement learning methods struggle with limited diversity of learned skills, leading to difficulty in adapting policies to domains different from the training data set, resulting in inefficient policy adaptation.
Innovation Solution
A policy learning apparatus and method that utilizes an encoder to obtain domain-invariant and domain-variant skill embeddings, combined with a decoder to expand skill diversity and adapt policies to new domains through few-shot imitation learning and online reinforcement learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hierarchical skill learning structure is used to learn skills from offline data set, then learning efficiency for long-horizon tasks is improved, but skill diversity is limited to the training domain
Solution Approach 1:
The skill embedding is segmented into two independent components: domain-invariant skill embedding (capturing task-specific skills) and domain-variant skill embedding (capturing domain-specific characteristics). This segmentation allows the system to learn efficient skills from offline data while maintaining adaptability to different domains through separate encoding pathways.
Solution Approach 2:
The encoder is designed with multi-functionality to simultaneously produce both domain-invariant and domain-variant skill embeddings from the same input data. This universal encoder can adapt to different domains while preserving core task skills, enabling the system to achieve both learning efficiency and domain adaptability.
2Measurement precision
If skills are learned from domain-specific data set, then learning precision for that domain is improved, but adaptability to different domains deteriorates
Solution Approach 1:
The system applies local quality by making different parts of the skill embedding serve different purposes: the domain-invariant part maintains high learning precision for the training domain, while the domain-variant part enables adaptation to local characteristics of different domains. This localized specialization resolves the contradiction between precision and adaptability.
Solution Approach 2:
The skill representation is extended to a higher-dimensional space by adding the domain-variant embedding dimension. This dimensional expansion allows the system to maintain precise task skills in one dimension while gaining domain adaptability in another dimension, effectively resolving the precision-adaptability trade-off.
3Speed
If pre-trained skill is used for policy adaptation, then adaptation speed is improved, but skill applicability to different domains is limited
Solution Approach 1:
The skill embedding structure is made dynamic by introducing domain parameters that can be adjusted based on the target domain. The domain-variant skill embedding allows the pre-trained skills to dynamically adapt to different domains while maintaining the core task capabilities, enabling both fast adaptation and broad applicability.
Data Source
AI summary
The present invention relates to a skill diffusion-based policy learning device and method, as well as a diffused skill-based policy-based operation device. A policy learning apparatus comprises an encoder configured to be trained to obtain a domain-invariant skill embedding based on at least one state and at least one action, and obtain a domain-variant skill embedding based on a domain parameter corresponding to the at least one state and the at least one action and the domain-invariant skill embedding and a decoder configured to be trained to obtain a skill as an output value using the domain-invariant skill embedding and the domain-variant skill embedding as inputs.