Deep Reinforcement Learning Decision-Making With Selective Imitation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional online reinforcement learning methods require high data collection costs and conventional imitation learning limits the range of available datasets due to imitating undesirable actions.
Innovation Solution
A deep reinforcement learning-based decision-making method utilizing prior data and selective imitation learning, which includes a prior data collection unit, processing unit, and policy learning unit, with a selective imitation learning term in the objective function to prioritize high-reward actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional online reinforcement learning is used, then the agent can learn optimal policies through real-time interaction, but it requires high data collection costs and time
Solution Approach 1:
The patent applies preliminary action by collecting and processing prior data from other agents before the ego agent begins its learning process. The prior data collection unit gathers trajectory information from multiple other agents, and the prior data processing unit transforms this data into the required state-action-next state-reward format. This preliminary data preparation allows the policy learning unit to start learning from pre-processed data rather than collecting data in real-time, significantly reducing the time and computational resources required for policy learning while maintaining high learning speed.
2Productivity
If conventional imitation learning is used, then the agent can imitate trajectory information, but it imitates undesirable actions and limits the range of available datasets
Solution Approach 1:
The patent applies local quality by selectively using only the beneficial portions of prior data through the selective imitation learning term in the objective function. The objective function includes a selective imitation learning term with a selective imitation learning weight that determines the degree of imitation according to the magnitude of the reward. This allows the system to imitate high-reward actions from prior data while avoiding undesirable actions, effectively filtering and selecting only the quality portions of available data for learning.
Solution Approach 2:
The patent applies parameter changes by dynamically adjusting the selective imitation learning weight based on the reward magnitude of sampled data. When the reward of sampled data is greater than a preset threshold, the selective imitation learning term is added to the objective function with an adjusted weight. This parameter adjustment mechanism enables the system to adaptively select and emphasize high-reward actions while downweighting or excluding low-reward or undesirable actions, thereby expanding the effective range of usable datasets.
3Productivity
If prior data is collected and processed, then sample efficiency is improved, but the system complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the decision-making system into distinct functional modules: a prior data collection unit that gathers trajectory information from other agents, a prior data processing unit that transforms the data into the required format, and a policy learning unit that learns the policy using the processed data. This modular segmentation allows each component to perform its specific function independently, making the overall complex system more manageable and easier to implement while maintaining high sample efficiency through the coordinated operation of these segmented units.
Data Source
AI summary
A deep reinforcement learning-based decision-making apparatus through prior data and selective imitation learning is disclosed. The deep reinforcement learning-based decision-making apparatus comprises a prior data collection unit configured to collect prior data from one or more other agents; a prior data processing unit configured to process the collected prior data into data including state, action, next state, and reward; and a policy learning unit configured to learn policy of an ego agent using the processed prior data and interaction data including state, action, next state, and reward obtained through real-time interaction with environment.