Reinforcement Learning Policy Training With Mimicry Rewards
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In reinforcement learning, particularly for training defense AI against attack AI, learning becomes difficult when sparse rewards are obtained, leading to a lack of good experience data for effective training.
Innovation Solution
A reinforcement learning device that generates mimicry rewards to evaluate how closely an action mimics normal behavior, collects experience data based on these rewards, and learns a policy to improve training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If defense AI is trained together with attack AI to improve defense capability, then defense capability is improved, but learning becomes difficult due to sparse rewards
Solution Approach 1:
The patent introduces a behavior generation unit that creates mimicry behaviors as an intermediary between the attack AI and defense AI. This mediator generates training data that provides dense rewards to the attack AI, enabling it to learn effective attack strategies without directly competing with the defense AI, thereby resolving the sparse reward problem while maintaining improved defense capability through joint training
Solution Approach 2:
The patent applies preliminary action by pre-generating mimicry behaviors through the behavior generation unit before the main training process. These pre-generated behaviors serve as a foundation for training the attack AI, allowing it to learn from dense reward signals associated with mimicking normal behaviors, which then enables more effective learning against the defense AI
2Reliability
If sparse reward learning continues until high reward is obtained, then learning completeness is improved, but learning efficiency deteriorates
Solution Approach 1:
The patent applies partial action by separating the learning process into two stages: first, the attack AI learns to generate mimicry behaviors with dense rewards (partial learning objective); second, it applies this knowledge to learn effective attack strategies (complete learning objective). This partial approach to learning mimicry first improves overall learning efficiency while maintaining completeness
Solution Approach 2:
The mimicry behavior generation acts as an intermediary training objective that provides dense rewards during the learning process. This intermediate training phase with abundant rewards improves learning efficiency, while the final attack learning phase ensures learning completeness, thus resolving the contradiction between efficiency and completeness
Data Source
AI summary
To reduce difficulty in learning. A reinforcement learning device includes: a generation unit configured to generate a behavior of an environment; a calculation unit configured to calculate, based on an action on the environment and the behavior generated by the generation unit, a mimicry reward indicating how much the action mimics the behavior; a collection unit configured to select the action on the environment based on a policy and collect experience data including the action, a state of the environment when the action is performed on the environment, and a reward obtained from the environment as a result of performing the action; and a learning unit configured to learn the policy based on the reward collected by the collection unit and the mimicry reward calculated by the calculation unit.