Reinforcement learning device, reinforcement learning method, and reinforcement learning program
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reinforcement learning techniques face challenges with slow learning convergence and the risk of falling into local solutions due to incomplete measures and inefficient search strategies.
Innovation Solution
A reinforcement learning device and method that dynamically adjusts the search space based on prediction rewards, using an agent model estimation unit, a behavior determination unit, and a search amount estimation unit to update the agent model and search amount iteratively until predetermined conditions are met.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If reinforcement learning is performed with a fixed search amount, then the learning process is simple to implement, but the learning convergence time becomes excessively long
Solution Approach 1:
The patent applies dynamics by making the search amount variable rather than fixed. The search amount estimation unit dynamically adjusts the search amount based on the prediction reward from previous trials, allowing the system to adapt the exploration intensity to the current learning state and environment feedback, thereby reducing convergence time without excessive complexity increase
Solution Approach 2:
The patent implements feedback by using the prediction reward obtained from previous trials to update the search amount for subsequent trials. This closed-loop feedback mechanism allows the system to learn from past performance and continuously optimize the search strategy, balancing exploration and exploitation efficiently
2Reliability
If the search space is kept large to avoid local solutions, then the risk of falling into local solutions decreases, but the calculation time per trial increases excessively
Solution Approach 1:
The patent applies dynamics by dynamically adjusting the search space size (search amount) based on the prediction reward. When the prediction reward indicates good performance, the search amount is reduced to decrease calculation time. When the prediction reward suggests poor performance or potential local optima, the search amount is increased to maintain exploration and avoid local solutions
Solution Approach 2:
The patent changes the parameter of search amount based on the prediction reward. This parameter adjustment allows the system to adapt the balance between exploration (larger search space) and exploitation (smaller search space) to the current learning state, optimizing both solution quality and computational efficiency
Data Source
AI summary
Provided is a reinforcement learning device that performs reinforcement learning for a continuous behavior space. In the reinforcement learning device, predetermined settings for simulation and an agent model are stored. The reinforcement learning device includes: an agent model estimation unit that inputs a state acquired by the simulation to the agent model and acquires a measure; a behavior determination unit that calculates a behavior based on the measure and a search amount defined in advance; and a search amount estimation unit for estimating the search amount, the agent model estimation unit updates the agent model according to the setting of the agent model based on the state, a reward, a flag, and the behavior, the search amount estimation unit updates the search amount based on a prediction reward obtained for the reward and the search amount in a previous trial, and the calculation of the behavior, the update of the agent model, and the update of the search amount are repeated until a predetermined condition according to the flag and the setting is satisfied.