Reinforcement Learning Reward Correction for Local Optima
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning methods often converge to local optima early in the learning process due to underestimation of cumulative rewards, leading to incorrect learning outcomes, especially when the probability density of action selection is calculated unreasonably low.
Innovation Solution
The proposed learning method corrects the reward by adding a second reward that increases as the probability density decreases, updating control parameters based on current and next observation data, and using a neural network to estimate cumulative rewards, thereby preventing convergence to local optima.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reinforcement learning is applied to action control, then learning can be performed through repeated evaluation, but learning may converge to a local optimum in the early stage
Solution Approach 1:
The patent applies preliminary action by performing exploration actions before formal learning begins. The system executes a predetermined number of exploration actions to collect initial data about the environment, which serves as a foundation for subsequent learning. This preliminary data collection helps prevent early convergence to local optima by ensuring the learning algorithm starts with a broader understanding of the action space and reward structure.
2Ease of operation
If probability density of action selection is calculated, then action selection can be optimized, but probability density may be calculated unreasonably low leading to incorrect learning
Solution Approach 1:
The patent applies preliminary anti-action by implementing a mechanism that counteracts the tendency of probability density to become unreasonably low. When the calculated probability density falls below a predetermined threshold, the system adds exploration bonuses to the reward signal, which prevents the probability from collapsing to zero. This counter-action ensures that actions with potentially high value are not prematurely discarded, maintaining measurement precision in probability density calculation.
3Loss of information
If cumulative reward is estimated, then learning direction can be determined, but cumulative reward may be underestimated leading to local optimum convergence
Solution Approach 1:
The patent applies feedback by implementing a mechanism that monitors the probability density of action selection and provides corrective feedback when underestimation is detected. When an action's probability density falls below the threshold, the system feeds back an exploration bonus to the cumulative reward calculation, ensuring that the reward signal accurately reflects both the immediate reward and the potential long-term value of exploratory actions. This feedback loop prevents systematic underestimation of cumulative rewards.
Solution Approach 2:
The system performs preliminary exploration actions to gather initial information about cumulative rewards before formal learning begins. This preliminary data collection ensures that the cumulative reward estimation is based on actual observed outcomes rather than initial guesses, reducing the risk of underestimation and subsequent convergence to local optima.
Data Source
AI summary
According to one embodiment, a learning method includes calculating a probability distribution indicating a distribution of a probability density or a distribution of a probability at which actions are selected, selecting a first action based on the probability distribution, causing a control target to execute the first action, receiving a reward and next observation data, calculating a probability density or a probability of the first action, correcting the reward, and updating the control parameter. The reward is corrected such that the reward increases as the probability density or probability decreases.