Multiple Q-Function Models for Reinforcement Learning Stability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning processes are typically time-consuming due to issues with overestimation of Q-function models, which leads to unstable learning and prolonged training times.
Innovation Solution
The implementation of multiple Q-function models with different parameter values and prioritized experience replay to mitigate overestimation, where the smaller output value from these models is used for updating, and experiences with higher error priorities are used for learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If reinforcement learning is performed using conventional Q-function models, then the learning process can be executed, but the learning time becomes excessively long due to overestimation issues
Solution Approach 1:
The patent divides the single Q-function model into multiple Q-function models (first Q-function model and second Q-function model). Each model independently evaluates the Q-values, and the minimum value among them is selected for updating. This segmentation prevents overestimation by cross-validation among multiple models, thereby improving learning stability and reducing learning time.
2Reliability
If multiple evaluation models are used to calculate evaluation values, then overestiction is mitigated, but the device complexity increases
Solution Approach 1:
The patent merges multiple Q-function models into a unified framework where they share the same network architecture and parameter update mechanism. The models are trained jointly using the same loss function based on the minimum Q-value, which reduces the overall complexity compared to completely independent models while still providing the benefits of multiple evaluations.
3Productivity
If experiences with higher error priorities are prioritized for learning, then the learning efficiency is improved, but the difficulty of detecting and measuring increases
Solution Approach 1:
The patent implements a feedback mechanism where the priority of each experience is dynamically determined based on the absolute difference between Q-values from multiple models. Experiences with larger discrepancies receive higher priorities, creating a feedback loop that automatically directs learning resources to the most informative samples, thereby improving learning efficiency without requiring complex external detection systems.
Data Source
AI summary
A learning device uses, on the basis of a second state corresponding to a first action of a control subject in a first state and a second action calculated from the second state using a policy model, uses a plurality of evaluation models that each calculate a second evaluation value, which is an index value of the goodness of the second action in the second state, to respectively calculate the second evaluation values; and updates the evaluation model on the basis of the smallest second evaluation value among the plurality of second evaluation values, and a first evaluation value, which is an index value of the goodness of the first action in the first state.