Reinforcement Learning Policy Model Using Multiple Q-Function Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning methods, such as Q-learning, often require a significant amount of time to converge due to overestimation issues in the Q-function model, which hampers efficiency and speed.
Innovation Solution
The implementation of multiple Q-function models that calculate evaluation values with noise, allowing for the selection of the smallest value to update the policy model, thereby mitigating overestimation and stabilizing the learning process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single Q-function model is used for reinforcement learning, then the learning process is simple, but the time required for learning is long due to overestimation issues
Solution Approach 1:
The patent divides the single Q-function model into multiple Q-function models (first Q-function model and second Q-function model). Each model independently calculates Q-values, and the minimum value among them is selected. This segmentation reduces overestimation bias and accelerates learning convergence, directly addressing the contradiction between learning simplicity and learning speed.
Solution Approach 2:
The patent implements a feedback mechanism where the selected Q-value (minimum of multiple models) is used to update the policy, which in turn generates new experiences that feed back into the Q-function models. This feedback loop with multiple models provides more accurate value estimates, reducing the time required for effective learning while maintaining systematic updates.
2Measurement precision
If multiple Q-function models are used to reduce overestimation, then learning accuracy improves, but device complexity increases
Solution Approach 1:
The patent segments the evaluation function into multiple independent Q-function models that each calculate Q-values separately. By selecting the minimum value among these segmented models, the system achieves more accurate evaluation with reduced overestimation bias, accepting controlled complexity increase for significant accuracy improvement.
Solution Approach 2:
The patent changes the parameter selection criterion from using a single Q-value to using the minimum of multiple Q-values. This parameter change in the selection logic improves evaluation accuracy by reducing overestimation, while the computational overhead remains manageable through efficient implementation of the minimum selection among a limited number of models.
Data Source
AI summary
A learning device calculates each of a plurality of second evaluation values that include noise using a plurality of evaluation models, each of which calculates, on the basis of both a second state resulting from a first action performed by a control target in a first state, and a second action calculated from the second state using a policy model, a second evaluation value obtained by including noise in an index value indicating the result of evaluating the second action in the second state; and updates the policy model or the parameters of the policy model on the basis of the smallest of the plurality of second evaluation values and a first evaluation value, which is an index value indicating the result of evaluating the first action in the first state.