Reinforcement Learning Model Selection by Dual Index Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In reinforcement learning, especially when rewards are sparse or high-dimensional states are involved, learning models may not converge to good performance, leading to inefficient use of resources and prolonged calculation times due to the need for empirical reward design and frequent evaluation with test data.
Innovation Solution
A learning device and method that includes a reinforcement learning unit, evaluation index value calculation unit, and model extraction unit to select a trained model with predetermined performance criteria, such as cumulative reward and winning rate, without requiring extensive evaluation during learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reinforcement learning is performed with sparse rewards or high-dimensional states, then learning stability deteriorates, but evaluation with test data increases calculation time and resource consumption
Solution Approach 1:
The patent applies preliminary action by calculating multiple evaluation index values (first index value and second index value) during the learning process itself, rather than waiting until the end to evaluate with test data. This allows the system to predict model performance in advance and identify suitable trained models before formal evaluation, thereby reducing calculation time and resource consumption while maintaining learning stability
Solution Approach 2:
The patent implements feedback mechanisms by continuously calculating evaluation index values during learning and using these values to determine whether to continue or terminate learning. The system feeds back the first index value (indicating reward acquisition tendency) and second index value (indicating performance) to adjust the learning process, enabling stable learning convergence without extensive final evaluation
2Measurement precision
If detailed evaluation of trained models is performed using test data, then model performance accuracy improves, but computer resources are excessively occupied
Solution Approach 1:
The patent applies partial action by performing evaluation based on two specific index values (first index value for reward acquisition and second index value for performance) rather than comprehensive test data evaluation. This partial evaluation approach provides sufficient information to identify suitable trained models without consuming excessive computer resources, achieving a balance between evaluation accuracy and resource efficiency
3Reliability
If learning continues until convergence, then model performance improves, but learning time increases and resources are occupied
Solution Approach 1:
The patent implements feedback by continuously monitoring the first index value (reward acquisition tendency) and second index value (performance) during learning. When the first index value indicates sufficient reward acquisition capability and the second index value shows adequate performance, the system terminates learning early, avoiding unnecessary continued training that would consume additional time and resources while still achieving good model performance
Data Source
AI summary
There is provided a learning device that performs learning of a learning model of an agent, the learning device including: a reinforcement learning unit that performs learning of the learning model such that a reward assigned to the agent under a predetermined environment is maximized; an evaluation index value calculation unit that calculates a first index value and a second index value of the learning model; and a model extraction unit that extracts, as a trained model, the learning model in which the number of learning steps is equal to or larger than a predetermined number. The model extraction unit selects, as the trained model to be evaluated, the trained model in which each of the first index value and the second index value satisfies a predetermined condition, from the trained models.