Unbiased Optimization Evaluation via Non-Overlapping Sample Groups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In prediction optimization, it is challenging to evaluate the effectiveness of optimal strategies without optimistic bias, as existing methods tend to overestimate results, and there is a lack of a general principle for unbiased estimators across various applications.
Innovation Solution
An evaluation system that generates multiple prediction models using non-overlapping sample groups, optimizes an objective function based on predicted variables, and evaluates optimization results using a sample group not used in learning, thereby mitigating optimistic bias through cross-validation and empirical risk minimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If prediction models are trained on available data to optimize strategies, then the accuracy of prediction improves, but optimistic bias in evaluation increases
Solution Approach 1:
The patent segments the available data into multiple non-overlapping sample groups (first sample group, second sample group, third sample group) to enable independent evaluation. This segmentation allows the system to train prediction models on one group while evaluating on another, thereby reducing optimistic bias and improving evaluation reliability without sacrificing prediction accuracy.
Solution Approach 2:
The patent introduces an intermediary evaluation mechanism using a third sample group that has not been used for training any prediction models. This intermediary group serves as an unbiased reference for evaluating the optimized strategy, mediating between the training process and final evaluation to eliminate overfitting effects and optimistic bias.
2Ease of operation
If the same data is used for both training and evaluation, then the evaluation process is simple, but the evaluation result becomes overly optimistic
Solution Approach 1:
The patent divides the data into distinct segments (sample groups) with specific assignments: the first sample group trains the initial prediction model, the second sample group trains alternative prediction models, and the third sample group evaluates the optimized strategy. This segmentation maintains evaluation simplicity through systematic organization while ensuring evaluation accuracy by preventing data reuse.
Solution Approach 2:
The patent performs preliminary actions by pre-dividing the data into non-overlapping sample groups before the optimization process. This preliminary segmentation establishes clear boundaries between training and evaluation data, allowing the evaluation process to proceed systematically without the need for complex cross-validation procedures, thus maintaining simplicity while ensuring accuracy.
Data Source
AI summary
A learning unit 81 generates a plurality of sample groups from samples to be used for learning, and generates a plurality of prediction models while inhibiting overlapping of a sample group to be used for learning among the generated sample groups. An optimization unit 82 generates an objective function based on an explained variable predicted by the prediction model and based on a constraint condition for optimization, and optimizes a generated objective function. An evaluation unit 83 evaluates an optimization result by using a sample group that has not been used in learning of a prediction model used for generating an objective function targeted for the optimization.