Prediction Model Evaluation Graph for Biased Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for evaluating prediction results, such as those using ROC curves, face challenges in determining cut-off values for significantly biased data, requiring extensive memory and processing resources due to the need to calculate and store true and false positive rates for all data selections.
Innovation Solution
An evaluation result display method that sorts records based on prediction values, calculates the number of selections and correct decisions, and generates a graph plotting these values, reducing memory usage by storing only two parameters and allowing for efficient evaluation of prediction results, especially for biased data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional ROC curve methods are used to evaluate prediction results, then measurement precision of prediction accuracy is improved, but device complexity and memory usage increase due to requiring calculation and storage of true and false positive rates for all data selections
Solution Approach 1:
The patent extracts only the essential information needed for evaluation by storing records in sorted order by prediction values and calculating metrics sequentially. Instead of storing all possible TP/FP rate combinations, it extracts and stores only the sorted records and computes metrics on-the-fly, significantly reducing memory requirements while maintaining evaluation precision.
Solution Approach 2:
The patent performs preliminary sorting of records by prediction values before evaluation. This preliminary action organizes the data in advance, enabling efficient sequential processing and cut-off value determination without requiring complex real-time calculations, thus reducing both memory usage and computational complexity.
2Measurement precision
If all data are checked and TP rates and FP rates are calculated in order to determine cut-off values, then measurement precision of cut-off value determination is improved, but loss of time increases due to extensive processing requirements
Solution Approach 1:
Records are pre-sorted by prediction values before evaluation begins. This preliminary organization allows the system to determine cut-off values by simple sequential scanning and counting rather than performing exhaustive calculations on all data combinations, dramatically reducing processing time while maintaining precise cut-off value determination.
Solution Approach 2:
The patent calculates metrics sequentially through the sorted records, computing TP and FP counts incrementally as it scans through the data. This partial action approach stops once sufficient information is gathered to determine optimal cut-off values, avoiding the need to process and store all possible evaluation scenarios.
3Measurement precision
If conventional evaluation methods are used for significantly biased data, then measurement precision of prediction evaluation is improved, but device complexity increases due to difficulty in determining useful cut-off values
Solution Approach 1:
By pre-sorting records according to prediction values, the system creates an organized structure that simplifies cut-off value determination even for biased data. This preliminary organization allows straightforward sequential processing and clear identification of optimal cut-off points without complex algorithms, reducing determination complexity while maintaining evaluation precision.
Solution Approach 2:
The patent segments the evaluation process into distinct sequential steps: sorting records, scanning through sorted data, counting TP and FP occurrences, and identifying cut-off values. This segmentation breaks down the complex evaluation task into manageable segments, making it easier to handle biased data systematically while maintaining measurement precision.
Data Source
AI summary
A method is provided for reducing computer memory usage and processing time in assessing effectiveness of a prediction model. The method includes acquiring a plurality of records and sorting the records based on prediction values. The method also includes determining two of three performance data parameters, including a number of selected records and a number of correct decisions, and determining a level of effectiveness of the prediction model based on the two of three performance data parameters. The method further includes generating a graph illustrating the level of effectiveness of the prediction model based on the two of three performance parameters, and visually conveying the level of effectiveness of the prediction model utilizing the generated graph.