Predictive Model Accuracy Testing via Data Richness Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing predictive models face challenges in accurately assessing their performance over time due to changing input data, and managing large volumes of training data is resource-intensive, requiring skilled expertise and significant computational resources.
Innovation Solution
A computer-implemented system that assigns richness scores to data samples, ranks them, and selects a test data set to retrain predictive models, ensuring accuracy assessments are dynamic and reflective of current data, allowing for the selection and updating of the most accurate models based on new data sets, thereby reducing the burden on client computing systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large volumes of training data are retained and processed to maintain model accuracy over time, then predictive output accuracy is improved, but computational resources and storage requirements increase
Solution Approach 1:
The patent extracts only the most relevant and information-rich data samples from the training data set to form a reduced test data set. By identifying and removing redundant or less valuable data samples, the system maintains model accuracy while significantly reducing the computational resources required for testing and validation.
Solution Approach 2:
The patent applies different quality assessments to different data samples, identifying regions of high information value versus low information value. By focusing computational efforts on testing with high-quality, information-rich data samples rather than uniformly processing all data, the system achieves accurate model assessment with reduced resource consumption.
2Measurement precision
If comprehensive test data sets are used to accurately assess model performance, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the comprehensive training data set into distinct categories based on information richness and relevance. By dividing the data into high-value test samples and lower-value training samples, the system achieves precise model performance measurement using a manageable, segmented data approach rather than attempting to process the entire data set.
Solution Approach 2:
The patent applies partial action by using only a portion of the available training data for testing purposes. By selecting and using only the most informative data samples for model assessment rather than the complete data set, the system achieves sufficient measurement precision while avoiding the complexity of managing and processing all available data.
3Adaptability or versatility
If training data is continuously updated with new data sets, then adaptability is improved, but loss of time in data processing increases
Solution Approach 1:
The patent performs preliminary assessment and categorization of new data samples as they arrive, identifying their information richness and relevance before full processing. By pre-sorting and prioritizing data samples based on their potential value, the system enables continuous model adaptation while minimizing processing delays, as high-value data can be quickly identified and integrated.
Solution Approach 2:
The patent establishes a continuous data processing pipeline where new data samples are constantly evaluated, categorized, and integrated into the training data repository. By maintaining continuous rather than batch processing, the system achieves ongoing model adaptability while keeping individual processing steps efficient and time-constrained.
Data Source
AI summary
A system includes a computer(s) coupled to a data storage device(s) that stores a training data repository and a predictive model repository. The training data repository includes retained data samples from initial training data and from previously received data sets. The predictive model repository includes at least one updateable trained predictive model that was trained with the initial training data and retrained with the previously received data sets. A new data set is received. A richness score is assigned to each of the data samples in the set and to the retained data samples that indicates how information rich a data sample is for determining accuracy of the trained predictive model. A set of test data is selected based on ranking by richness score the retained data samples and the new data set. The trained predictive model is accuracy tested using the test data and an accuracy score determined.