Machine Learning Accuracy Checks With Dynamic Data Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in balancing resource usage with performance, as retraining can be resource intensive and impact model accuracy and system efficiency, with traditional methods providing static and often inaccurate accuracy calculations.
Innovation Solution
Implementing a dynamic data selection method that adjusts the number of data point pairs used for accuracy calculations based on previous iterations and actions, triggering retraining or updated training when accuracy fails to meet criteria, thereby optimizing resource utilization and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If retraining of the machine learning model is performed frequently to maintain accuracy, then model accuracy is improved, but resource consumption increases
Solution Approach 1:
The patent implements dynamic adjustment of the data subset size used for accuracy evaluation. The system starts with a smaller data subset and progressively increases it if accuracy thresholds are not met, rather than using a fixed large dataset for every evaluation. This dynamic approach reduces resource consumption during accuracy checks while maintaining the ability to detect when retraining is truly needed.
Solution Approach 2:
The system performs accuracy evaluation using only a partial subset of available data rather than the complete dataset. By evaluating model accuracy on a smaller, representative subset of data points, the system reduces the computational resources required for accuracy checks while still obtaining meaningful accuracy measurements to trigger retraining when necessary.
2Reliability
If retraining is triggered based on static accuracy thresholds, then model accuracy is maintained, but unnecessary retraining occurs increasing resource usage
Solution Approach 1:
The system dynamically adjusts the evaluation stringency by varying the size of the data subset used for accuracy checks. When the model is performing well, smaller subsets are used for evaluation, reducing the likelihood of spurious accuracy failures. When accuracy concerns arise, larger subsets provide more rigorous evaluation. This dynamic approach reduces unnecessary retraining triggers while maintaining appropriate oversight of model performance.
Solution Approach 2:
The patent changes the parameter of data subset size based on the iteration number and previous actions taken. The system modifies this parameter dynamically throughout the evaluation process, starting with smaller subsets and increasing them if needed, rather than using a fixed parameter. This allows the system to balance between reducing unnecessary retraining and maintaining accurate performance monitoring.
3Measurement precision
If a large number of data point pairs is used for accuracy calculation, then accuracy measurement precision is improved, but calculation time increases
Solution Approach 1:
The system uses a partial subset of available data point pairs for accuracy calculation rather than processing all available data. This partial action approach provides sufficient measurement precision to determine whether the model meets accuracy thresholds while significantly reducing the calculation time required compared to using the complete dataset.
Solution Approach 2:
The data subset size used for accuracy calculation is dynamically adjusted based on the iteration context and previous performance. The system starts with smaller subsets for faster initial evaluations and can increase the subset size if more precise measurement is needed, creating a dynamic balance between measurement precision and calculation time.
Data Source
AI summary
Embodiments implement a machine learning prediction model with dynamic data selection. A number of data predictions generated by a trained machine learning model can be accessed, where the data predictions include corresponding observed data. An accuracy for the machine learning model can be calculated based on the accessed number of data predictions and the corresponding observed data. The accessing and calculating can be iterated using a variable number of data predictions, where the variable number of data predictions is adjusted based on an action taken during a previous iteration, and, when the calculated accuracy fails to meet an accuracy criteria during a given iteration, a training for the machine learning model can be triggered.