Automated Regression Test Set Creation for Machine Learning Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning model regression testing is challenging due to the difference in software development paradigms between traditional control logic-based programs and machine learning systems, where new data may not cover all patterns and labels used in previous versions, leading to potential incorrect labeling issues when new data distributions shift.
Innovation Solution
An automated method for creating data and label subsets for supervised machine learning regression testing, involving splitting input datasets into training and test sets, training multiple machine learning models, applying them to the test set, computing agreement metrics, and selecting strongly agreed data examples to form a regression test set.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If new data is used to train machine learning models, then the models can adapt to emerging data patterns, but the models may incorrectly label previously correct data due to distribution shifts
Solution Approach 1:
The patent applies preliminary action by creating a regression test set before deploying new machine learning models. This regression test set contains data examples with true labels that serve as a baseline for comparing new model versions. By establishing this reference dataset in advance, the patent enables automated regression testing that detects when new models incorrectly label previously correct data, thus preventing reliability degradation while allowing adaptability to new patterns.
2Reliability
If manual test data design is performed for regression testing, then comprehensive test coverage can be achieved, but the process becomes extremely difficult and time-consuming
Solution Approach 1:
The patent applies self-service by enabling machine learning models to automatically generate and maintain their own regression test sets. The system uses the models' own predictions and agreement metrics to identify suitable data examples for the regression test set, eliminating the need for manual test data design. This automated approach maintains comprehensive test coverage while dramatically reducing the time and effort required, as the system serves itself without human intervention in the test data creation process.
3Measurement precision
If multiple machine learning models are trained and applied to compute agreement metrics, then more reliable regression test data can be selected, but the computational complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the regression test set creation process into distinct phases: (1) splitting training data into multiple subsets, (2) training separate models on each subset, (3) applying models to test data independently, and (4) computing agreement metrics by comparing results. This segmented approach enables reliable regression testing through multiple models while managing computational complexity through systematic organization and parallel processing of each segment.
Data Source
AI summary
Split an input dataset into training and test datasets; the former includes a plurality of data examples, each represented as a feature vector, and having an associated true label. Split the training dataset into a plurality of training data subsets; for each, train a corresponding machine learning model to obtain a plurality of such models, and apply same to the test dataset to obtain a plurality of predicted labels and prediction scores. For each of the plurality of examples, compute an agreement metric based on a corresponding one of the associated true labels; corresponding ones of the predicted labels; and corresponding ones of the prediction scores. Based on the computed metric, select, for at least some of the true label values, appropriate ones of the data examples to be added to a regression set. Add the appropriate ones of the data examples from the test dataset to the regression set.