ML Error Policy Generation via Dataset Slicing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for fault localization in machine learning engines lack human-interpretability, compactness, and robustness, making it difficult to diagnose and improve the performance of these engines effectively.
Innovation Solution
A computer-implemented method using a slice generator and a policy generator to slice a dataset into interpretable and robust slices based on feature value conditions, generating an error policy that identifies observations with predicted values not equaling true values, and applying this policy to modify the machine learning engine and predict error concentrations in new datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If existing methods use metrics on individual observations (e.g., machine learning model confidence or uncertainty) to identify errors, then the method is simple to implement, but the resulting error policy lacks human-interpretability, compactness, and robustness
Solution Approach 1:
The patent divides the dataset into multiple slices based on feature value conditions rather than treating individual observations independently. Each slice represents a subset of observations sharing common characteristics, enabling the error policy to capture systematic error patterns while maintaining interpretability through the slice definitions
Solution Approach 2:
The patent introduces slices as an intermediary structure between individual observations and the error policy. These slices aggregate observations with similar error characteristics, providing a compact representation that enhances human-interpretability while maintaining the ability to identify systematic errors
2Quantity of substance
If the error policy includes many individual observations with low confidence levels, then the coverage of errors increases, but the policy becomes less compact and harder to interpret
Solution Approach 1:
The patent merges multiple individual observations into slices based on shared feature value conditions. This consolidation maintains the coverage of errors by including all relevant observations while improving compactness through the aggregated slice representation, making the error policy more interpretable
3Loss of time
If the error policy is based on individual observation metrics, then the method can be applied quickly, but it lacks robustness and cannot generalize well to diagnose underlying error patterns
Solution Approach 1:
The patent performs preliminary grouping of observations into slices based on feature value conditions before error analysis. This pre-organization enables faster processing by reducing the search space and improves robustness by capturing systematic error patterns that individual observation metrics would miss
Data Source
AI summary
A computer hardware system includes a slice generator and a policy generator and performs the following. The slice generator slices a first dataset including true values and predicted values of a class variable into a plurality of slices each defining a plurality of observations within the first dataset. A first one and another one of the plurality of slices are selected, and a union of observations is generated by adding observations within the selected another one to observations within the selected first one of the plurality of slices. The selecting another one of the plurality of slices and the generating the union is repeated until a number of observations within the union reaches a predetermined value. Using the policy generator and after the number of observations within the union reaches the predetermined value, an error policy is generated. The predicted values were generated by a machine learning engine.