Data Slicing for ML Performance Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models often fail to adequately address business requirements, leading to performance gaps due to biased training and testing data, and insufficient data coverage, which results in underperformance in specific areas.
Innovation Solution
Adapting combinatorial testing methodologies from classical software testing to identify and address coverage gaps by creating data slices that represent different business requirement combinations, and using multiple machine learning models with varying hyper-parameters to improve performance across these slices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single machine learning model is used to handle all data, then device complexity is reduced, but performance reliability deteriorates due to inability to address specific business requirement areas effectively
Solution Approach 1:
The patent divides the data space into multiple data slices based on business requirements and metadata attributes. Each slice represents a specific segment of the data space with unique characteristics. Multiple machine learning models are then trained on different slices, allowing each model to specialize in specific business requirement areas rather than attempting to handle all data uniformly.
Solution Approach 2:
Different machine learning models with varying hyperparameters are assigned to different data slices based on their performance characteristics. Each model is optimized for specific local conditions (data slices) rather than being a general-purpose model. This allows tailoring model properties to match the specific requirements of each data segment.
2Reliability
If comprehensive data coverage is ensured across all business requirements, then performance reliability improves, but data quantity and processing complexity increase
Solution Approach 1:
The patent segments the comprehensive data space into manageable data slices based on business requirements and metadata. This segmentation allows thorough coverage of all business requirements while organizing data into smaller, more manageable units that can be processed efficiently by specialized models.
Solution Approach 2:
The patent varies hyperparameters of machine learning models to optimize performance on different data slices. By changing model parameters rather than increasing data quantity, the system achieves comprehensive coverage and improved reliability without proportionally increasing processing complexity.
3Manufacturing precision
If multiple machine learning models with varying hyperparameters are used, then performance across different data slices improves, but device complexity increases
Solution Approach 1:
The patent applies local quality by training different machine learning models with specific hyperparameter configurations for different data slices. Each model is optimized for the local characteristics of its assigned data slice, improving precision for specific business requirements while maintaining overall system performance.
Solution Approach 2:
The patent systematically varies hyperparameters across multiple models to create a diverse set of models that perform well on different data slices. This parameter variation approach enables high precision across diverse data types without requiring fundamentally different model architectures.
4Measurement precision
If data is sliced based on business requirements and metadata, then measurement precision of model performance improves, but processing time and complexity increase
Solution Approach 1:
The patent segments data into slices based on business requirements and metadata attributes, enabling precise performance measurement for each slice. This segmentation allows identifying specific weak areas in model performance that would be obscured in aggregate metrics.
Solution Approach 2:
The patent performs preliminary actions by pre-processing data to extract metadata and pre-defining data slices based on business requirements before model evaluation. This preliminary organization of data into slices enables efficient and precise performance measurement without repeating complex processing during evaluation.
Data Source
AI summary
A method and a computer program product for identification and improvement of machine learning (ML) under-performance The method comprises slicing data of ML model based on a functional model representing requirements of a system utilizing the ML model. The functional model comprises a set of attributes and respective domain of values. Each data slice is associated with a different valuation of one or more attributes of the functional model. Each data instance of the ML model is mapped to one or more data slices, based on valuation of the attributes. A performance measurement of the ML model over is computed for each data slice, based on an application of the ML model on each data instance that is mapped to the data slice. A Determination whether ML model adheres to a target performance requirement may be performed based on the performance measurements of the data slices.