Automated Data Slicing for Neural Network Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for validating Artificial Neural Networks (ANNs) fail to identify poor performance on specific data slices within a dataset, as overall performance metrics mask below-threshold performance in some slices with above-threshold performance in others, and require manual definition of data slices.
Innovation Solution
Automatically determining data slices based on attribute vectors from ANN nodes, computing performance metrics for each slice, and taking actions when thresholds are not met, such as retraining the model or switching to a different predictive model, to improve performance and handle poor-performing slices effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If overall performance metrics are used to validate ANN, then validation is simplified and faster, but poor performance on specific data slices is masked and undetected
Solution Approach 1:
The patent divides the dataset into multiple data slices based on attribute vectors derived from ANN node outputs. Each slice represents a distinct subset of data instances with similar characteristics. By evaluating performance metrics separately for each slice rather than on the entire dataset, the method enables detection of poor performance in specific segments that would otherwise be masked by aggregate metrics, thus resolving the contradiction between validation speed and detection accuracy.
2Measurement precision
If manual definition of data slices is performed, then performance validation becomes more thorough, but the process requires human intervention and is more complex
Solution Approach 1:
The patent employs the ANN itself to automatically generate attribute vectors from its own node outputs, which are then used to define data slices. This self-service mechanism eliminates the need for manual slice definition by domain experts. The system autonomously partitions the dataset based on its internal feature representations, achieving thorough performance validation while reducing process complexity and human intervention requirements.
3Productivity
If data slices are automatically determined using ANN nodes, then manual intervention is eliminated and processing is faster, but the ability to define meaningful slices may be reduced
Solution Approach 1:
The patent leverages the ANN's learned parameter representations (node outputs and attribute vectors) to automatically define data slices. These parameters capture meaningful patterns that the ANN has discovered during training, enabling automatic slice determination that both accelerates processing and maintains semantic relevance. The approach adapts to different data types and problem domains by utilizing the ANN's inherent feature extraction capabilities, thus achieving both speed and adaptability.
Data Source
AI summary
A method, apparatus and a computer program product for automated data slicing based on an Artificial Neural Network (ANN). The method comprising: obtaining an ANN, wherein the ANN is configured to provide a prediction for a data instance, wherein the ANN comprises a set of nodes having interconnections therebetween; determining an attribute vector based on a subset of the nodes of the ANN; determining, based on the attribute vector, a plurality of data slices; obtaining a testing dataset comprising testing data instances; computing, for each data slice, a performance measurement of the ANN over the data slice, wherein said computing is based on an application of the ANN on each testing data instance that is mapped to the data slice; and performing an action based on at least a portion of the performance measurements of the data slices.