Inspector Model for ML Accuracy Degradation Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods, such as using T2 statistic, struggle to detect accuracy degradation in machine training models, especially with high-dimensional data like image data, as they lose important information during dimension reduction, making it difficult to properly classify and detect abnormal data.
Innovation Solution
A detection method involving an inspector model that calculates distances from a decision boundary to training and operation data, using knowledge distillation to create a model that can detect accuracy degradation by monitoring changes in the critical area around the decision boundary, even in high-dimensional data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional methods like T2 statistic are used to detect accuracy degradation, then the detection process is simple, but important information is lost during dimension reduction making it difficult to properly classify and detect abnormal data
Solution Approach 1:
The patent introduces an inspector model as an intermediary component that receives the trained machine learning model and operates independently to detect accuracy degradation. This mediator calculates distances from decision boundaries and identifies concept drift without requiring direct modification of the original model or loss of feature information, thus resolving the contradiction between simple detection and information preservation
Solution Approach 2:
Instead of reducing dimensions like conventional T2 statistic methods, the patent operates in the original high-dimensional feature space by calculating distances from decision boundaries. This approach maintains all feature information while detecting accuracy degradation, effectively moving from dimension reduction to dimension preservation strategy
2Ease of manufacture
If dimension reduction is performed to simplify detection, then the detection method becomes easier to implement, but accuracy degradation detection becomes less accurate especially with high-dimensional data like image data
Solution Approach 1:
The inspector model serves as a specialized intermediary that handles the complexity of high-dimensional distance calculations and decision boundary analysis. This mediator makes the detection method implementable by encapsulating complex operations in a reusable component while maintaining high detection accuracy in the original feature space
Solution Approach 2:
The patent replaces conventional statistical methods (T2 statistic) with a machine learning-based inspector model that uses distance calculations in high-dimensional space. This substitution maintains ease of implementation through automated model-based detection while significantly improving measurement precision for high-dimensional data
3Measurement precision
If the machine training model is retrained frequently to maintain accuracy, then detection accuracy is maintained, but system productivity and efficiency decrease
Solution Approach 1:
The inspector model performs preliminary detection of accuracy degradation by monitoring distance metrics from decision boundaries. This preliminary action identifies when retraining is actually needed, preventing unnecessary retraining operations and maintaining system efficiency while ensuring accuracy is maintained when truly degraded
Solution Approach 2:
The system implements continuous feedback through the inspector model that monitors model performance and triggers retraining only when accuracy degradation is detected. This feedback mechanism maintains measurement precision by responding to actual degradation while preserving productivity by avoiding unnecessary retraining cycles
Data Source
AI summary
A computing system trains an inspector model for training a decision boundary that divides a feature space of data into two application areas based on an output result of the operation model, the inspector model being configured to calculate a distance from the decision boundary to input data. The computing system calculates, by inputting training data to the inspector model, a first distance from the decision boundary to the training data.The computing system calculates, by inputting first data to the inspector model, a second distance from the decision boundary to the operation data. The computing system detects, when a difference between the first distance and the second distance is larger than or equal to a threshold, an accuracy degradation of the machine learning model caused according to the difference between the training data and the first data.