Inspector Model for ML Accuracy Degradation Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods, such as using T2 statistic, struggle to detect accuracy degradation in machine training models, especially with high-dimensional data like image data, as they lose important information during dimension reduction, making it difficult to properly classify and detect abnormal data.
Innovation Solution
A detection method that uses knowledge distillation to train an inspector model, which calculates the distance from the decision boundary to operation data, allowing for the detection of changes in output results and accuracy degradation by comparing proportions of data within critical areas.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional methods like T2 statistic are used to detect accuracy degradation, then the detection process is simple, but important information is lost during dimension reduction making it difficult to properly classify and detect abnormal data
Solution Approach 1:
The patent introduces an inspector model as an intermediary component that bridges the gap between simple detection methods and accurate classification. The inspector model receives operation data and compares its output with the main machine training model's output, enabling accurate detection of accuracy degradation without requiring complex dimension reduction that would lose important information.
2Measurement precision
If the inspector model is trained using knowledge distillation, then classification accuracy is maintained across high-dimensional data, but the training process becomes more complex
Solution Approach 1:
The inspector model is created as a copy of the main machine training model through knowledge distillation. This copying process transfers the classification knowledge and decision boundaries from the trained model to the inspector model, enabling it to maintain the same classification accuracy while serving the additional function of detecting accuracy degradation in operation data.
3Productivity
If dimension reduction is applied to simplify data processing, then computational load is reduced, but important information is lost making it difficult to detect abnormal data
Solution Approach 1:
Instead of applying dimension reduction to the entire data set, the patent extracts only the critical information needed for accuracy degradation detection. The inspector model is trained to identify and extract features that are most relevant for detecting concept drift and accuracy changes, leaving out unnecessary dimensions that would otherwise require reduction.
Data Source
AI summary
A computing system trains a machine learning by using a plurality of pieces of training data including first data associated with a first class and second data associated with a second class. The computing system trains an inspector model for training a decision boundary between an area of the first class and an area of the second class based on knowledge distillation of the operation model, the inspector model being constructed for calculating a distance from the decision boundary to operation data. The computing system detects, based on a result obtained by inputting the plurality of pieces of training data and a plurality of pieces of data to the inspector model, a change in an output result of the operation model caused by a difference between training data and data.