Classification Model Degradation Detection Under Data Drift
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing classification models, particularly those using High-Resistance Learning (HDL), face accuracy degradation due to imbalanced and drifted data during operation, making it difficult to maintain accurate classification without ground truth labels.
Innovation Solution
A classification processing apparatus that includes a model degradation determination unit to assess the accuracy of additional data pieces with known labels, allowing for real-time detection of model degradation and a model recovery unit to restore the classification model, thereby maintaining accuracy by adjusting the class ratio in the augmented data set.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the classification model is updated using HDL technique during operation, then the model adapts to data drift, but the accuracy degrades due to imbalanced data distribution
Solution Approach 1:
The patent applies preliminary action by generating synthetic data with known class distribution before the actual classification task. This synthetic data is used to create a balanced training set that compensates for the imbalanced real data, allowing the model to maintain accuracy while adapting to drift. The synthetic data generation happens in advance to establish a reference distribution.
Solution Approach 2:
The patent changes the parameter of class distribution by generating synthetic data with controlled class ratios that differ from the imbalanced real data. By adjusting the class distribution parameter in the synthetic data generation process, the model learns to handle various distribution scenarios and maintains robustness against imbalance-induced accuracy degradation.
2Productivity
If ground truth labels are not available for classification target data, then real-time classification is enabled, but effective machine learning update becomes difficult
Solution Approach 1:
The patent uses copying by creating synthetic data that mimics the structure and characteristics of real data but with known ground truth labels. These synthetic copies serve as proxies for the unlabeled real data, enabling the model to learn from labeled examples that replicate the statistical properties of the target distribution without requiring actual labels on production data.
Solution Approach 2:
The synthetic data acts as an intermediary between the unlabeled real data and the learning process. It bridges the gap by providing labeled training examples that reflect the target distribution, allowing the model to update effectively without direct access to ground truth labels on real classification targets.
3Adaptability or versatility
If the class ratio in training data is imbalanced, then real-world data distribution is reflected, but classification accuracy on minority classes deteriorates
Solution Approach 1:
The patent deliberately creates asymmetric treatment of different data sources: real data maintains its imbalanced distribution to preserve authenticity, while synthetic data is generated with balanced or adjusted distributions to compensate. This asymmetric approach allows the model to learn from both the realistic imbalance and the corrective balanced signals.
Solution Approach 2:
The synthetic balanced data acts as a counterweight to the imbalanced real data. By introducing data with opposite distribution characteristics (balanced vs. imbalanced), the model learns to overcome the negative effects of imbalance on minority class accuracy while still adapting to the overall real-world distribution.
Data Source
AI summary
A non-transitory computer-readable recording medium having stored therein a classification processing program that causes a computer to execute a process includes: inputting, into a machine learning model that classifies input data pieces into one of a plurality of classes, a first input data piece that does not have a ground truth label and a second input data piece that has a known ground truth label; and determining whether the machine learning model is degraded based on an output result for the second input data piece out of output results output from the machine learning model, and the ground truth label.