Anomaly Detection Model for Imbalanced Data Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In the field of computer technologies, the imbalance between normal and abnormal data samples, such as in recognizing fraudulent behaviors or risky transactions, leads to low accuracy in classifier training due to the low occurrence probability of abnormal events, and the need for rapid risk recognition in high-concurrent online systems poses additional challenges.
Innovation Solution
A method involving an anomaly detection model and a classification model, where the anomaly detection model identifies abnormal data, and the classification model is trained with a balanced abnormal sample data set to distinguish between normal and abnormal data, improving recognition accuracy and reducing scoring events.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If black and white samples are determined according to historical data for classifier training, then the classifier can be obtained through training, but the accuracy of recognition may be low due to serious imbalance between black and white samples
Solution Approach 1:
The patent segments the recognition process into two stages: first using an anomaly detection model to identify abnormal data, then using a classification model to further analyze the abnormal data. This segmentation allows the classification model to focus only on the minority abnormal samples, effectively addressing the class imbalance problem without needing to process the entire dataset.
Solution Approach 2:
The patent extracts the abnormal data from the overall dataset using the anomaly detection model, separating it from the normal data. This extraction creates a dedicated subset of abnormal samples for training the classification model, ensuring that the model receives sufficient abnormal samples for effective training despite their scarcity in the original dataset.
2Measurement precision
If all data samples are input into the classification model for recognition, then comprehensive analysis can be performed, but the system efficiency decreases due to unnecessary processing of normal data
Solution Approach 1:
The patent divides the data processing workflow into two distinct segments: anomaly detection followed by classification. The anomaly detection model quickly filters out normal data, and only the identified abnormal data is passed to the classification model. This segmentation eliminates unnecessary processing of normal data while maintaining comprehensive analysis of abnormal cases.
Solution Approach 2:
The patent performs preliminary anomaly detection before classification, pre-filtering the data to identify which samples require further analysis. This preliminary action ensures that the classification model only processes abnormal data that needs detailed examination, significantly improving system efficiency without compromising recognition accuracy.
3Reliability
If the classification model is trained with balanced abnormal sample data set, then the recognition accuracy improves, but additional processing steps are required to balance the data
Solution Approach 1:
The patent extracts abnormal samples from the overall dataset using anomaly detection, creating a dedicated training subset. This extraction process naturally provides a balanced dataset for classification model training, as the abnormal samples are specifically isolated for this purpose. The complexity is reduced compared to traditional methods because the anomaly detection model automatically identifies and separates the relevant samples without requiring manual balancing techniques.
Data Source
AI summary
Data type recognition and model training methods and apparatuses, and computer devices are provided. The model training method includes acquiring a first sample data set, and using the first sample data set to train an anomaly detection model; and detecting an abnormal sample data set from a second sample data set by means of the anomaly detection model, and using the abnormal sample data set to train a classification model. By using this method, an amount of scoring events of the classification model can be reduced, and relatively balanced sample data sets can also be provided for training, to obtain the classification model with a higher accuracy.