Neural Network Training with Impure Data via Two-Stage Classifier Ensemble
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models for anomaly detection, such as malware detection and video surveillance, face challenges due to noise in large datasets, requiring significant computing resources and lacking robustness.
Innovation Solution
A two-stage training method using an ensemble of machine learning one-class classifiers, where a coarse classifier and a noise classifier are applied to create a refined training set, allowing for the detection of signals amidst noise, and a final classifier is trained to enhance robustness and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If huge datasets are used for training unsupervised learning models, then anomaly detection capability is improved, but model robustness deteriorates due to significant noise in the datasets
Solution Approach 1:
The training process is segmented into two distinct stages: a first stage training a coarse classifier on the complete huge dataset to capture general anomaly patterns, and a second stage training a final classifier on a refined subset to achieve robustness. This segmentation allows the model to benefit from large data while maintaining reliability through focused refinement.
Solution Approach 2:
A refined training set is extracted from the huge original dataset by applying the coarse classifier to identify and remove noise components. This extraction process isolates the most relevant data points for final model training, enabling the system to achieve high robustness without sacrificing anomaly detection capability.
2Measurement precision
If huge datasets are used for training, then anomaly detection capability is improved, but computing resource requirements increase significantly
Solution Approach 1:
The coarse classifier performs preliminary action by pre-processing the huge dataset to create a refined training set before the final model training. This preliminary filtering reduces the data volume that requires intensive computing resources, significantly lowering energy and computational requirements while preserving anomaly detection capability.
Solution Approach 2:
The refined training set is extracted from the huge dataset, containing only the most relevant data points. This extraction dramatically reduces the computational burden for final model training while maintaining the anomaly detection performance that would otherwise require processing the entire huge dataset.
3Quantity of substance
If noise is present in training data, then more data points are available for training, but the quality of learned patterns deteriorates
Solution Approach 1:
The coarse classifier is trained on the complete dataset including noise, and its predictions are then used to identify and extract clean patterns. The noise-containing data is converted into a benefit by using it to train the coarse classifier, which subsequently enables the extraction of high-quality patterns for the refined training set.
Solution Approach 2:
The coarse classifier acts as an intermediary between the noisy huge dataset and the final model training. It processes the noisy data and produces a refined training set, mediating the transition from quantity to quality by filtering out noise while preserving relevant patterns.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods and systems for training a neural network. In a first stage of training, a coarse machine learning one-class classifier is trained using a first training set including a signal and noise and a noise machine learning one-class classifier is trained using a second training set excluding the signal. An assembly of models including the noise machine learning one-class classifier and the coarse machine learning one-class classifier is applied to the first training set to create a third training set representing the signal for a second stage of training. A final machine learning one-class classifier is trained in the second stage of training using the third training set representing the signal.