Autoencoder Anomaly Detection with Abnormal Training Data Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing abnormality detection methods using autoencoders face precision issues when learning data includes abnormal data, leading to deteriorated detection performance.
Innovation Solution
An abnormality detection device comprising a detector, a remover, and a learner, where the detector identifies abnormal data using a first autoencoder, the remover removes associated abnormal data from the learning target data using a second autoencoder, and the learner refines the model with the cleaned data to improve detection precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If learning process is performed using learning data including abnormal data, then unsupervised learning can be applied without labeling work, but precision of abnormality detection deteriorates
Solution Approach 1:
The patent applies preliminary action by performing abnormality detection on learning data before the actual learning process. The detector identifies abnormal data points in the learning dataset, and the remover eliminates these abnormal points before training the autoencoder. This preliminary cleaning of data ensures that the learning model is trained only on normal data, thereby maintaining high abnormality detection precision while still using unsupervised learning without manual labeling.
2Measurement precision
If abnormal data is removed from learning target data, then detection precision is improved, but additional processing steps and device complexity increase
Solution Approach 1:
The patent merges multiple functions into an integrated processing pipeline. The detector, remover, and learner are combined into a unified system where abnormality detection and data cleaning are seamlessly integrated with the learning process. This merging approach improves detection precision by ensuring clean training data while avoiding the need for separate, complex preprocessing steps, thereby managing device complexity through functional integration rather than additive complexity.
3Measurement precision
If learning is performed with purified data, then model accuracy is improved, but processing time increases due to multiple processing steps
Solution Approach 1:
The patent ensures continuity of useful action by making the abnormality detection and data cleaning processes continuous and automated rather than discrete and manual. The detector continuously identifies abnormal data points, the remover continuously eliminates them, and the learner continuously trains on the cleaned data. This continuous pipeline minimizes idle time and ensures that the model accuracy improvement is achieved through an efficient, uninterrupted processing flow rather than time-consuming batch operations.
Data Source
AI summary
An abnormality detection device according to an embodiment includes a detector, a remover, and a learner. The detector detects first abnormal data in detection target data which is an abnormality detection target by inputting the detection target data to a first autoencoder which performed learning based on first learning target data which is a learning target. The remover removes data associated with the first abnormal data from the first learning target data to generate second learning target data by inputting the first learning target data to a second autoencoder which performed learning based on the first abnormal data detected by the detector. The learner causes the first autoencoder to perform learning based on the second learning target data generated by the remover.