Autoencoder Transfer Learning for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional unsupervised anomaly detection approaches using deep learning require a large amount of normal data to function effectively, which is not feasible immediately after a new ICT system is constructed or when the normal state of the ICT system changes, leading to delayed anomaly detection.
Innovation Solution
A learning apparatus that utilizes a combination of autoencoders and a discriminator in a generative adversarial network-based transfer learning approach, allowing for anomaly detection in a target system with a small amount of normal data by leveraging information from a source system with a large amount of normal data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional unsupervised anomaly detection approaches using deep learning are used, then anomaly detection capability is improved, but a large amount of normal data is required which cannot be obtained immediately after system construction or when system state changes
Solution Approach 1:
The patent applies preliminary action by pre-training the autoencoder model using normal data from a source domain (similar system) before deploying it to the target domain. This allows the model to learn normal patterns in advance from available source data, so that when the target system is newly constructed or undergoes state changes and lacks sufficient normal data, the pre-trained model can still perform effective anomaly detection. The source domain data is collected and used for training beforehand, eliminating the waiting period required for accumulating target domain normal data.
Solution Approach 2:
The patent uses an intermediary approach by introducing a source domain as a mediator between the limited target domain data and the anomaly detection task. The source domain, which has abundant normal data and shares similar characteristics with the target domain, serves as an intermediate resource to train the model. This intermediary source domain enables the model to acquire detection capabilities without requiring large amounts of target domain normal data, effectively bridging the data gap.
2Measurement precision
If unsupervised anomaly detection is delayed until sufficient normal data is collected, then detection accuracy is improved, but the time required for anomaly detection increases
Solution Approach 1:
The patent eliminates the time loss by performing the data collection and model training actions in advance using source domain data. The model is pre-trained before it is needed for target domain anomaly detection, so no time needs to be spent waiting for target domain normal data to accumulate. This preliminary training action allows immediate deployment of the anomaly detection system while maintaining accuracy through transfer learning from the source domain.
3Reliability
If normal data collection is extended to ensure sufficient training data, then model performance is improved, but the system cannot adapt to new services or state changes quickly
Solution Approach 1:
The patent enables quick adaptation to new services or state changes by performing preliminary training using source domain data that represents the new system state. When a new service is introduced or the system state changes, source domain data reflecting the new normal state can be collected and used for pre-training, allowing the model to adapt quickly without requiring extensive target domain data collection. This preliminary action on source data accelerates the adaptation process.
Solution Approach 2:
The patent improves adaptability by using the source domain as an intermediary that can be quickly updated to reflect new system states. When new services are introduced, the source domain data can be refreshed to represent the new normal behavior, and the model can be re-trained using this updated source data. This intermediary approach allows the system to adapt to changes quickly without being constrained by the need to collect large amounts of target domain data for each new service or state change.
Data Source
AI summary
A learning apparatus according to one embodiment includes an input unit configured to input a normal data collection for a first system that is a target domain and to input a normal data collection for a second system that is a source domain. The learning apparatus includes a learning unit configured to train a model that includes a first autoencoder configured to input normal data for the target domain, based on the normal data collection for the first system and the normal data collection for the second system. The model includes a second autoencoder configured to input normal data for the source domain, and includes a discriminator configured to output a probability that output data is data representing a feature for any one of the target domain and the source domain, while using, as an input, output data, output data of a first encoder included in the first autoencoder or a second encoder included in the second autoencoder.