Dual VAE Anomaly Detection for IoT Communication Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems for IoT devices, such as those using variational autoencoders (VAE), face issues with erroneous detection of normal communication as abnormal, particularly when excess detection occurs, and struggle with precise learning when the amount of excess detection data is significantly lower than the initial learning data set, making it difficult to improve detection precision.
Innovation Solution
An evaluation apparatus that generates and utilizes two VAE models: a learning data VAE model for normal data and an excess detection VAE model for abnormal data, with the latter being learned from a small amount of excess detection data, and combines these models to improve detection precision by concatenating their probability densities for anomaly scoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a VAE model is trained only on normal communication data to detect anomalies, then the system can identify unknown threats effectively, but erroneous detection occurs where normal communication is mistakenly identified as abnormal
Solution Approach 1:
The patent segments the detection system into two specialized VAE models: one trained on normal communication data and another trained on excess detection data. This segmentation allows each model to focus on specific patterns, reducing erroneous detection while maintaining the ability to identify unknown threats.
Solution Approach 2:
The patent implements a feedback mechanism where excess detection data (normal communication erroneously detected as abnormal) is collected and used to train a second VAE model. This feedback loop continuously improves detection precision by learning from past erroneous detections.
2Measurement precision
If excess detection data is fed back by mixing it with initial learning data and retraining the VAE model, then detection precision can be improved, but the initial learning data set must be saved indefinitely increasing storage requirements
Solution Approach 1:
The patent segments the training process into two independent phases: initial training on normal data, and subsequent training on excess detection data. This eliminates the need to store the entire initial learning data set indefinitely, as only the trained model parameters need to be preserved.
Solution Approach 2:
The patent performs preliminary training on normal communication data to establish a baseline detection model. This preliminary action enables the system to identify excess detection data, which is then used for further refinement without requiring the original large data set to be retained.
3Measurement precision
If excess detection data is fed back by mixing with initial learning data for retraining, then the model can be updated, but when the amount of excess detection data is substantially lower than initial learning data, the excess detection data cannot be precisely learned
Solution Approach 1:
The patent segments the training approach so that the second VAE model is trained exclusively on excess detection data rather than mixing it with initial learning data. This dedicated training approach allows the model to precisely learn from even small amounts of excess detection data without being overwhelmed by the volume of normal data.
Solution Approach 2:
The patent applies partial training action by training the second VAE model only on the specific excess detection data subset that needs refinement, rather than retraining on the complete data set. This focused approach ensures precise learning from limited excess detection examples.
4Quantity of substance
If only a small amount of excess detection data is available for feedback, then storage requirements are reduced, but it becomes difficult to collect sufficient data for effective learning and improve detection precision
Solution Approach 1:
The patent changes the training parameter approach by using separate VAE models with different training objectives. The second model is specifically optimized to learn from small amounts of excess detection data, transforming the limitation of small data volume into an effective training regime that maintains high detection precision.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An evaluation apparatus (1) includes an acceptance unit (120) configured to accept an input of communication data of an evaluation target, and an evaluation unit (123) configured to estimate a probability density of the communication data of the evaluation target by using a learning data VAE model (111) in which a feature of a probability density of normal initial learning data is learned, and an excess detection VAE model (112) in which a feature of a probability density of normal excess detection data detected as abnormal in a course of evaluation processing is learned, and evaluate presence or absence of an anomaly of the communication data of the evaluation target based on the estimated probability density.