VAE Anomaly Detection Training with Corrupted Sample Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in processing and consuming voluminous logs and other data sets due to their intricate nature and variability, making it difficult to identify irregular patterns that signify noteworthy occurrences such as system malfunctions or security infringements, necessitating automated anomaly detection systems.
Innovation Solution
An anomaly detection model is trained using a variational autoencoder to identify and remove corrupted samples from the training dataset based on estimated sample weights, ensuring the model is trained exclusively on normal samples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional anomaly detection models are trained on datasets containing corrupted samples, then training efficiency is improved, but model accuracy deteriorates due to learning from imperfect data
Solution Approach 1:
The system performs preliminary identification and removal of corrupted samples from the training dataset before training the anomaly detection model. By pre-filtering the training data to eliminate corrupted entries, the model trains exclusively on clean data, thereby maintaining high training efficiency while achieving superior anomaly detection accuracy without being exposed to harmful corrupted patterns during the learning process.
2Measurement precision
If corrupted samples are removed from the training dataset, then model accuracy is improved, but data processing complexity increases
Solution Approach 1:
The system employs a self-service mechanism where the training data automatically identifies and flags corrupted samples through statistical analysis and anomaly detection algorithms during the preprocessing phase. This automated self-filtering process eliminates the need for manual data cleaning, reducing processing complexity while ensuring high accuracy by removing corrupted entries before model training begins.
3Measurement precision
If manual cleaning of training data is performed to remove corrupted samples, then model accuracy is improved, but time consumption increases
Solution Approach 1:
The system replaces manual mechanical data cleaning processes with automated computational algorithms that can rapidly identify and remove corrupted samples through statistical analysis, pattern recognition, and machine learning-based detection. This substitution dramatically reduces the time required for data preparation while maintaining high anomaly detection accuracy, as the automated systems can process large datasets much faster than manual methods.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Apparatus and method of training Machine Learning (ML) models. In an embodiment, the apparatus performs initial training of an anomaly detection model based on training samples of a training dataset over multiple epochs, where the anomaly detection model comprises a variational autoencoder (VAE). For each training sample during an epoch, the initial training comprises inputting an original data sequence of the training sample into the VAE encoder to output a multivariant distribution in latent space, sampling the multivariant distribution to generate multiple latent vectors, inputting the latent vectors into the VAE decoder to output reconstructed data sequences, and computing an estimated sample weight for the training sample. The apparatus identifies, after multiple epochs, corrupted samples from the training dataset based on the estimated sample weights, removes the corrupted samples to generate a filtered training dataset, and performs final training of the anomaly detection model based on the filtered training dataset.