VAE Anomaly Detection Training for Corrupted Log Sample Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in processing and consuming voluminous logs and other data sets due to their intricate nature, making it difficult to identify irregular patterns that signify noteworthy occurrences such as system malfunctions or security infringements, necessitating automated anomaly detection systems.
Innovation Solution
An anomaly detection model is trained using a variational autoencoder to identify and remove corrupted samples from the training dataset, ensuring the model is trained exclusively on normal samples by computing sample weights during multiple epochs and incorporating human feedback to enhance accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional anomaly detection models are trained on voluminous log data containing corrupted samples, then the model can process diverse data sources, but the detection accuracy deteriorates due to noise from corrupted samples
Solution Approach 1:
The patent applies preliminary action by performing sample weight computation and corrupted sample identification before the final training phase. The system computes sample weights during an initial training phase, identifies corrupted samples based on these weights, removes them from the training dataset, and then performs final training on the cleaned data. This preliminary filtering action ensures that the main training process uses only high-quality samples, resolving the contradiction between using voluminous data and maintaining detection accuracy.
2Reliability
If manual review processes are used to identify corrupted samples, then data quality can be ensured, but the processing time and operational complexity increase significantly
Solution Approach 1:
The patent applies self-service by enabling the system to automatically identify and remove corrupted samples without requiring manual review. The anomaly detection model computes sample weights autonomously during training, identifies corrupted samples based on these weights, and performs self-correction by removing identified corrupted samples from the training dataset. This automated self-service mechanism ensures data quality while eliminating the operational complexity and time requirements of manual review processes.
3Productivity
If all training samples are used without filtering, then the training process is efficient, but corrupted samples reduce the reliability of the anomaly detection model
Solution Approach 1:
The patent applies preliminary action by implementing a two-phase training approach: an initial phase where sample weights are computed and corrupted samples are identified, followed by a final phase where training is performed on filtered data. This preliminary filtering action removes corrupted samples before the main training process, ensuring that training efficiency is maintained while model reliability is significantly improved through the use of cleaned training data.
Data Source
AI summary
Apparatus and method of training Machine Learning (ML) models. In an embodiment, the apparatus performs initial training of an anomaly detection model based on training samples of a training dataset over multiple epochs, where the anomaly detection model comprises a variational autoencoder (VAE). For each training sample during an epoch, the initial training comprises inputting an original data sequence of the training sample into the VAE encoder to output a multivariant distribution in latent space, sampling the multivariant distribution to generate multiple latent vectors, inputting the latent vectors into the VAE decoder to output reconstructed data sequences, and computing an estimated sample weight for the training sample. The apparatus identifies, after multiple epochs, corrupted samples from the training dataset based on the estimated sample weights, removes the corrupted samples to generate a filtered training dataset, and performs final training of the anomaly detection model based on the filtered training dataset.