Memory-less Anomaly Detection via Autoencoder Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems in distributed environments face challenges in reducing computing resource expenditure and ensuring data security, as they require significant computational resources and storage of large amounts of data, making them vulnerable to malicious attacks.
Innovation Solution
The system employs an inference model that performs unsupervised anomaly detection by mapping input values to a fixed output value, reducing the need for storage and computational resources. It also implements a partial re-training process for the inference model, where only specific portions are modified, and discards data after use to enhance security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional anomaly detection systems store large amounts of data for analysis, then detection accuracy is improved, but data security and vulnerability to malicious attacks worsen
Solution Approach 1:
The patent extracts only the essential features needed for anomaly detection by using autoencoders to learn compressed representations of normal data patterns. Instead of storing and analyzing entire datasets, the system extracts key characteristics during the encoding phase, achieving accurate anomaly detection with minimal data retention.
Solution Approach 2:
The system performs preliminary action by pre-training autoencoders on normal data patterns before actual anomaly detection occurs. The autoencoders learn to reconstruct normal patterns in advance, so during operation, any deviation from these pre-learned patterns can be quickly identified as anomalies without needing to store the original training data.
2Reliability
If comprehensive data is stored for anomaly detection analysis, then detection capability is improved, but computing resource expenditure increases
Solution Approach 1:
The patent segments the anomaly detection process into two distinct phases: an offline training phase where autoencoders are trained on normal data to learn patterns, and an online detection phase where only lightweight inference is performed. This segmentation allows comprehensive learning during training while maintaining minimal resource consumption during actual anomaly detection operations.
Solution Approach 2:
The system creates a compressed model copy (the trained autoencoder) that captures the essential patterns of normal data. Instead of repeatedly accessing and processing the original large datasets, the system uses this compact model copy for all anomaly detection decisions, dramatically reducing computing resource requirements.
3Measurement precision
If data is retained for model re-training, then model accuracy is improved, but data security and storage requirements worsen
Solution Approach 1:
The patent applies parameter changes by using differential learning to selectively update only specific parameters (weights and biases) of the autoencoder model when re-training is needed. Instead of storing and re-processing entire datasets, the system computes gradient updates for model parameters using incoming data streams, achieving model improvement with minimal data retention.
Solution Approach 2:
The system performs partial action by implementing continuous incremental re-training where only portions of the model are updated at a time using streaming data. This allows the model to adapt to changing patterns without requiring storage of complete historical datasets, as updates are computed and applied incrementally.
Data Source
AI summary
Methods and systems for anomaly detection in a distributed environment are disclosed. To manage anomaly detection, a system may include an anomaly detector and one or more data collectors. The anomaly detector may detect anomalies in data and classify the anomalies based on magnitudes of anomalies using an inference model. Different magnitudes of anomalies may be keyed to different action sets in response to the presence of anomalies in data. To perform anomaly detection, the inference model may require re-training. Data collected from the one or more data collectors may be used to re-train the inference model as needed. Following anomaly detection and/or inference model re-training, the data may be discarded to remove the data from the anomaly detector.