Memory-Less Autoencoder Anomaly Detection with Partial Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems in distributed environments require significant computing resources and data storage, which can lead to resource exhaustion and data vulnerability to malicious attacks.
Innovation Solution
Implementing a memory-less anomaly detection system using an autoencoder that re-trains incrementally and discards data, performing partial re-training to minimize resource usage and secure data from unauthorized access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional anomaly detection systems are implemented, then anomaly detection capability is improved, but computing resource expenditure increases significantly
Solution Approach 1:
The patent applies partial action by implementing partial re-training of the autoencoder model, where only certain components or parameters are updated rather than the entire model. This reduces computing resource expenditure while maintaining anomaly detection capability. The system performs selective updates based on incoming data patterns, avoiding full model re-training cycles.
Solution Approach 2:
The system dynamically adjusts parameters such as the anomaly threshold and re-training frequency based on operational conditions. By changing these parameters, the system optimizes the balance between detection accuracy and resource consumption, allowing it to adapt to varying workload and resource availability scenarios.
2Reliability
If traditional anomaly detection systems are implemented, then anomaly detection capability is improved, but data storage requirements increase
Solution Approach 1:
The patent implements a memory-less approach where training data is discarded after the autoencoder model learns from it. The system recovers useful patterns and knowledge by updating model parameters, then discards the raw training data. This allows continuous learning without accumulating large storage requirements, as the model retains knowledge in its weights rather than storing original datasets.
3Measurement precision
If data is stored for model training, then model accuracy is improved, but data vulnerability to malicious attacks increases
Solution Approach 1:
The system uses temporary, disposable training data that is processed and discarded after use. Instead of maintaining persistent storage of training datasets, the system loads data temporarily for model updates, processes it to improve accuracy, then discards it. This approach maintains model accuracy through continuous learning while minimizing data vulnerability, as stored data has minimal persistence and exposure time.
4Measurement precision
If full re-training is performed continuously, then model accuracy is improved, but computing resource expenditure increases
Solution Approach 1:
The system performs partial re-training rather than full re-training continuously. Only specific portions of the model are updated based on incoming data, and the frequency and extent of re-training are adjusted based on performance degradation thresholds. This maintains model accuracy by updating only when necessary and only to the extent required, significantly reducing computing resource expenditure compared to continuous full re-training.
Data Source
AI summary
Methods and systems for anomaly detection in a distributed system are disclosed. To manage anomaly detection, a system may include an anomaly detector and one or more data collectors. The anomaly detector may detect anomalies in data obtained from one or more of the data collectors using an inference model. The inference model may be an autoencoder trained to reconstruct data that is intended to match input data to an extent considered acceptable by the system. To accurately perform anomaly detection, the inference model may require re-training. Data collected from the one or more data collectors may be used to re-train the inference model as needed. Following anomaly detection and/or inference model re-training, the data may be discarded to remove the data from the anomaly detector.