Memory-Less Anomaly Detection With Probability-Based Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems require significant computing resources and data storage, which increases vulnerability to malicious attacks and resource consumption.
Innovation Solution
Implement a memory-less anomaly detection system that discards data after use and performs partial re-training of the inference model, using anomaly and calibration thresholds to identify and update models efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is stored for anomaly detection and model re-training, then detection accuracy is improved, but data security deteriorates due to increased vulnerability to malicious attacks
Solution Approach 1:
The system implements memory-less anomaly detection where data is discarded after use rather than stored permanently. Training data is used to re-train the inference model and then discarded, creating a disposable data handling approach that eliminates persistent data storage vulnerabilities while maintaining detection capabilities through periodic model re-training.
Solution Approach 2:
The system extracts only the essential anomaly detection functionality from the data, using inference models that can operate without storing the actual data. The inference model captures the necessary patterns for anomaly detection while the raw data is discarded, separating the detection capability from the data itself.
2Measurement precision
If data is stored for inference model re-training, then model accuracy is improved, but computing resource expenditure increases
Solution Approach 1:
The system performs partial re-training of the inference model using only the necessary training data rather than complete re-training datasets. This partial action approach re-trains the model with sufficient data to maintain accuracy while avoiding the excessive computing resources required for full dataset re-processing.
Solution Approach 2:
The system performs inference model re-training proactively before data is discarded, ensuring the model is updated with the latest patterns while the data is still available. This preliminary action allows accurate model updates without requiring long-term data storage, as re-training occurs immediately when needed.
3Reliability
If anomaly detection is performed continuously, then security monitoring is improved, but computing resource consumption increases
Solution Approach 1:
The inference model performs self-service anomaly detection by processing incoming data independently without requiring continuous human intervention or complex centralized processing. The model autonomously evaluates data against learned patterns and discards processed data, enabling continuous security monitoring with reduced computing overhead compared to centralized analysis systems.
Data Source
AI summary
Methods and systems for anomaly detection in a distributed environment are disclosed. To manage anomaly detection, a system may include an anomaly detector and one or more data collectors. The anomaly detector may detect anomalies in data obtained from one or more of the data collectors using an inference model and an anomaly threshold. The anomaly threshold may be determined by fitting a normal distribution to output of the inference model when the inference model is exercised across an input range of the inference model. The anomaly threshold may correspond to a portion of the normal distribution. To perform anomaly detection, the inference model may require periodic re-training. Data collected from the one or more data collectors may be used to re-train the inference model as needed. Following anomaly detection and/or inference model re-training, the data may be discarded to remove the data from the anomaly detector.