Dynamic Anomaly Threshold Adjustment for Concept Drift
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models used for anomaly detection face challenges in maintaining accuracy due to concept drift, where data distribution changes over time, leading to model decay and increased false positives, requiring frequent retraining that causes service outages and is computationally expensive.
Innovation Solution
A dynamic anomaly score thresholding approach that uses statistical moving parameters to normalize and adjust anomaly scores, allowing the system to adapt to changing data distributions without the need for continuous retraining, thereby extending the model's operational life and reducing false positives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the ML model is retrained frequently to handle concept drift, then the model accuracy is maintained, but the service outage time and computational cost increase
Solution Approach 1:
The patent applies dynamics by making the anomaly threshold dynamic rather than static. The threshold automatically adjusts based on the distribution of anomaly scores over time, allowing the system to adapt to concept drift without requiring model retraining. This resolves the contradiction by maintaining accuracy through adaptive thresholding while avoiding the service outages associated with frequent retraining.
Solution Approach 2:
The patent changes the parameter of the anomaly threshold based on the probabilistic distribution of anomaly scores. By monitoring how anomaly scores distribute over time and adjusting the threshold accordingly, the system maintains detection accuracy without the need for complete model retraining, thus reducing service outage time and computational resources.
2Reliability
If the ML model is retrained frequently to handle concept drift, then the model accuracy is maintained, but the computational resources are consumed
Solution Approach 1:
The system uses a dynamic threshold that adapts to changing data distributions without requiring full model retraining. This approach maintains model accuracy while significantly reducing computational resource consumption compared to frequent retraining operations.
Solution Approach 2:
Instead of investing heavy computational resources into frequent model retraining, the patent uses a lighter-weight approach of adjusting the anomaly threshold based on score distributions. This disposable-like approach continuously adapts the threshold without the expensive overhead of complete model retraining.
3Ease of operation
If a constant threshold is used for anomaly detection, then the system is simple to operate, but the detection accuracy decreases due to concept drift
Solution Approach 1:
The patent transforms the static constant threshold into a dynamic threshold that automatically adapts to concept drift. The threshold is adjusted based on the probabilistic distribution of anomaly scores, maintaining high detection accuracy while preserving operational simplicity as the system self-adjusts without manual intervention.
Solution Approach 2:
The system performs self-adjustment by monitoring the distribution of anomaly scores and automatically modifying the threshold accordingly. This self-service capability maintains detection accuracy under concept drift without requiring external intervention or complex manual retraining procedures.
Data Source
AI summary
Approaches herein relate to model decay of an anomaly detector due to concept drift. Herein are machine learning techniques for dynamically self-tuning an anomaly score threshold. In an embodiment in a production environment, a computer receives an item in a stream of items. A machine learning (ML) model hosted by the computer infers by calculation an anomaly score for the item. Whether the item is anomalous or not is decided based on the anomaly score and an adaptive anomaly threshold that dynamically fluctuates. A moving standard deviation of anomaly scores is adjusted based on a moving average of anomaly scores. The moving average of anomaly scores is then adjusted based on the anomaly score. The adaptive anomaly threshold is then adjusted based on the moving average of anomaly scores and the moving standard deviation of anomaly scores.