Transient Data Drift Detection Using Dual Inference Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems face challenges in efficiently managing computing resource expenditure and data security while performing anomaly detection and detecting data drift, as re-training inference models to adapt to data drift can be computationally costly and lead to downtime, and data drift may be transient or repetitive, increasing resource expenditure.
Innovation Solution
The system employs both continuous and quantized inference models to detect anomalies and data drift, with the quantized model being less sensitive to small inconsistencies, allowing for reduced re-training and adaptation only when necessary, and monitoring data for transient drift to minimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If inference models are re-trained to adapt to data drift, then anomaly detection accuracy is improved, but computing resource expenditure increases
Solution Approach 1:
The system performs partial re-training only when necessary by comparing drift detection results from both continuous and quantized models. When the quantized model detects drift that the continuous model misses, re-training is triggered. This partial action approach avoids continuous full re-training while maintaining detection accuracy, thereby reducing computing resource expenditure.
Solution Approach 2:
The system changes the parameter of model sensitivity by using two inference models with different sensitivity levels - a continuous model and a quantized model. The quantized model with reduced precision serves as a filter to detect significant drift events, allowing the system to adjust re-training frequency based on actual drift conditions rather than continuously, thus optimizing computing resource usage.
2Reliability
If inference models are re-trained to adapt to data drift, then anomaly detection accuracy is improved, but system downtime increases
Solution Approach 1:
The system performs partial re-training only when drift is confirmed by both continuous and quantized models, rather than continuous re-training. This reduces the frequency of re-training operations and associated downtime while maintaining detection accuracy for actual drift events.
Solution Approach 2:
The system implements periodic drift detection using both continuous and quantized models, triggering re-training only when drift is detected. This periodic action based on actual conditions rather than fixed schedules reduces unnecessary re-training operations and associated system downtime while maintaining anomaly detection accuracy.
3Reliability
If continuous anomaly detection is performed, then detection coverage is improved, but computing resource expenditure increases
Solution Approach 1:
The system segments the anomaly detection function into two parallel inference models: a continuous model for comprehensive detection coverage and a quantized model for drift filtering. By dividing the detection task and using the quantized model to filter false positives, the system maintains detection coverage while reducing the frequency of expensive re-training operations, thus lowering computing resource expenditure.
Solution Approach 2:
The quantized inference model serves as an intermediary between raw data and the continuous inference model. It filters and pre-processes drift detection signals, allowing the continuous model to operate with reduced re-training frequency while maintaining detection coverage. This intermediary reduces the computational burden on the primary detection system.
Data Source
AI summary
Methods and systems for identifying transient data drift while performing anomaly detection in a distributed environment are disclosed. To identify transient data drift, a system may include an anomaly detector and one or more data collectors. The anomaly detector may identify a first data drift using a first pair of inference models. The anomaly detector may obtain additional data from the one or more data collectors and determine whether a second data drift has occurred using a second pair of inference models. If a second data drift has occurred, the anomaly detector may utilize the first pair of inference models to determine whether the first data drift was a transient data drift. If the first data drift was a transient data drift, the second pair of inference models may be replaced with the first pair of inference models.