Multi-tier anomaly detection using univariate and multivariate models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection systems often require significant resources and are not efficient in providing accurate and timely predictions, especially when dealing with complex systems, as they rely on single-tier models that struggle to handle multiple input features effectively.
Innovation Solution
A multi-tier anomaly detection system is implemented, utilizing a combination of univariate and multi-variate machine learning models, where univariate models detect initial anomalies with reduced resource requirements and multi-variate models refine predictions using multiple input features, with continuous training and feedback loops to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single-tier anomaly detection models are used, then device complexity is reduced, but anomaly detection accuracy deteriorates
Solution Approach 1:
The anomaly detection system is segmented into multiple tiers: a first tier using univariate machine learning models for initial anomaly detection, and a second tier using multi-variate machine learning models for refined analysis. This segmentation allows the system to handle complexity by dividing the detection process into manageable stages, each with specific models optimized for particular tasks, thereby improving overall accuracy without overwhelming complexity.
Solution Approach 2:
The system transitions from univariate analysis (single feature) in the first tier to multi-variate analysis (multiple features) in the second tier. This dimensional expansion enables the system to capture complex relationships between multiple input features, significantly improving anomaly detection accuracy by considering interactions and correlations across different dimensions of data.
2Measurement precision
If multi-variate machine learning models are used for all detections, then anomaly detection accuracy is improved, but use of energy and computational resources increase
Solution Approach 1:
Instead of applying the more resource-intensive multi-variate machine learning models to all anomaly detection tasks, the system uses them partially - specifically for refining anomalies detected by the first tier. This partial application ensures that computational resources are allocated only where needed most, improving accuracy for complex cases while avoiding unnecessary energy consumption for routine detections that can be handled by simpler univariate models.
Solution Approach 2:
The univariate machine learning models perform preliminary anomaly detection first, filtering and pre-processing the data before it reaches the multi-variate models. This preliminary action reduces the volume of data requiring intensive analysis, thereby lowering overall computational resource consumption while maintaining high accuracy through the subsequent refined analysis by multi-variate models.
3Quantity of substance
If univariate models are used alone, then resource requirements are reduced, but detection precision for complex patterns deteriorates
Solution Approach 1:
The detection system is segmented into two functional layers: univariate models for initial screening and multi-variate models for precision analysis. This segmentation allows the system to maintain low resource requirements for the bulk of detections while reserving computational resources for the precision analysis of potential anomalies, achieving both efficiency and accuracy.
Solution Approach 2:
The system merges the strengths of both univariate and multi-variate models by combining them in a hierarchical architecture. The univariate models provide efficient initial detection, while the multi-variate models provide refined precision analysis. This merging of complementary approaches enables the system to achieve detection precision comparable to using only multi-variate models, while resource consumption remains lower due to the efficient filtering by univariate models.
Data Source
AI summary
For each corresponding configuration item type of a plurality of different configuration item types, a corresponding multi-variate machine learning model of a plurality of multi-variate machine learning models is trained to perform anomaly detection for a corresponding configuration item type of the plurality of different configuration item types. In response to detecting, via a univariate machine learning model, an anomaly associated with a specific configuration item type of the plurality of different configuration item types, an execution of a particular multi-variate machine learning model of the plurality of multi-variate machine learning models is initiated for the specific configuration item type. An output of the execution of the particular multi-variate machine learning model is evaluated to determine an anomaly detection result.