Microservices Anomaly Detection with Ensemble Model Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing microservice anomaly detection systems suffer from high false positives and false negatives, overwhelming system administrators with alerts and failing to efficiently classify and address anomalies, leading to potential network issues and reduced system performance.
Innovation Solution
A system that uses an ensemble of machine learning models, such as Isolation Forest and DBSCAN, to detect anomalies in microservices, combined with a set of rules to suppress false positives, and triggers automated countermeasures like self-healing or alerts based on anomaly severity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional anomaly detection techniques are used, then anomaly detection capability is provided, but false positives increase and system administrators are overwhelmed with alerts
Solution Approach 1:
The anomaly detection system is segmented into multiple independent anomaly detection models, each specialized in detecting specific types of anomalies. This segmentation allows each model to focus on particular patterns, reducing false positives while maintaining comprehensive detection coverage across different anomaly types.
Solution Approach 2:
A classification system acts as an intermediary between the anomaly detection models and the alert generation mechanism. The classification system processes anomaly detections and determines whether to generate alerts based on predefined criteria, filtering out false positives before they reach system administrators.
2Measurement precision
If more anomaly detection models are used to improve accuracy, then detection precision increases, but device complexity increases
Solution Approach 1:
Multiple anomaly detection models are merged into a unified system where they share common infrastructure components such as data collection mechanisms, processing frameworks, and alert management systems. This merging approach enables the system to leverage the strengths of multiple models while avoiding the full complexity burden of each individual model.
Solution Approach 2:
The system employs a universal anomaly detection framework that can accommodate multiple specialized models. This multi-functional framework provides common services like data normalization, model training coordination, and alert aggregation, reducing overall system complexity while supporting high-precision detection through multiple models.
3Reliability
If classification systems are implemented to reduce false positives, then anomaly classification capability is improved, but difficulty of detecting and measuring increases
Solution Approach 1:
Classification rules and criteria are established in advance before anomaly detection occurs. These pre-defined classification mechanisms enable the system to automatically categorize and filter anomalies without real-time complex analysis, reducing the difficulty of measurement while maintaining high classification accuracy.
Data Source
AI summary
Presented herein are system and methods for countermeasures to address anomalies in microservices. A server having one or more processors coupled with memory may receive a first plurality of metrics from a defined set of microservices for a function. The server may apply the first plurality of metrics to an ensemble of anomaly detection models to generate a plurality of classifications. Each classification may indicate the first plurality of metrics as one of anomalous or normal from a respective model of the ensemble of anomaly detection models. The server may identify a majority of the plurality of classifications as corresponding to an anomaly event. The server may determine, responsive to identifying the majority, that at least one of the first plurality of metrics satisfies a criterion of a policy of a plurality of policies. The server may perform a countermeasure identified by the policy to address the anomaly event.