Anomaly Detection Ensemble for High Dimensional Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional anomaly detection algorithms are ineffective in detecting small, meaningful anomalies, known as 'slow bleed' anomalies, and struggle to display complex information in a way that allows users to effectively interpret and remediate issues in large-scale monitoring systems, such as those related to website platforms.
Innovation Solution
The implementation of an ensemble of machine learning algorithms with a multi-agent voting system, combined with visual representation techniques like radar-based and tree map visuals, to detect and display anomalies in real-time, enabling users to identify and address problems within large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional anomaly detection algorithms are used, then large dips or spikes in metrics can be detected, but small meaningful anomalies (slow bleed anomalies) cannot be detected
Solution Approach 1:
The patent segments the anomaly detection task into multiple specialized algorithms, each optimized for different types of anomalies. The ensemble includes algorithms specifically designed to detect slow bleed anomalies (gradual deviations) separately from algorithms detecting sharp anomalies (dips/spikes). This segmentation allows each algorithm to excel at its specific detection task, resolving the contradiction between detecting large obvious anomalies and small subtle anomalies.
Solution Approach 2:
The patent merges multiple anomaly detection algorithms into an ensemble system that combines their results. By integrating the outputs of various algorithms (including those specialized for slow bleed detection with those for sharp anomaly detection), the system achieves both high precision for small anomalies and high reliability for large anomalies, resolving the contradiction between these two detection capabilities.
2Adaptability or versatility
If hundreds of metrics are monitored, then comprehensive system coverage is achieved, but user ability to interpret and take action on anomalies is reduced
Solution Approach 1:
The patent introduces an intermediary visualization layer that translates complex multi-metric anomaly data into intuitive visual representations. The visual anomaly report uses graphical elements to represent different anomaly types and their relationships, acting as a mediator between the comprehensive metric data and the user's interpretation ability. This allows comprehensive monitoring coverage while maintaining ease of operation through intuitive visual displays.
Solution Approach 2:
The patent employs color changes and visual encoding to represent different anomaly types, severities, and categories across hundreds of metrics. By using color-coded visual representations, the system maintains comprehensive monitoring coverage while making the data interpretable and actionable for users, resolving the contradiction between monitoring versatility and operational ease.
3Measurement precision
If multiple anomaly detection algorithms are combined, then detection accuracy for small anomalies improves, but system complexity increases
Solution Approach 1:
The patent segments the complex ensemble system into modular algorithm components, each with a specific detection function. This segmentation allows the system to combine multiple algorithms for high precision detection while managing complexity through clear modular architecture. Each algorithm module can be independently configured and maintained, reducing the operational complexity despite the multiple algorithms involved.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Described are computing systems and methods configured to detect a small, but meaningful, anomaly within one or more metrics associated with a platform. The system displays visuals of the metrics so that a user monitoring the platform can effectively notice a problem associated with the anomaly and take appropriate action to remediate the problem. An operational visual includes a radar-based visual with a heatmap arranging metrics, and a node representing a state of the metrics. Moreover, the system uses an ensemble of unsupervised machine learning algorithms for multi-dimensional clustering of hundreds of thousands of monitored metrics. Via the visuals and the implementation of the machine learning algorithms, the described techniques provide an improved way of representing and simulating many metrics being monitored for a platform. Moreover, the techniques are configured to expose actionable and useful information associated with the platform in a manner that can be effectively interpreted.