Multivariate Path-Based Anomaly Prediction for Distributed Applications
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed applications generate vast amounts of metrics data, making it computationally challenging to detect anomalies and predict future patterns, as monitoring agents collect thousands of measurements per hour, leading to noise and ineffective results due to the volume of patterns detected.
Innovation Solution
A multivariate path-based anomaly detection service that performs topology-based feature selection, multivariate clustering analysis, and path analysis to identify known and unknown anomalies, generating prediction events for proactive measures and root cause analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If monitoring agents collect thousands of measurements per hour from distributed application components, then the volume of detected patterns increases, but the results become ineffective due to noise and computational challenges
Solution Approach 1:
The patent segments the execution path into distinct components and identifies specific metrics for each component. Instead of analyzing all metrics from all components uniformly, the system divides the monitoring task by transaction type and execution path segment, focusing analysis on relevant portions of the system. This segmentation reduces noise by excluding irrelevant metrics while maintaining comprehensive coverage of critical areas.
Solution Approach 2:
The patent extracts and selects only the most relevant metrics from the vast available data based on execution path topology. The system identifies and extracts specific blame metrics associated with particular execution path segments, removing unnecessary data from the analysis. This extraction process converts the raw large-volume data into a focused set of meaningful indicators for anomaly detection.
2Adaptability or versatility
If all metrics from all components are monitored, then comprehensive coverage is achieved, but computational complexity increases making anomaly detection challenging
Solution Approach 1:
The patent applies local quality by tailoring the set of monitored metrics to each specific execution path segment and transaction type. Different components and path segments have different relevant metrics identified and monitored. This approach maintains comprehensive coverage of all critical areas while reducing overall computational complexity by not uniformly monitoring all possible metrics across all components.
Solution Approach 2:
The patent performs preliminary analysis to establish execution paths and identify relevant metrics before actual anomaly detection occurs. The system pre-processes the data by mapping metrics to execution path segments and transaction types in advance, creating a structured framework that simplifies subsequent real-time anomaly detection. This preliminary action reduces computational burden during active monitoring.
3Measurement precision
If traditional anomaly detection methods are used on large volumes of metrics data, then all patterns are detected, but noise increases leading to ineffective results
Solution Approach 1:
The patent introduces execution path topology as an intermediary framework that mediates between raw metrics data and anomaly detection analysis. This intermediary structure organizes metrics according to their relationship with specific execution paths and transaction types, filtering out noise before analysis. The topology-based approach acts as a mediator that preserves meaningful patterns while eliminating spurious correlations from the vast available data.
Data Source
AI summary
A multivariate path-based anomaly detection and prediction service (“anomaly detector”) can generate a prediction event for consumption by the APM manager that indicates a likelihood of an anomaly occurring based on path analysis of multivariate values after topology-based feature selection. To predict that a set of metrics will travel to a cluster that represents anomalous application behavior, the anomaly detector analyzes a set of multivariate date slices that are not within a cluster to determine whether dimensionally reduced representations of the set of multivariate data slices fit a path as described by a function.