Matrix Profile Anomaly Detection in Streaming Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computing systems face challenges in analyzing ever-changing data streams to detect anomalies at both group and granular levels, leading to difficulties in identifying and resolving incidents promptly, which can result in increased costs and service disruptions.
Innovation Solution
The method involves creating a matrix profile from one or more parameters of a system by identifying subsequences, computing distance profiles, and determining minimum distances. This matrix profile is used to identify states based on threshold comparisons, allowing for the detection of potential discords and anomalies in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional anomaly detection methods are used on streaming data, then the system can identify incidents, but the analysis of ever-changing data streams at group and granular levels becomes difficult and time-consuming
Solution Approach 1:
The patent segments the streaming data into fixed-size windows and further divides each window into subsequences of length m. This segmentation allows the system to analyze data at multiple granularities (group level via window aggregation, granular level via subsequence comparison) without overwhelming computational complexity. The matrix profile algorithm then efficiently compares these segmented subsequences to identify anomalies.
2Measurement precision
If the system monitors data at high granularity to detect anomalies accurately, then detection precision improves, but the computational resources and time required increase significantly
Solution Approach 1:
The patent pre-computes the matrix profile for each subsequence, storing the minimum distance to any other subsequence in advance. This preliminary action allows the system to quickly compare new data points against historical patterns without performing exhaustive comparisons in real-time, thus maintaining high detection precision while reducing analysis time.
Solution Approach 2:
The system dynamically updates the matrix profile as new data arrives in streaming fashion. Instead of re-analyzing all historical data, the algorithm incrementally updates the distance profiles and matrix values, allowing continuous monitoring with constant or linear time complexity rather than quadratic complexity.
3Reliability
If the system processes all data points to ensure comprehensive anomaly detection, then detection coverage improves, but the computational cost and resource expenditure increase
Solution Approach 1:
The patent extracts only the essential features needed for anomaly detection by computing distance profiles and matrix values rather than processing raw data points directly. The algorithm extracts subsequences and computes their pairwise distances, storing only the minimum distance for each subsequence. This extraction reduces the data volume and computational requirements while maintaining comprehensive detection coverage.
Data Source
AI summary
A method for processing live streaming data includes creating matrix profiles for one or more parameters of a system including: identifying a set of subsequences from a stream of data, computing a distance profile for each subsequence of the set of subsequences, identifying a minimum calculated distance for each subsequence, and determining a matrix profile that includes one or more minimum calculated distances for each subsequence from the set of subsequences, identifying a first state for a first interval of the stream of data based on a comparison of the one or more minimum calculated distances from the matrix profile being below a threshold value over a span of time, and identifying a second state for a second interval of the stream of data based on a comparison of the one or more minimum calculated distances from the matrix profile being above the threshold value over a span of time.