Time-Series Pattern Matching via Matrix Profile
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for identifying similar subsequences in time-series data, such as those used in trading decisions, are computationally intensive, require significant experience, and are impractical for large datasets, leading to missed opportunities and inefficiencies.
Innovation Solution
A system utilizing a pattern engine to generate similarity self-joins and incremental pattern engines for real-time updates, along with a user interface for visual representation and alert systems, enables efficient identification of similar subsequences through z-normalized Euclidean distance calculations and matrix profiles, allowing for scalable analysis without false negatives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual review of time-series charts is used for trading decisions, then investment intuition can be applied, but significant time and computational power are required
Solution Approach 1:
The system creates a matrix profile that is a compressed representation copying the essential pattern information from the full time-series data. This matrix profile serves as a simplified copy that can be quickly queried without processing the entire original dataset, enabling fast pattern matching while preserving the ability to identify meaningful patterns accurately.
Solution Approach 2:
The system performs preliminary computation to generate the matrix profile and its index structure before actual pattern matching queries are executed. This pre-processing step organizes the data in a way that enables exponentially faster querying, so when trading decisions need to be made, the heavy computational work has already been completed.
2Reliability
If full similarity self-join is performed on large datasets, then complete pattern matching is achieved, but computational intensity and storage requirements increase significantly
Solution Approach 1:
The system segments the pattern matching task into two parts: (1) computing the matrix profile that stores only the essential similarity information in a compressed form, and (2) querying this compressed representation. This segmentation avoids the need to perform full similarity joins on the entire dataset while still providing reliable pattern matching results.
Solution Approach 2:
The system extracts only the essential pattern information from the full time-series data and stores it in the matrix profile. Instead of keeping and processing all the original data relationships, it extracts and retains only the minimum necessary information (the distance to the nearest neighbor pattern) needed for reliable pattern matching, significantly reducing storage and computational requirements.
3Productivity
If real-time monitoring of multiple investments is implemented, then investment opportunities are not missed, but computational tractability is lost
Solution Approach 1:
The matrix profile system provides a universal solution that can be applied to monitor any number of different investments simultaneously. Once the matrix profile is computed for a dataset, it can answer multiple different pattern matching queries without requiring additional heavy computation, enabling scalable monitoring of multiple investments with constant computational overhead after the initial setup.
Data Source
AI summary
A system includes a pattern engine that, in response to selection by a user of a first data set, generates a similarity self-join of the first data set for a specified length. The similarity self-join indicates, for each reference subsequence, a minimum value of distances between the reference subsequence and other subsequences within the first data set. A user portal generates a user interface visually representing the first data set and identifying two subsequences that correspond to the lowest value of the similarity self-join. An alert system receives an alert request specifying a second data set and a pattern specification. An incremental pattern engine generates an initial state of a similarity join of the second data set and the pattern specification. In response to data being received for the first data set, the alert system transmits an alert message if any value of the similarity join meets a threshold.