Stream Event Sequence Detection Using Hidden Markov Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to effectively detect anomalous sequences of events in high-frequency, high-resolution stream data using machine learning models, particularly in contexts like financial transactions to identify malicious activities.
Innovation Solution
Utilizing Hidden Markov Models (HMMs) to analyze and classify event sequences by preprocessing data, tokenizing actions, and training models on known normal and anomalous sequences to generate scores for likelihood of anomalous behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are used to detect anomalies in high-frequency stream data, then detection capability is improved, but computational complexity and processing time increase
Solution Approach 1:
The patent segments the stream data into discrete event sequences with specific time intervals. By dividing the continuous high-frequency data into manageable event sequences, the system can apply HMMs to each sequence independently, reducing overall computational complexity while maintaining detection capability.
Solution Approach 2:
The patent performs preliminary preprocessing of stream data before anomaly detection, including filtering, aggregation, and event sequence construction. This preliminary action reduces the volume and complexity of data that needs to be processed by the HMM, improving computational efficiency while preserving anomaly detection reliability.
2Measurement precision
If HMMs are trained on historical data to distinguish normal and anomalous sequences, then detection accuracy is improved, but training time and computational resources increase
Solution Approach 1:
The patent applies HMMs to specific event sequences rather than processing all stream data. By focusing on relevant event sequences that exhibit anomalous patterns, the system achieves high detection accuracy without requiring exhaustive training on all possible data scenarios, reducing training time and computational resources.
3Speed
If real-time processing of stream data is implemented, then responsiveness to anomalies is improved, but processing accuracy may deteriorate due to computational constraints
Solution Approach 1:
The patent segments stream data into discrete event sequences that can be processed independently and efficiently. This segmentation enables real-time processing of each sequence through HMMs while maintaining detection accuracy, as each sequence is analyzed completely before moving to the next, avoiding the trade-off between speed and precision.
Solution Approach 2:
The patent introduces event sequences as an intermediary representation between raw stream data and anomaly detection results. This intermediary structure allows for efficient real-time processing while preserving the temporal and contextual information needed for accurate anomaly detection, resolving the contradiction between processing speed and detection precision.
Data Source
AI summary
A method and a system for detecting an anomalous sequence of events in stream data are provided. The method includes: receiving a first set of raw data; analyzing the first set of raw data in order to determine a first event sequence; applying a first Hidden Markov Model (HMM) to the first event sequence in order to generate a first output; and determining, based on the first output, whether the first event sequence is classifiable as being an anomalous event sequence. The HMM is trained by using known sequences of normal events and event sequences that are known to be anomalous.