Automaton-Based Pattern Matching in Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database models are not well-suited for processing data streams, which are continuous and unbounded, making it inefficient to detect patterns in real-time data events such as sensor readings and financial tickers.
Innovation Solution
The use of regular expressions to specify patterns in data streams, with a system that processes events in real-time to detect pattern occurrences without backtracking, by constructing an automaton based on the pattern type and maintaining bindings for partial or full matches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional database models are used to process data streams, then data storage and querying capability are provided, but real-time pattern detection efficiency deteriorates
Solution Approach 1:
The patent segments the pattern matching process into distinct phases: compilation phase where the regular expression is converted into an automaton, and execution phase where the automaton processes data stream events. This segmentation allows pre-computation of matching logic, improving real-time detection efficiency while maintaining reliable pattern matching capability.
Solution Approach 2:
The patent performs preliminary action by compiling the regular expression pattern into an automaton structure before processing the data stream. This pre-processing creates an optimized matching machine that can efficiently evaluate patterns against incoming events without re-parsing the regex during real-time execution, thus improving productivity while preserving reliability.
2Measurement precision
If backtracking is performed during pattern matching, then complete pattern detection is achieved, but processing time increases
Solution Approach 1:
The patent substitutes the traditional backtracking mechanism (a mechanical search process) with an automaton-based state transition system. The automaton processes events sequentially through defined states, eliminating the need for backtracking while maintaining complete pattern detection accuracy. This substitution replaces inefficient mechanical re-examination with a streamlined state-machine approach.
Solution Approach 2:
The patent introduces an automaton as an intermediary between the regular expression pattern and the data stream events. This intermediary structure pre-processes the pattern into a state transition table that guides matching without requiring backtracking, thus achieving both complete detection accuracy and reduced processing time.
3Reliability
If data events are re-processed for pattern matching, then complete pattern detection is ensured, but real-time processing capability deteriorates
Solution Approach 1:
The patent performs preliminary compilation of the regular expression into an automaton structure before real-time processing. This pre-computation creates a ready-to-execute matching machine that processes events in a single pass without re-processing, ensuring complete pattern detection while maintaining real-time processing speed.
Solution Approach 2:
The patent enables continuous processing of data stream events through the automaton without interruption or re-processing. The state machine continuously transitions through defined states as events flow through the system, maintaining both complete detection reliability and continuous real-time processing capability.
Data Source
AI summary
Techniques for detecting patterns in one or more data or event streams. A pattern to be detected may be specified using a regular expression. Events received in a data stream are processed during runtime to detect occurrences of the specified pattern in the data stream. In one embodiment, a pattern type or class is determined for the specified pattern and pattern matching is performed using a technique selected based upon the type or class determined for the specified pattern.