Pattern Matching in Out-of-Order Event Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed IT systems face challenges in identifying and addressing incorrect behavior, particularly malicious or noncompliant behavior, due to out-of-order and missing event streams, which existing technologies fail to handle efficiently, leading to delayed detection and false positives/negatives.
Innovation Solution
A method that constructs automata and complement automata based on patterns, processes logged events with timestamps, determines gaps, and precomputes event matrices to iteratively match patterns, allowing for real-time detection of incorrect behavior without buffering or prior sorting, enabling prompt and accurate identification of policy violations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If events are buffered and ordered before analysis, then detection accuracy is improved, but detection delay increases
Solution Approach 1:
The patent precomputes event matrices for all possible events before actual event processing. This preliminary preparation allows the system to immediately match incoming events against pre-computed patterns without requiring buffering or sorting operations, thereby maintaining high detection accuracy while eliminating detection delay.
Solution Approach 2:
The system dynamically processes events as they arrive using timestamp-based ordering rather than static buffering. The event matrices are updated incrementally as new events are received, allowing the system to adapt to incoming data in real-time without requiring events to be buffered or reordered.
2Productivity
If events are processed in real-time without buffering, then detection speed is improved, but ability to handle out-of-order events deteriorates
Solution Approach 1:
The patent changes the parameter used for event ordering from arrival time to timestamp. By using timestamps embedded in events rather than processing order, the system can handle out-of-order events correctly while maintaining real-time processing speed. The event matrices are constructed based on timestamp sequences rather than input order.
3Stability of the object's composition
If buffering is used to handle out-of-order events, then event ordering is improved, but event loss is not prevented
Solution Approach 1:
The system uses feedback mechanisms to track the state of event processing through automata. When events are received out-of-order or with gaps, the feedback from the automata state transitions allows the system to identify and handle missing events appropriately, preventing information loss while maintaining correct ordering.
4Measurement precision
If complex pattern matching is performed on out-of-order streams, then detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent segments the pattern matching problem into separate event matrices for each possible event type. Instead of performing complex matching on the entire event stream at once, the system divides the computation into manageable matrix operations that can be processed independently and then combined, reducing overall computational complexity while maintaining accuracy.
Solution Approach 2:
The patent replaces traditional mechanical pattern matching algorithms with matrix-based computations. By substituting complex iterative matching procedures with efficient matrix operations, the system achieves the same pattern matching accuracy with significantly reduced computational complexity.
Data Source
AI summary
A method for determining incorrect behavior of components in a distributed information technology (IT) system includes receiving a pattern useable to indicate an incorrect behavior of a component. An automaton and a complement automaton are constructed based on the pattern, the automaton and complement automaton comprising one or more states. One or more logged events are received, each event in the one or more logged events including a timestamp. Gaps are determined in the one or more logged events. Event matrices are precomputed for the gaps and for each event in the one or more logged events based on the states of the automaton and the complement automaton. The pattern is matched to the one or more logged events by iteratively processing the one or more logged events and the gaps and maintaining a combination matrix. The incorrect behavior is determined based on an output of the pattern matching.