Event Processing Language Retain Clause for Real-Time Stream Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database languages, such as SQL, are inadequate for processing real-time data streams in complex event processing due to the high volume and speed requirements, as they typically involve storing data before processing, which is not feasible in event processing systems.
Innovation Solution
The development of an Event Processing Language (EPL) that includes clauses like RETAIN, MATCHING, and OUTPUT to manage and process event streams in real-time, allowing for the definition of windows for data retention, pattern matching, and output throttling, enabling efficient processing of streaming data without the need for pre-storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional database languages like SQL are used to process data, then data can be processed with standard querying capabilities, but data must be stored in a database before processing which is not feasible for real-time high-volume event processing
Solution Approach 1:
The patent extracts the data storage requirement from the data processing function by implementing an in-memory event processing system that eliminates the need for traditional database storage. Events are processed directly in memory as they arrive, separating the processing capability from storage infrastructure and enabling real-time high-volume event processing without database bottlenecks.
2Loss of information
If all events are retained for processing, then complete data is available for analysis, but memory resources are consumed excessively under high volume scenarios
Solution Approach 1:
The patent implements dynamic event retention windows that automatically adjust the retention period based on processing requirements. The RETAIN clause allows events to be held in memory for a specified time window or number of events, after which they are automatically discarded. This dynamic approach ensures complete data availability within the retention window while preventing unbounded memory consumption under high volume scenarios.
3Measurement precision
If complex patterns are detected in event streams, then accurate event recognition is achieved, but processing time increases
Solution Approach 1:
The patent applies preliminary filtering and grouping operations before complex pattern matching. The MATCHING clause is optimized to work with pre-filtered event streams that have already been segmented by the RETAIN clause into manageable time windows. This preliminary action reduces the search space for pattern matching algorithms, maintaining high detection accuracy while minimizing processing time by avoiding unnecessary comparisons across the entire event history.
Data Source
AI summary
An event processor can use event processing queries to operate an event. Event processing queries can include a “retain” clause that limits the amount of data over which the query is run.