Continuous Query Processing for Unbounded Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database systems are inadequate for processing continuous data streams due to their inability to handle long-running queries over unbounded data sets, lacking the necessary processing flexibility required for modern event stream-based applications.
Innovation Solution
A computer-implemented method and system for detecting patterns across multiple input data streams by receiving and processing continuous queries, generating dynamic data types, and combining data streams based on a homogeneous schema to identify patterns, such as occurrences of events in one stream followed by events in another.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional database systems are used to store and query data, then data can be managed in structured tables with SQL queries, but the system cannot handle continuous unbounded data streams or execute long-running queries effectively
Solution Approach 1:
The patent combines traditional database management capabilities with stream processing capabilities into a unified system. The query processor can handle both static database queries and continuous stream queries, merging two previously separate functionalities into one system that can process both finite datasets and unbounded data streams using a common query language extension
Solution Approach 2:
The system provides universal query processing capability that works across different data types (static databases and continuous streams) and different query modes (one-time queries and long-running queries). The extended SQL language and query processor are designed to be multi-functional, handling various data formats and processing requirements through a single unified interface
2Duration of action of stationary object
If SQL queries are executed on finite data sets, then results are produced efficiently, but the queries cannot run continuously over unbounded data streams
Solution Approach 1:
The system transforms static SQL queries into dynamic continuous queries that can adapt to ongoing data streams. The query processor executes queries continuously over time, automatically updating results as new data arrives in the stream, rather than producing a single static result. This enables queries to run indefinitely over unbounded data while maintaining productivity through efficient incremental processing
Solution Approach 2:
The system enables continuous execution of queries over data streams, maintaining useful processing action indefinitely. Rather than completing and terminating, the query processor continues to evaluate incoming data and update results in real-time, ensuring continuous useful action on unbounded data streams without loss of processing efficiency
3Adaptability or versatility
If data streams with different attributes are processed, then comprehensive event detection is possible, but schema compatibility issues arise when combining streams
Solution Approach 1:
The system introduces an intermediary schema resolution layer that mediates between data streams with different attributes. When combining streams, the system automatically resolves schema differences by identifying common attributes and handling stream-specific attributes, acting as an intermediary that translates between different stream schemas and enables compatible joining without manual intervention
Solution Approach 2:
The system dynamically adjusts schema parameters when processing multiple data streams. It identifies common attributes across streams and modifies the query execution plan to handle attribute differences, changing parameters such as which attributes to join on and how to handle stream-specific fields, enabling flexible processing of heterogeneous streams
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for detecting patterns across multiple input data streams related to one or more applications is disclosed. The method includes receiving multiple input data streams and generating one or more dynamic data types for one or more attributes of the input data streams. In some embodiments, the method may include combining the input data streams to generate a combined input data stream based on the dynamic data types and processing a continuous query over the combined data stream to detect a pattern.