Streaming Query Engine for Progressive Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for processing large data sets, such as 'big data,' often result in non-interactive experiences due to the long time required to complete queries, making it difficult for developers to debug and receive timely results, especially when dealing with massive datasets.
Innovation Solution
The technique involves converting relational queries into streaming queries and associating progress intervals with data items, allowing a stream engine to provide incremental results, enabling partial and meaningful results to be generated and updated as the query progresses, even for atemporal data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional batch processing is used for large data sets, then complete query results are obtained, but the query execution time becomes excessively long and non-interactive
Solution Approach 1:
The patent segments the query execution process into incremental progress intervals, where results are computed and returned in discrete chunks rather than waiting for complete processing. Each progress interval represents a portion of the data set that has been processed, allowing users to receive partial results progressively while the system continues processing the remaining data in the background.
Solution Approach 2:
The system performs preliminary processing actions on portions of the data set before the complete query execution finishes. By pre-computing results for early progress intervals and making them available immediately, the system provides useful output without waiting for the entire data set to be processed, effectively performing useful work in advance of complete execution.
2Measurement precision
If traditional batch processing is used for large data sets, then accurate complete results are obtained, but interactive debugging becomes difficult
Solution Approach 1:
The system implements feedback by continuously providing incremental query results at each progress interval to the user interface. This allows developers to observe intermediate results, identify errors in their queries, and debug issues in real-time without waiting for complete execution. The feedback loop enables iterative query refinement while maintaining accuracy through progressive validation.
3Productivity
If progressive query computation is implemented, then interactive results are provided, but the system complexity increases
Solution Approach 1:
The patent introduces a streaming query engine as an intermediary layer between the traditional batch processing system and the user interface. This mediator translates standard SQL queries into streaming computations, managing the complexity of progressive execution internally while presenting a familiar query interface to users. The intermediary handles progress interval management, result aggregation, and coordination without requiring users to directly interact with the underlying streaming architecture.
Data Source
AI summary
The described implementations relate to processing of electronic data. One implementation is manifest as a technique that can include obtaining a relational query that references one or more data items and associating progress intervals with the data items. The technique can also include converting the relational query into a corresponding streaming query, and providing the streaming query and the data items with the progress intervals to a stream engine that produces incremental results of the query. For example, the progress intervals can be based on row numbers of a relational database table. The progress intervals can be used to define event lifetimes of streaming events that are provided as inputs to the stream engine.