Intermediate Aggregate Query Generation for Data Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face scalability issues when handling large volumes of data streams, such as network traffic records, due to the high computational load of processing aggregate queries independently for each incoming record, which cannot keep up with high stream rates.
Innovation Solution
The method involves determining a set of aggregate queries, generating intermediate aggregate queries to pre-aggregate data, and executing these queries on pre-aggregated data to reduce computational overhead, using techniques like query execution plans, intermediate aggregates, and filter coalescing to optimize CPU cycles and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If aggregate queries are processed independently for each incoming record, then query accuracy is maintained, but system throughput deteriorates and cannot keep up with high stream rates
Solution Approach 1:
The patent segments the query processing into two distinct phases: a pre-aggregation phase that processes data in batches to generate intermediate results, and a final query phase that applies specific query logic to these pre-computed results. This segmentation allows the system to maintain accuracy for individual queries while dramatically improving throughput by avoiding redundant processing across multiple queries.
Solution Approach 2:
The patent implements preliminary action by performing pre-aggregation computations on incoming data streams before specific queries are executed. Intermediate aggregate results are computed and stored in advance, so when queries arrive, they can be answered using these pre-computed values rather than processing raw data from scratch, thereby maintaining accuracy while enhancing productivity.
2Productivity
If separate processing is used for each query, then query specificity is preserved, but computational overhead increases and processing efficiency decreases
Solution Approach 1:
The patent merges multiple query processing operations by identifying common aggregate computations across different queries and computing them once during the pre-aggregation phase. These shared intermediate results are then reused by multiple queries, reducing redundant computations and lowering overall computational overhead while maintaining the specificity of individual query results.
Solution Approach 2:
The pre-aggregation module performs universal processing that serves multiple query types simultaneously. By computing intermediate aggregate results that are applicable to various queries, the system achieves multi-functionality where a single processing pass benefits multiple downstream queries, thereby improving processing efficiency without sacrificing query-specific requirements.
3Loss of time
If all aggregate queries are executed on raw data streams, then complete data coverage is ensured, but computation time increases and real-time processing capability is lost
Solution Approach 1:
The patent extracts and computes only the essential aggregate features from raw data streams during the pre-aggregation phase, storing these intermediate results for later query processing. This extraction approach ensures that complete data coverage is maintained for all queries while significantly reducing computation time, as the heavy lifting of aggregate computation is performed once rather than repeatedly for each query.
Solution Approach 2:
By performing preliminary pre-aggregation computations on the entire data stream before queries are executed, the system ensures that all data is covered and processed once, then these pre-computed results are efficiently reused by multiple queries. This eliminates redundant processing and reduces computation time while maintaining complete data coverage across all queries.
Data Source
AI summary
A method includes determining, using a processor, a set of aggregate queries to be executed on a data stream, the set of aggregate queries comprising queries that perform respective sets of aggregation operations on respective sets of attribute values over respective time intervals. The method also includes generating, using the processor, at least one intermediate aggregate query for a subset of the set of aggregate queries, the at least one intermediate aggregate query combining a subset of aggregation operations for the subset of aggregate queries and a subset of attribute values. The method further includes executing, using the processor, the at least one intermediate aggregate query to generate pre-aggregated data from the data stream for the subset of aggregate queries and executing, using the processor, the subset of aggregate queries subsequent to executing the at least one intermediate aggregate query on the pre-aggregated data.