Data Stream Aggregation Computation Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream management systems (DSMS) are limited in their ability to support user-defined aggregations during normal operation, allowing only built-in aggregations such as SUM, COUNT, and AVG, and do not allow query plans to be changed once processing begins, restricting real-time adaptability and flexibility in processing data streams.
Innovation Solution
A DSMS is programmed to accept commands for creating new user-defined aggregations based on other aggregations, allowing registration and execution of these aggregations during normal operation, with the system automatically determining the order of execution and sharing computation between new and existing aggregations to avoid redundant calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the system only supports built-in aggregations (SUM, COUNT, AVG) and does not allow query plans to be changed during normal operation, then the system maintains simplicity and stability, but the adaptability and flexibility for processing complex data streams are limited
Solution Approach 1:
The system dynamically allows registration of new user-defined aggregations during normal operation without requiring system restart or plan recompilation. The query execution engine can adapt to new aggregation types on-the-fly, making the system flexible and dynamic in handling evolving query requirements while maintaining operational stability
Solution Approach 2:
The system provides a universal aggregation framework that supports both built-in aggregations (SUM, COUNT, AVG) and user-defined aggregations through a common interface. The query execution engine is designed to handle multiple aggregation types uniformly, allowing the system to perform diverse aggregation functions through a single unified mechanism
2Adaptability or versatility
If the system allows user-defined aggregations based on other aggregations, then the versatility and functionality are improved, but the computation redundancy and processing overhead increase
Solution Approach 1:
The system merges the computation of base aggregations into a single shared execution path. When a user-defined aggregation depends on multiple base aggregations, the engine executes the base aggregations once and shares their results across all dependent user-defined aggregations, eliminating redundant calculations and reducing overall computation overhead
Solution Approach 2:
The system performs preliminary execution of base aggregations before computing user-defined aggregations that depend on them. By pre-computing and caching the results of base aggregations, the system avoids re-computing them multiple times for different user-defined aggregations, thereby reducing computation overhead
3Productivity
If the system executes multiple aggregations independently without sharing computation, then the implementation is simple, but the processing efficiency and performance are reduced
Solution Approach 1:
The query execution engine merges the execution of multiple aggregations by identifying common base aggregations and executing them once with shared computation logic. The results are then distributed to all user-defined aggregations that depend on them, improving processing efficiency while maintaining manageable implementation complexity through systematic optimization
Data Source
AI summary
A computer is programmed to process a continuous query that is known to perform a new aggregation on one or more stream(s) of data, using one or more other aggregations on the stream(s). The computer creates an operator to execute the continuous query, and schedules the operator for execution in a specific order. In several embodiments, the computer determines the order based on dependency of the new aggregation on other aggregation(s), and on the order of performance of the other aggregation(s). The new aggregation is scheduled for performance after performance of each of the other aggregations. The computer is further programmed to pass results of the other aggregations to the new aggregation, by execution of a predetermined function. Support for use of the other aggregations results within the new aggregation eliminates redundant computation of the other aggregations within the new aggregation. The new aggregation may be user defined or built-in.