Dynamic User-Defined Aggregation in Data Stream Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream management systems (DSMS) are limited in their ability to support user-defined aggregations during normal operation, allowing only built-in aggregations such as SUM, COUNT, and AVG, and do not permit changes to query plans once processing has begun.
Innovation Solution
A DSMS is implemented to accept user-defined aggregation commands during normal operation, creating metadata for new aggregations and instantiating operators dynamically during query execution, allowing for the processing of new continuous queries using user-defined functions like secondMax, which can be reused across multiple queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If only built-in aggregations (SUM, COUNT, AVG) are supported in a DSMS, then the system maintains simplicity and reliability, but the adaptability and versatility are limited
Solution Approach 1:
The patent implements a universal aggregation operator that can handle both built-in aggregations (SUM, COUNT, AVG) and user-defined aggregations through a common interface and execution mechanism. The operator is designed to be multi-functional, accepting different aggregation types without requiring separate dedicated operators for each type, thereby extending adaptability while controlling complexity through code reuse and standardized processing paths.
2Adaptability or versatility
If query plans are fixed once processing begins, then the system maintains stability and reliability, but the adaptability to new queries is reduced
Solution Approach 1:
The patent implements dynamic query plan acceptance by allowing the DSMS to receive and compile new continuous queries during normal operation. The system dynamically creates and integrates new query plans into the existing execution framework, enabling adaptability to new user-defined aggregations while maintaining stability through structured plan management and incremental integration of new query components into the running system.
3Reliability
If a new operator is created for each new aggregation, then the system ensures reliability and correctness, but the device complexity and resource usage increase
Solution Approach 1:
The patent employs a universal aggregation operator that can process multiple aggregation types including built-in functions and user-defined aggregations through a single unified interface. This multi-functional operator design ensures reliability and correctness through consistent processing logic while reducing device complexity by eliminating the need to create separate dedicated operators for each aggregation type, thereby reusing the same operator infrastructure across different aggregation scenarios.
4Adaptability or versatility
If user-defined aggregations are supported during normal operation, then the adaptability improves, but the ease of operation and system management become more difficult
Solution Approach 1:
The patent implements self-service capabilities that allow users to define and register their own aggregation functions during normal system operation without requiring complex administrative intervention. The system provides automated mechanisms for validating user-defined aggregations, integrating them into the query processing framework, and managing their lifecycle, thereby improving adaptability while maintaining ease of operation through automated onboarding and management of custom aggregation functions.
Data Source
AI summary
A computer is programmed to accept a command to create a new aggregation defined by a user during execution of continuous queries on streams of data. The computer is further programmed to thereafter accept and process new continuous queries using the new aggregation, in a manner similar to built-in aggregations. The user typically writes a set of instructions to perform the new aggregation, and identifies in the command, a location of the set of instructions. In response to such a command, the computer creates metadata identifying the new aggregation. The metadata is used to instantiate one aggregation for each group of data in a current window, grouped by an attribute identified in a new query.


