Dynamic User-Defined Aggregation in Data Stream Management

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing data stream management systems (DSMS) are limited in their ability to support user-defined aggregations during normal operation, allowing only built-in aggregations such as SUM, COUNT, and AVG, and do not permit changes to query plans once processing has begun.

Innovation Solution

A DSMS is implemented to accept user-defined aggregation commands during normal operation, creating metadata for new aggregations and instantiating operators dynamically during query execution, allowing for the processing of new continuous queries using user-defined functions like secondMax, which can be reused across multiple queries.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If only built-in aggregations (SUM, COUNT, AVG) are supported in a DSMS, then the system maintains simplicity and reliability, but the adaptability and versatility are limited

Engineering Contradiction:
Improvesupport for user-defined aggregationsVSAvoidsystem complexity
Core Design Contradiction:
Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements a universal aggregation operator that can handle both built-in aggregations (SUM, COUNT, AVG) and user-defined aggregations through a common interface and execution mechanism. The operator is designed to be multi-functional, accepting different aggregation types without requiring separate dedicated operators for each type, thereby extending adaptability while controlling complexity through code reuse and standardized processing paths.

Inventive Principle:
Principle #6Universality (Multi-functionality)

2Adaptability or versatility

If query plans are fixed once processing begins, then the system maintains stability and reliability, but the adaptability to new queries is reduced

Engineering Contradiction:
Improvedynamic query plan acceptanceVSAvoidquery plan stability
Core Design Contradiction:
Adaptability or versatilityVSStability of the object's composition

Solution Approach 1:

The patent implements dynamic query plan acceptance by allowing the DSMS to receive and compile new continuous queries during normal operation. The system dynamically creates and integrates new query plans into the existing execution framework, enabling adaptability to new user-defined aggregations while maintaining stability through structured plan management and incremental integration of new query components into the running system.

Inventive Principle:
Principle #15Dynamics

3Reliability

If a new operator is created for each new aggregation, then the system ensures reliability and correctness, but the device complexity and resource usage increase

Engineering Contradiction:
Improveaggregation processing correctnessVSAvoidnumber of operators
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent employs a universal aggregation operator that can process multiple aggregation types including built-in functions and user-defined aggregations through a single unified interface. This multi-functional operator design ensures reliability and correctness through consistent processing logic while reducing device complexity by eliminating the need to create separate dedicated operators for each aggregation type, thereby reusing the same operator infrastructure across different aggregation scenarios.

Inventive Principle:
Principle #6Universality (Multi-functionality)

4Adaptability or versatility

If user-defined aggregations are supported during normal operation, then the adaptability improves, but the ease of operation and system management become more difficult

Engineering Contradiction:
Improvereal-time aggregation supportVSAvoidsystem management ease
Core Design Contradiction:
Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent implements self-service capabilities that allow users to define and register their own aggregation functions during normal system operation without requiring complex administrative intervention. The system provides automated mechanisms for validating user-defined aggregations, integrating them into the query processing framework, and managing their lifecycle, thereby improving adaptability while maintaining ease of operation through automated onboarding and management of custom aggregation functions.

Inventive Principle:
Principle #25Self-service

Data Source

PatentUS8204875B2Support for user defined aggregations in a data stream management system
Publication Date: 2012.06.19 ORACLE INT CORP
  • US8204875B2 patent drawing
  • US8204875B2 patent drawing
  • US8204875B2 patent drawing

AI summary

A computer is programmed to accept a command to create a new aggregation defined by a user during execution of continuous queries on streams of data. The computer is further programmed to thereafter accept and process new continuous queries using the new aggregation, in a manner similar to built-in aggregations. The user typically writes a set of instructions to perform the new aggregation, and identifies in the command, a location of the set of instructions. In response to such a command, the computer creates metadata identifying the new aggregation. The metadata is used to instantiate one aggregation for each group of data in a current window, grouped by an attribute identified in a new query.