User-Defined Function Instance Reuse in Data Stream Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data stream management systems (DSMS) are limited in allowing only built-in functions for queries, failing to support user-defined functions, which restricts flexibility and adaptability in processing real-time data streams.
Innovation Solution
Implementing a data stream management system that allows users to create and integrate new functions during operation, using a metadata repository to manage user-defined functions and invoke them dynamically within continuous queries, optimizing memory usage by reusing function instances across multiple queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If only built-in functions are supported in DSMS, then system stability and ease of implementation are improved, but adaptability and versatility deteriorate
Solution Approach 1:
The system separates built-in functions from user-defined functions into distinct management mechanisms. Built-in functions remain in the core system for stability, while user-defined functions are managed through external registration and metadata repositories, allowing independent evolution and reducing system coupling.
Solution Approach 2:
A metadata repository acts as an intermediary layer between the query compiler and user-defined functions. This intermediary stores function signatures, implementations, and registration information, enabling the system to dynamically resolve and invoke user-defined functions without requiring direct integration into the core execution engine.
2Adaptability or versatility
If user-defined functions are allowed during normal operation, then adaptability and versatility are improved, but device complexity and difficulty of management increase
Solution Approach 1:
User-defined functions are registered and validated before being invoked during query execution. The system performs preliminary registration of function signatures, implementations, and metadata in a dedicated repository, ensuring that all user-defined functions are properly configured and available before normal query processing begins.
Solution Approach 2:
The query compiler automatically resolves user-defined function names by querying the metadata repository and retrieving the appropriate function implementations. This self-service mechanism eliminates the need for manual configuration or complex system administration during query execution, as the compiler autonomously manages function resolution and invocation.
3Ease of operation
If function instances are created for each query, then ease of operation is improved, but memory usage and resource consumption increase
Solution Approach 1:
Multiple queries that reference the same user-defined function share a single function instance from the metadata repository. The system merges function resolutions by storing function implementations once in the repository and allowing multiple queries to reference and reuse the same registered function, eliminating redundant instantiations.
Solution Approach 2:
Instead of creating full function instances for each query, the system creates lightweight references or pointers to the registered function definitions in the metadata repository. These references enable queries to access function implementations without duplicating the actual function code or state, significantly reducing memory consumption.
Data Source
AI summary
A data stream management system (DSMS) is designed to support a new user-defined function, by creating and using at least two structures as follows. A first structure (“metadata entry”) is created in response to a command for creation of the new function, and maps a single instance of a class to the function's name. A second structure is created with creation of an operator on receipt of each new continuous query that uses the new function. The second structure (“operator specific data structure”) contains a path to the newly-created instance, which path is obtained by looking up the first structure. Additional second structures are created on receipt of additional continuous queries which use the new function, but all second structures contain the same path. All continuous queries use the same instance. Repeated use of a single instance to compile and execute multiple queries eliminates repeated instantiation of the same function.