Global Context Structure for Parallel Database UDF Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for implementing user-defined functions (UDFs) in parallel databases require end users to have knowledge of the underlying architecture, leading to inefficient use of parallel machines and UDFs, as they must invoke UDFs multiple times for each phase and manage data partitioning strategies manually.
Innovation Solution
The solution allows users to define UDFs using a global context structure that is set up once and reused across multiple executions, enabling efficient execution of multi-phase processes without requiring users to specify partitioning strategies, by utilizing a leader node to initialize and distribute the global context to compute nodes, which process slices of data independently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional techniques are used to implement UDFs in parallel databases, then users can customize database functionality, but users must have knowledge of underlying architecture and manually manage data partitioning, increasing operation complexity
Solution Approach 1:
The patent introduces a database system intermediary that automatically manages data partitioning and UDF invocation across compute nodes. This intermediary layer handles the complexity of parallel architecture management, allowing users to invoke UDFs without needing to understand underlying partitioning strategies or architecture details, thus resolving the contradiction between customization capability and operational simplicity
2Productivity
If users invoke UDFs multiple times for each phase in parallel execution, then data processing can be distributed across compute nodes, but execution time increases due to repeated function invocations
Solution Approach 1:
The patent implements preliminary actions by pre-distributing UDF code and global context structures to all compute nodes before query execution begins. This allows each compute node to independently execute UDFs locally without repeated network invocations, maintaining parallel processing capability while significantly reducing execution time through localized function execution
3Productivity
If global context structure is set up once and reused across multiple executions, then resource utilization improves, but memory management complexity increases
Solution Approach 1:
The patent creates a universal global context structure that serves multiple query executions and can be shared across different compute nodes. This single context structure is designed to handle multiple functions including storing lookup tables, configuration data, and intermediate results, thereby improving resource utilization while the database system automatically manages the complexity of memory allocation and deallocation
Data Source
AI summary
A parallel database system processes database queries invoking a user defined function (UDF). The database query is specified in a database query language, for example, the structured query language (SQL) and the UDF is specified in a programming language different from the database query language, for example, C/C++. The UDF may include multiple phases. The parallel database system generates optimized code for the UDF. The parallel database system includes a leader node and a plurality of compute nodes. The database query processes one or more database tables such that each compute node processes a slice of each database table. The leader node sets up a global context structure and provides it to each compute node. The global context may be shared across multiple executions of the query. The UDF may be used for efficient execution of extraction, transformation, and loading of data from external systems.