Source Code Annotations for Parallel User-Defined Function Partitioning
Solution Overview
Problem
Current computing systems face challenges in optimizing the execution of user-defined functions within large data sets due to the lack of known pre-conditions, post-conditions, and behavior, making it difficult to achieve efficient parallel processing.
Innovation Solution
The implementation of source code annotations that specify properties such as partition arrangements, grouping, sorting, and runtime characteristics for user-defined functions, allowing the optimizer to generate an optimized query execution plan that leverages these properties for parallel execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If user-defined functions are executed in parallel on large data sets, then processing throughput is improved, but execution efficiency deteriorates due to lack of optimization information
Solution Approach 1:
The patent applies preliminary action by requiring developers to annotate user-defined functions with execution characteristics (such as partitioning behavior, sorting requirements, and grouping properties) before the query optimization phase. This advance information allows the query optimizer to generate optimized parallel execution plans without needing to analyze the actual function behavior at runtime, thus improving both throughput and execution efficiency.
2Adaptability or versatility
If custom functionality is added to handle large data sets, then processing capability is improved, but system reliability deteriorates due to unknown function behavior
Solution Approach 1:
The patent implements feedback by having developers provide annotation information about user-defined function behavior (such as whether the function preserves partitioning, performs sorting, or requires specific data arrangements). This feedback mechanism allows the query optimizer to make informed decisions about execution plan generation, ensuring that custom functionality is integrated reliably into the parallel processing framework while maintaining system-wide optimization goals.
Data Source
AI summary
Annotations can be placed in source code to indicate properties for user-defined functions. A wide variety of properties can be implemented to provide information that can be leveraged when constructing a query execution plan for the user-defined function and associated core database relational operations. A flexible range of permitted partition arrangements can be specified via the annotations. Other supported properties include expected sorting and grouping arrangements, ensured post-conditions, and behavior of the user-defined function.