Function Descriptors for Cross-Platform Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data processing systems face challenges in optimizing queries that invoke data set functions (DSFs) due to unknown properties such as output schema, required input columns, and predicate evaluation, leading to inefficiencies in data transfer and query execution.
Innovation Solution
The introduction of function descriptors expressed in markup language, which define properties of DSFs, allowing the database system to infer and optimize query execution by specifying output schema, input schema, predicate push, and cardinality estimation, thereby reducing data transfer and improving query planning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the database system executes DSFs as blackboxes without property knowledge, then the system can maintain simplicity and avoid complex property specification mechanisms, but query optimization is limited and data transfer efficiency is poor
Solution Approach 1:
The function descriptor is defined and registered before the DSF execution occurs. This preliminary specification of input schema, output schema, and predicate push capabilities allows the query optimizer to make informed decisions during query compilation and execution planning, eliminating the need for runtime property discovery mechanisms.
Solution Approach 2:
The function descriptor acts as an intermediary information carrier between the DSF definition and the query optimizer. It mediates the communication by providing structured metadata about the DSF's input/output schemas and optimization capabilities, enabling the optimizer to generate efficient execution plans without direct inspection of the DSF implementation.
2Quantity of substance
If the database system transfers all input columns to the DSF, then the DSF can process complete data without missing information, but data transfer volume increases and network bandwidth is consumed
Solution Approach 1:
The input schema specification in the function descriptor allows the query optimizer to extract and send only the necessary columns required by the DSF, excluding redundant columns. This selective data transfer reduces network bandwidth consumption while ensuring the DSF receives exactly the information it needs for correct processing.
3Ease of manufacture
If the database system does not know the output schema of the DSF, then the system avoids complex schema negotiation protocols, but query compilation and result processing are inefficient
Solution Approach 1:
The output schema is pre-specified in the function descriptor during DSF registration. This advance declaration eliminates the need for runtime schema negotiation between the database system and coprocessor, allowing the query compiler to immediately generate efficient execution plans and result processing statements without iterative schema discovery protocols.
4Loss of time
If the database system cannot push predicates inside the ON clause, then the query structure remains simple and easy to process, but filtering operations are performed after DSF execution increasing processing time
Solution Approach 1:
The predicate push capability is dynamically configured through the function descriptor, allowing the query optimizer to adaptively rewrite queries by pushing predicates inside the ON clause when the DSF supports it. This dynamic optimization enables filtering before DSF execution, reducing the amount of data processed while maintaining query correctness through the descriptor's capability specifications.
Data Source
AI summary
A database system receives a request from a user. The request invokes a data set function (DSF) and uses a property to be provided by the DSF. The database system determines that a function descriptor is available for the DSF. The function descriptor is expressed as markup language instructions. The function descriptor defines the property of the DSF. The database system uses the function descriptor to define a property for the DSF.