Dynamic Column Materialization Scheduling in RDBMS
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing column-oriented relational database management systems (RDBMSs) employ fixed materialization strategies (early or late) that do not optimize column materialization based on individual query requirements, leading to suboptimal performance.
Innovation Solution
A method for dynamically determining an optimal materialization schedule for each column in a query execution by transforming query text into a Rel DAG, performing bottom-up traversal, and computing a column-specific materialization schedule using dynamic programming to minimize CPU, disk, and network costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If fixed early materialization is used, then columns are fetched at leaf nodes and transmitted to parent operators, but interconnect bandwidth requirements increase and query processing costs increase
Solution Approach 1:
The patent implements dynamic materialization scheduling that adapts to each query's specific requirements. The system determines optimal materialization points by analyzing the query plan and computing schedules that minimize data transmission while ensuring columns are available when needed. This dynamic approach replaces static early materialization with query-specific materialization strategies.
Solution Approach 2:
The patent applies different materialization strategies to different columns based on their specific characteristics and query requirements. The system computes column-specific materialization schedules that determine exactly where and when each column should be materialized in the query plan, rather than applying a uniform early materialization policy to all columns.
2Loss of energy
If fixed late materialization is used, then columns are fetched just before processing, but query processing time increases due to repeated data retrieval
Solution Approach 1:
The patent determines optimal materialization points in advance by analyzing the query plan and computing schedules before execution. The system identifies exactly where columns should be materialized to minimize both transmission costs and retrieval delays, performing this optimization as a preliminary step before query execution.
Solution Approach 2:
The system uses cost-based optimization to determine materialization schedules, considering factors such as data size, access patterns, and query structure. This feedback-driven approach allows the system to make intelligent decisions about when to materialize columns based on the specific characteristics of each query.
3Productivity
If column-oriented storage is used, then data is stored as columns rather than rows, but columns need to be stitched together during query execution which increases processing complexity
Solution Approach 1:
The patent breaks down the complex materialization scheduling problem into manageable components by analyzing the query plan structure and computing schedules for individual columns and operators. The system segments the query execution plan and determines optimal materialization points for each segment, making the overall complex problem tractable.
Data Source
AI summary
A method of dynamically computing an optimal materialization schedule for each column in a column oriented RDBMS. Dynamic column-specific materialization scheduling in a distributed column oriented RDBMS is optimized by choosing a materialization strategy based on execution cost including central processing unit (CPU), disk, and network costs for each individual exchange operator. The dynamic programming approach is computationally feasible because the optimal schedule for a sub-plan is path independent.