Non-homogenous Synopsis for Partition Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing in-memory databases face inefficiencies in query processing due to uniform synopsis strategies applied to all data partitions, which do not account for varying data distributions and attributes, leading to suboptimal performance and resource utilization.
Innovation Solution
Implementing non-homogenous synopsis information and strategies, where the query optimizer dynamically selects the best synopsis strategy based on data partition metadata, such as skewed or uniformly distributed data, to determine which partitions to load into memory, thereby optimizing query execution and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a uniform synopsis strategy is applied to all data partitions, then the system structure is simple and easy to implement, but query performance is suboptimal due to inability to adapt to varying data distributions
Solution Approach 1:
The patent applies different synopsis strategies to different data partitions based on their specific characteristics. Each partition is evaluated for data distribution properties (skewed vs. uniform) and assigned the most appropriate synopsis strategy accordingly. This local differentiation enables optimal query performance for each partition while maintaining overall system manageability through automated selection mechanisms.
2Productivity
If more synopsis strategies are implemented to handle different data distributions, then query performance improves, but the complexity of the system increases
Solution Approach 1:
The system dynamically selects the appropriate synopsis strategy for each data partition based on its data distribution characteristics. Rather than using a fixed uniform strategy, the system adapts its behavior at runtime by evaluating partition properties and choosing the optimal strategy (e.g., histogram-based for skewed data, uniform sampling for uniform data). This dynamic adaptation improves query processing efficiency without requiring manual configuration of multiple strategies.
Solution Approach 2:
The system automatically determines which synopsis strategy to apply to each partition based on its own data distribution properties. The query optimizer evaluates partition characteristics and self-selects the most appropriate strategy without external intervention. This self-service mechanism enables the system to optimize query performance while managing the complexity of multiple strategies through automated decision-making.
3Reliability
If all data partitions are loaded into memory for processing, then query accuracy is maintained, but resource utilization is inefficient due to unnecessary data loading
Solution Approach 1:
The system performs preliminary evaluation of data partitions using synopsis information before actually loading data into memory for query processing. By pre-evaluating partition properties and selecting appropriate synopsis strategies in advance, the system can accurately predict which partitions are necessary to load and which can be safely pruned. This preliminary action maintains query result accuracy while significantly improving memory resource utilization by avoiding unnecessary data loading.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed herein are system, method, and computer program product embodiments for partition pruning via non-homogenous synopsis information. An embodiment operates by maintaining synopsis information for a data partition in accordance with a first synopsis strategy, monitoring performance of the synopsis information within a partition pruning system, determining that the performance of the synopsis information is insufficient, and updating the synopsis information in accordance with a second synopsis strategy better suited for the attributes of the data partition. In some embodiments, a first data partition of a partitioned data table may employ a first synopsis strategy and a second data partition of the partitioned data table may employ a second synopsis strategy.