Partition Aware Query Processing in Distributed Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face inefficiencies in processing large datasets due to slow data processing times, particularly when dealing with partitioned data across multiple hosts, as current methods require transferring and combining data on a single host for query processing.
Innovation Solution
A calculation engine that detects partitioning flags and attributes to partition data across separate index servers for parallel processing, and unions intermediate results only when necessary, reducing unnecessary data transfer and improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is partitioned across multiple hosts and unioned on a single host for query processing, then data can be distributed to balance load, but data transfer time and processing time increase significantly
Solution Approach 1:
The patent divides the calculation scenario into multiple execution scopes, each handling a specific partition of the data. This segmentation allows parallel processing of different data partitions across multiple hosts without requiring complete data transfer to a single host, thus maintaining productivity while reducing data transfer time.
Solution Approach 2:
The patent introduces a new dimension of parallel execution by creating multiple execution scopes that can process different partitions simultaneously. This dimensional change from sequential union-then-process to parallel process-then-union enables efficient distributed processing while minimizing data transfer overhead.
2Reliability
If all partition data is transferred to a single host for unioning before processing, then complete data is available for query execution, but network bandwidth is consumed and processing slows down
Solution Approach 1:
The patent performs preliminary partitioning of the calculation scenario into multiple execution scopes before query execution. This preliminary action ensures that each execution scope has the necessary data locally available, eliminating the need to transfer complete data sets while maintaining query execution completeness through proper scope assignment.
Solution Approach 2:
The patent introduces execution scopes as intermediaries between the partitioned data sources and the final query processing. These execution scopes act as mediators that can independently process assigned partitions and produce intermediate results, which are then combined to achieve complete query execution without requiring complete data transfer.
3Productivity
If data is processed on separate hosts without unioning, then processing is faster and more distributed, but results cannot be combined for queries requiring all partitions
Solution Approach 1:
The patent creates dynamic execution scopes that can be independently assigned to different hosts based on data partitioning. This dynamic approach allows each execution scope to process data in parallel on separate hosts while maintaining the flexibility to combine results through the execution scope framework, achieving both parallel efficiency and result integrability.
Solution Approach 2:
The patent creates copies of the calculation scenario for each execution scope, allowing identical processing logic to be applied across multiple partitions simultaneously. This copying approach enables parallel processing on separate hosts while ensuring consistent result formats that can be easily combined, reducing the complexity of result integration.
4Ease of operation
If a single host processes all partitioned data after unioning, then processing is simpler to manage, but the host becomes a bottleneck and processing time increases
Solution Approach 1:
The patent segments the calculation scenario into multiple execution scopes that can be independently managed and assigned to different hosts. This segmentation maintains operational simplicity through structured scope definition while enabling parallel execution that eliminates the single-host bottleneck, thus reducing query execution time without significantly increasing management complexity.
Data Source
AI summary
Described herein includes a calculation scenario of a calculation engine that efficiently partitions data for processing at separate hosts, including in parallel, and unions intermediate results from such separate processing when required for further processing. Such parallel processing of partitions can allow for faster processing times, and such unioning of data only when required for further processing can limit the transferring of data that results in slower processing.