Dynamic Split Node for Database Calculation Engine Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database management systems face inefficiencies in parallel processing due to limitations in data partitioning and processing paths, which hinder optimal utilization of multi-core CPUs and lead to suboptimal performance in handling complex queries and data operations.
Innovation Solution
A dynamic split node within a calculation model is introduced to partition data based on specific criteria, allowing separate processing paths for each partition, enabling efficient parallel execution and data distribution across multiple processing nodes, thereby optimizing data operations and query responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data are stored in column-based format to enable parallel processing, then processing efficiency on multi-core CPUs is improved, but random-access retrieval efficiency deteriorates
Solution Approach 1:
The patent segments data into multiple partitions distributed across different storage nodes, with each partition containing a subset of the data. This segmentation enables parallel processing operations on different partitions simultaneously while maintaining the ability to access specific partitions independently, thus resolving the contradiction between parallel processing efficiency and random-access retrieval speed.
Solution Approach 2:
The patent introduces a partitioning dimension orthogonal to the traditional row-column data organization. By adding this partitioning dimension, the system can perform parallel operations across partitions (improving productivity) while maintaining efficient access within each partition (preserving retrieval speed), effectively adding another dimension to the data access space.
2Productivity
If data are partitioned into multiple sections for parallel processing, then utilization of multi-core CPUs is improved, but system complexity increases
Solution Approach 1:
The patent creates partitioning mechanisms and processing frameworks that are universally applicable across different data types, query operations, and system configurations. The same partitioning infrastructure supports various parallel processing operations (aggregations, joins, filters) without requiring separate complex mechanisms for each operation, thus improving CPU utilization while controlling system complexity through multi-functionality.
3Quantity of substance
If compression is applied to column-based data to reduce storage space, then storage efficiency is improved, but access and processing speed deteriorates
Solution Approach 1:
The patent applies different compression strategies to different partitions or data segments based on their access patterns and characteristics. Frequently accessed partitions may use lighter compression or no compression to maintain fast access speed, while less frequently accessed partitions use aggressive compression to minimize storage space. This local differentiation resolves the contradiction between storage efficiency and access speed.
Data Source
AI summary
A dynamic split node defined within a calculation model can receive data being operated on by a calculation plan generated based on the calculation model. A partition specification can be applied to one or more reference columns in a table containing at least some of the received data. The applying can cause the table to be split such that a plurality of records in the table are partitioned according to the partition specification. A separate processing path can be set for each partition, and execution of the calculation plan can continue using the separate processing paths, each of which can be assigned to a processing node of a plurality of available processing nodes.