Inverse Distribution Function Operations in Parallel Relational Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Determining data values at specific percentiles in a large parallel database distributed across multiple nodes is inefficient and costly due to the need for data ordering, which is impractical and requires massive data movement.
Innovation Solution
The solution involves determining the number of elements in a dataset, sorting and ordering data in a distributed manner across multiple segments, and using processes like pickup global, pickup hash, and pickup sort to efficiently calculate data values at desired percentile levels, reducing the need for extensive data movement and processing costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is ordered globally across all nodes to determine percentile values, then measurement precision of percentile values is improved, but loss of time and processing efficiency deteriorate due to multiple data revisits and massive data movement
Solution Approach 1:
The patent divides the distributed dataset into multiple segments, each maintained in sorted order on its respective node. Instead of globally sorting all data, each node independently manages its segment's order, enabling parallel processing while maintaining the ordering constraint locally. This segmentation allows percentile calculations to proceed without massive data movement across the network.
Solution Approach 2:
The patent introduces intermediary structures (such as segment boundary trackers and distributed selection algorithms) that enable percentile calculations without requiring complete global sorting. These intermediaries facilitate the selection of percentile values by coordinating information across segments without moving all data, thus resolving the contradiction between accuracy and efficiency.
2Measurement precision
If data is sorted into a specific order before percentile calculation, then measurement precision of percentile values is improved, but device complexity and data movement requirements worsen
Solution Approach 1:
The patent segments the sorting operation into independent local sorts on each node rather than a single global sort. Each node maintains its segment in sorted order using local sorting algorithms, eliminating the need for complex global coordination and data movement infrastructure. This reduces device complexity while preserving the ordering necessary for accurate percentile calculation.
Solution Approach 2:
The patent performs preliminary local sorting on each data segment before the percentile calculation begins. By pre-sorting data locally on each node, the system eliminates the need for complex runtime sorting operations and reduces the overall complexity of the percentile calculation process, while ensuring data is ready for accurate percentile determination.
3Productivity
If parallel processing is used on separate data subsets, then productivity is improved, but measurement precision of percentile values deteriorates because the median of medians is not equal to the overall median
Solution Approach 1:
The patent segments the dataset into distributed portions that can be processed in parallel while maintaining local sorted order. Each segment is independently processed to identify local percentile-relevant positions, and these local results are then coordinated using distributed selection algorithms that account for the relative positions across segments, preserving overall percentile accuracy while enabling parallel execution.
Solution Approach 2:
The patent implements feedback mechanisms where information about segment boundaries and local data distributions is communicated across nodes. This feedback enables the parallel processing algorithm to adjust its calculations based on the relative positioning of segments, ensuring that the final percentile values are accurate despite the distributed parallel processing approach.
Data Source
AI summary
Inverse distribution operations are performed on a large distributed parallel database comprising a plurality of distributed data segments to determine a data value at a predetermined percentile of a sorted dataset formed on one segment. Data elements from across the segments may be first grouped, either by partitioning keys or by hashing, the groups are sorted into a predetermined order, and data values corresponding to the desired percentile are picked up at a row location of the corresponding data element of each group. For a global dataset that is spread across the database segments, a local sort of data elements is performed on each segment, and the data elements from the local sorts are streamed in overall sorted order to one segment to form the sorted dataset.