Parallel Sort Using Ranged Partitioned Key-Value Store
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current sorting techniques for large datasets in high performance computing environments are inefficient due to bottlenecks in the read, write, and communication phases, particularly when using conventional parallel key-value stores.
Innovation Solution
Implementing a ranged, partitioned key-value store using Multidimensional Data Hashing Indexing Middleware (MDHIM) with Message Passing Interface (MPI) communications, where reader threads perform local sorts and send sorted subsets to range servers for further sorting and concatenation to achieve a globally sorted result.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional parallel key-value stores are used for sorting large datasets, then the system can handle distributed data storage, but the read, write, and communication phases create performance bottlenecks
Solution Approach 1:
The patent divides the key-value store into multiple partitions distributed across different nodes, with each node responsible for specific key ranges. This segmentation allows parallel processing of sorting operations across multiple nodes, reducing the communication and I/O bottlenecks that occur in conventional single-node approaches. Each node independently sorts its partition and communicates only necessary data to other nodes, significantly reducing overall sorting time.
Solution Approach 2:
The patent introduces a new dimension to the sorting problem by utilizing the key-range distribution across multiple nodes as an additional sorting dimension. Instead of sorting all data at a single node and then distributing it, the system distributes data first by key range, then performs sorting operations in parallel across multiple dimensions (nodes and key ranges), thereby reducing the communication and I/O overhead associated with conventional single-node sorting.
2Productivity
If data is distributed across multiple nodes for parallel processing, then sorting can be performed in parallel, but the complexity of managing data distribution and communication increases
Solution Approach 1:
The patent implements a self-service mechanism where each node automatically determines its responsibility for specific key ranges and independently manages its local data sorting operations. The system uses key-range knowledge to automatically route data to appropriate nodes without requiring centralized coordination or complex manual management. Each node serves itself by maintaining awareness of its assigned key ranges and handling its own sorting and communication needs, thereby reducing overall system management complexity.
Solution Approach 2:
The patent introduces key-range information as an intermediary that mediates between the distributed nodes. This intermediary mechanism allows nodes to automatically determine routing decisions based on key ranges without requiring complex inter-node communication protocols or centralized control. The key-range knowledge acts as a simple mediator that enables efficient data distribution and sorting across multiple nodes while minimizing management overhead.
Data Source
AI summary
Improved sorting techniques are provided that perform a parallel sort using a ranged, partitioned key-value store in a high performance computing (HPC) environment. A plurality of input data files comprising unsorted key-value data in a partitioned key-value store are sorted. The partitioned key-value store comprises a range server for each of a plurality of ranges. Each input data file has an associated reader thread. Each reader thread reads the unsorted key-value data in the corresponding input data file and performs a local sort of the unsorted key-value data to generate sorted key-value data. A plurality of sorted, ranged subsets of each of the sorted key-value data are generated based on the plurality of ranges. Each sorted, ranged subset corresponds to a given one of the ranges and is provided to one of the range servers corresponding to the range of the sorted, ranged subset. Each range server sorts the received sorted, ranged subsets and provides a sorted range. A plurality of the sorted ranges are concatenated to obtain a globally sorted result.