Network-Attached Memory Burst Sorting for Large Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed sorting techniques face performance limitations due to communication overhead and memory constraints, especially when dealing with large datasets and non-uniformly distributed data, which current solutions are not optimized for.
Innovation Solution
A sorting technique that leverages network-attached memory (such as fabric-attached memory) to perform burst sort, allowing independent sorting by processing entities (PEs) with a single exchange point, using a two-phase approach to minimize communication and coordination between nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed sorting techniques are used to handle large datasets, then the sorting capability is improved, but communication overhead and coordination complexity increase
Solution Approach 1:
The sorting process is divided into two distinct phases: a gathering phase where data is collected and sorted at each node locally, and a distribution phase where sorted data is exchanged between nodes. This segmentation reduces communication overhead by minimizing the need for continuous coordination during the entire sorting process.
Solution Approach 2:
Each node performs preliminary sorting of its local data before the distribution phase. By pre-sorting data at each node during the gathering phase, the system reduces the complexity of coordination needed during data exchange in the distribution phase, as nodes only need to exchange complete sorted records rather than continuously coordinating during sorting.
2Manufacturing precision
If multiple passes and intermediate data exchange are used, then sorting accuracy is improved, but processing time increases
Solution Approach 1:
The patent extracts the sorting operation from the data exchange process. Each node performs complete sorting of its local data independently during the gathering phase, then exchanges only the sorted results. This separation eliminates the need for multiple passes of sorting and exchange, reducing processing time while maintaining accuracy.
Solution Approach 2:
The two-phase approach ensures continuous useful action by performing sorting operations during the gathering phase before data exchange begins. This allows nodes to prepare and sort their data continuously without interruption for coordination, then execute the distribution phase efficiently with pre-sorted data, reducing overall processing time.
3Reliability
If data is exchanged among nodes, then sorting completeness is improved, but network traffic increases
Solution Approach 1:
Each node creates and exchanges copies of sorted data records with other nodes. By sorting data locally first and then exchanging only the necessary sorted copies, the system ensures sorting completeness while minimizing network traffic compared to exchanging unsorted data or requiring multiple passes of exchange and coordination.
Data Source
AI summary
A system, by a processing entity (PE), determines records stored in a memory shared by other processing entities. A respective record comprises a key and a corresponding index indicating a physical location in the shared memory. The PE stores, in a DRAM partition, burst tree data obtained by performing a burst sort on a portion of the records. The PE moves data stored in the DRAM partition to a partition of the shared memory in response to a size of the stored data exceeding a predetermined threshold. The PE obtains sorted keys and corresponding indices for the records by sorting the burst tree data stored in shared memory partitions. The PE facilitates a sorted retrieval of the records in the shared memory while leaving the records in the respective physical location in the shared memory by writing the sorted keys and indices to the shared memory.