Memory Optimized Data Shuffle via In-Memory Compression and RDMA
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data shuffling techniques in distributed data processing systems consume significant system resources, impacting overall performance due to the data path from memory to disk storage and then to the network, leading to high resource consumption.
Innovation Solution
Implementing a memory-optimized data shuffling mechanism with hybrid memory configurations, including dynamic random access memory (DRAM) and non-volatile random access memory (NVRAM), and using in-memory compression with remote direct memory access (RDMA) for direct peer-to-peer data transfer between nodes, reducing resource consumption and enhancing performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transferred through disk storage in conventional data shuffling, then data can be persisted and processed, but system resource consumption increases and performance decreases
Solution Approach 1:
The patent extracts the data transfer path from the conventional memory-disk-network route and removes the disk storage intermediate step. By using remote direct memory access (RDMA) technology, data is transferred directly between memory pools of different nodes, eliminating the need to write data to disk and read it back, thus reducing I/O operations and system resource consumption while improving data shuffling performance
Solution Approach 2:
The patent introduces a memory pool as an intermediary structure between nodes, configured with hybrid memory (DRAM and NVRAM). This memory pool serves as a buffer that enables direct data transfer between producer and consumer nodes without involving disk storage, thereby reducing the data path length and system resource usage while maintaining data persistence capabilities
2Quantity of substance
If in-memory compression is applied to data sets, then memory usage is reduced and transfer efficiency improves, but compute resources are consumed for compression operations
Solution Approach 1:
The patent applies compression operations preliminarily to data sets before they are loaded into the memory pool or during their initial storage. By compressing data in advance using compression algorithms (such as LZ4, Zstandard, or Snappy), the system reduces the volume of data that needs to be stored and transferred, thereby decreasing memory usage and network bandwidth consumption. The compute resources are consumed once during compression, but this preliminary action yields continuous benefits throughout the data lifecycle
Solution Approach 2:
The patent changes the physical state of data by applying compression algorithms that transform raw data into compressed formats. This parameter change reduces the size of data sets stored in memory, allowing more data to be held in the same memory capacity. The hybrid memory configuration (DRAM and NVRAM) works in conjunction with compression to optimize both speed and capacity utilization
Data Source
AI summary
In a distributed data processing system with a set of multiple nodes, a first data shuffle memory pool is maintained at a data shuffle writer node, and a second data shuffle memory pool is maintained at a data shuffle reader node. The data shuffle writer node and the data shuffle reader node are part of the set of multiple nodes of the distributed data processing system. In-memory compression is performed on at least a portion of a data set from the first data shuffle memory pool. At least a portion of the compressed data is transmitted from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner. Each of the first data shuffle memory pool and the second data shuffle memory pool may include a hybrid memory configuration.