RDMA In-Memory Data Repartitioning for Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data processing systems face inefficiencies in data repartitioning, particularly during operations like distributed joins and aggregations, due to high job execution times and costs associated with traditional communication methods.
Innovation Solution
The implementation of a distributed computing system that utilizes single-sided communication via Remote Direct Memory Access (RDMA) for in-memory data repartitioning, allowing data to be transiently stored in a distributed in-memory file system, with metadata management and block allocation to reduce latency and cost, and includes mechanisms for flushing data to permanent storage when necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional communication methods are used for data repartitioning, then system compatibility and ease of implementation are maintained, but job execution time and processing cost increase significantly
Solution Approach 1:
The patent introduces a file system interface as an intermediary layer between the distributed computing system and the underlying storage infrastructure. This file system abstraction enables RDMA operations to be performed without requiring applications to directly manage complex communication protocols, thus reducing job execution time while maintaining implementation simplicity through a standardized interface.
Solution Approach 2:
The patent replaces traditional CPU-mediated communication mechanisms with RDMA (Remote Direct Memory Access), which allows memory-to-memory data transfer between nodes without involving the CPU in the data path. This substitution eliminates CPU overhead and communication bottlenecks, significantly reducing job execution time and processing cost.
2Speed
If data is stored in permanent memory, then data persistence and reliability are ensured, but access latency and processing speed increase
Solution Approach 1:
The patent segments data storage into two distinct layers: transient in-memory storage for active processing and permanent disk storage for persistence. The in-memory file system provides low-latency access for speed-critical operations, while the permanent storage layer ensures data reliability. This segmentation allows the system to optimize for both speed and reliability simultaneously by accessing data from the appropriate layer.
Solution Approach 2:
The patent implements preliminary data loading into the in-memory file system before processing operations begin. By pre-positioning data in the fast in-memory layer, the system eliminates access latency during processing while maintaining the option to flush data to permanent storage for reliability, thus achieving both low latency and data persistence.
3Productivity
If in-memory storage is used for all data, then processing speed and access latency are improved, but system cost and resource consumption increase
Solution Approach 1:
The patent implements a dynamic storage management system where data automatically transitions between in-memory and disk-based storage based on access patterns and processing requirements. Frequently accessed data remains in the in-memory file system for high-speed processing, while less frequently accessed data is flushed to disk, optimizing the balance between processing speed and memory resource consumption.
Solution Approach 2:
The patent changes the storage parameter from a static all-in-memory approach to a dynamic hybrid approach. The system monitors data access patterns and adjusts the storage location (memory vs. disk) based on these patterns, allowing the same data to be stored in different locations at different times, thus optimizing both processing speed and resource utilization.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.