Spark Shuffle RDMA Bypassing OS Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Shuffle stage in Spark operations consumes excessive resources and has low data throughput due to the need for network and disk operations across nodes, affecting overall performance.
Innovation Solution
A system and method for remote direct memory access (RDMA) based on Spark Shuffle, which includes a transmitting end and a receiving end with managers and channels, using RDMA to directly access and verify data without operating system intervention, and an ETL processing module for preprocessing data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional Shuffle operation is used in Spark, then data can be transferred between nodes, but excessive CPU, RAM, disk and network resources are consumed
Solution Approach 1:
The patent extracts the operating system intervention and data copying layers from the traditional Shuffle process by implementing RDMA (Remote Direct Memory Access) technology. This allows data to be transferred directly between node memories without passing through the operating system's file system and network stack, thereby eliminating unnecessary resource consumption while maintaining data transfer functionality.
Solution Approach 2:
The patent introduces an RDMA manager as an intermediary component that facilitates direct memory access between nodes. The RDMA manager handles the setup and management of remote memory access operations, enabling efficient data transfer while bypassing the traditional resource-intensive operating system pathways.
2Productivity
If traditional Shuffle operation is used in Spark, then data can be written to disk and read across networks, but data throughput is reduced due to multiple copying layers
Solution Approach 1:
The patent removes the intermediate operating system file system and network stack layers from the data path by using RDMA technology. Data is transferred directly between node memories through a optimized path, eliminating the time-consuming copying operations that occur in traditional Shuffle processes.
Solution Approach 2:
The patent performs preliminary setup of RDMA connections and memory mappings before actual data transfer occurs. The RDMA manager pre-configures the direct memory access pathways, so that when data needs to be transferred during Shuffle operations, the optimized path is already in place, reducing overall processing time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention provides a system and a method for remote direct memory access based on Spark Shuffle. The system comprises a transmitting end and a receiving end; the transmitting end comprises a transmitter, a transmitting channel and a transmitting manager; the transmitter stores the acquired transmission data into the transmitting channel, and invokes the transmitting manager; after being invoked, the transmitting manager generates checking information based on the transmission data and stores the checking information into the transmitting channel; the receiving end comprises a receiver, a receiving channel and a receiving manager; the receiver acquires the transmission data and the checking information in the transmitting channel through a remote direct memory access technique, stores the transmission data and the checking information into the receiving channel, and invokes the receiving manager; after being invoked, the receiving manager verifies the transmission data based on the checking information.