Ring Allreduce Processing with Double Buffer Overlap
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The ring allreduce method in computing systems is not optimal in terms of network bandwidth utilization due to data dependency constraints, which prevent overlap of communication and computation steps, leading to inefficiencies in processing large messages.
Innovation Solution
Implementing a double buffer technique that splits messages into 2*N chunks, allowing each node to transmit one chunk while reducing another, thereby overlapping communication and computation steps and improving bandwidth utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the ring allreduce method is used with data dependency constraints, then the communication steps can be performed sequentially, but the network bandwidth utilization deteriorates due to no overlap between communication and computation
Solution Approach 1:
The patent divides the message into multiple chunks (specifically 2*N chunks where N is the number of nodes) and processes them in parallel. Each node works on different chunks simultaneously, allowing communication and computation to overlap. This segmentation resolves the contradiction by enabling parallel processing while maintaining data dependency constraints through structured chunk distribution.
Solution Approach 2:
The patent performs preliminary actions by pre-dividing the message into chunks and preparing the reduction operation schedule before actual execution. Nodes prepare their computation tasks in advance based on the predetermined chunk assignment, allowing communication and computation to overlap during execution without violating data dependencies.
2Reliability
If communication and computation steps are performed sequentially due to data dependencies, then data correctness is maintained, but the processing time increases due to no overlap
Solution Approach 1:
By segmenting the message into 2*N chunks and assigning different chunks to different nodes for simultaneous processing, the patent achieves time overlap between communication and computation. Node i processes chunk 2*i while receiving data for chunk 2*i+1, maintaining correctness through structured data flow while reducing total processing time.
Solution Approach 2:
The patent ensures continuous useful action by overlapping communication and computation operations. While one node is receiving data for a chunk, another node is simultaneously processing a different chunk, eliminating idle waiting time and maintaining continuous productive activity throughout the allreduce operation.
3Power
If only one core is dedicated to each MPI operation, then computing capability is preserved, but the network bandwidth utilization deteriorates due to limited parallel processing
Solution Approach 1:
The patent segments the MPI operation into multiple parallel chunk processing tasks that can be handled by different cores simultaneously. Each core processes different chunks of the message in parallel, increasing network bandwidth utilization while preserving overall computing capability through distributed processing.
Solution Approach 2:
The patent applies partial action by having each core handle only the specific chunks assigned to it rather than the entire message, allowing multiple cores to work in parallel on different portions of the data. This partial processing approach increases bandwidth utilization while each core maintains adequate computing capability for its assigned task.
Data Source
AI summary
A method of performing ring allreduce operations is disclosed. The method includes sending a chunk of a message in a receive buffer at a current index of a send buffer to a next node in a virtual ring of nodes, receiving a chunk of the message from a previous node in the virtual ring of nodes and store the chunk at the current index of the receive buffer, and reducing a chunk in a send buffer at a previous index of the receive buffer and a chunk in the receive buffer at a previous index of the receive buffer and storing a result at the previous index of the receive buffer. The method includes repeating the sending, receiving and storing, and reducing and storing steps until all chunks of the message are reduced, and sending reduced chunks to the next node and receive reduced chunks from the previous node.