Vector Send Operation for MPI Collective Communication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Collective communication operations in multi-processing systems face bottlenecks due to high hardware and software overheads, particularly when sending small messages, which increases overall time for operations like all-to-all, broadcast, and barrier messages.
Innovation Solution
Implementing a vector send operation that accepts a vector of destination addresses and data buffers, reserving receive queues for each destination, and using DMA engines to transmit multiple commands simultaneously, reducing software setup costs and enabling multicast functionality without specialized hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional I/O operations with CPU involvement are used for each transaction, then data transfer can be performed, but the CPU becomes fully occupied and unavailable for other work, reducing overall system productivity
Solution Approach 1:
The patent segments the data transfer function from CPU operations by introducing DMA controllers that independently handle I/O transactions. The system divides responsibilities between CPU (higher-level control) and DMA controllers (lower-level data movement), allowing parallel execution of CPU computations and data transfers without mutual interference
Solution Approach 2:
The patent introduces DMA controllers as intermediary hardware components that mediate between memory and I/O devices. These controllers act as autonomous intermediaries that execute transfer operations without requiring continuous CPU intervention, freeing the CPU to perform other productive work while maintaining reliable data transfer completion
2Productivity
If DMA is used to bypass CPU for data transfer, then CPU availability improves, but hardware overhead and software setup costs increase, particularly for collective communication operations
Solution Approach 1:
The patent merges multiple individual DMA transfer operations into a single collective operation by introducing aggregation logic that combines destination addresses and data buffers into unified transfer commands. This reduces the number of separate software setup calls and hardware transaction initiations required for multicast operations
Solution Approach 2:
The patent creates universal data structures (destination vectors, buffer descriptors) that can handle both point-to-point and collective communication operations through a single interface. This multi-functional approach allows the same DMA infrastructure to efficiently support various communication patterns without requiring separate specialized hardware or software paths
3Adaptability or versatility
If multiple individual send operations are used for collective communication, then data can be sent to multiple destinations, but software overhead accumulates with each message, increasing total operation time
Solution Approach 1:
The patent performs preliminary actions by pre-organizing destination addresses into vectors and preparing buffer descriptors before initiating DMA transfers. This advance preparation consolidates software setup work into a single operation rather than repeating it for each destination, significantly reducing cumulative software overhead for multicast operations
Solution Approach 2:
The patent enables continuous useful action by allowing DMA controllers to execute multiple transfer operations in parallel streams without software intervention between them. Once the collective send operation is initiated with aggregated parameters, the DMA hardware continuously performs transfers to multiple destinations simultaneously, eliminating idle software processing time
Data Source
AI summary
Methods and systems for conducting vector send operations are provided. The processor of a sender node receives a request to perform a collective send operation (e.g., MPI_Broadcast) from a user application, requesting a copy of data in one or more send buffers by sent to each of a plurality of destinations in a destination vector. The processor invokes a vector send operation from a software communications library, placing a remote enqueue atomic send command for each destination node of the destination vector in an entry of a transmit data mover (XDM) command queue in a single call. The processor executes all of the commands in the XDM command queue and writes the data in the one or more send buffers into each receive queue of each destination identified in the destination vector.