Selective Message Aggregation in HPC Collective Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale high-performance computing (HPC) distributed applications, existing protocols struggle to efficiently manage collective communications, particularly in all-to-all-v and all-to-all-w operations, where message sizes vary, making it difficult to decide whether to aggregate or transmit messages directly, leading to inefficiencies in communication bandwidth and latency.
Innovation Solution
The method involves dynamically splitting data exchanges between direct exchange and aggregation schemes based on message size, where messages larger than a predefined threshold are transmitted directly and smaller messages are aggregated within sub-groups before being sent to their destinations, reducing the number of forwarding steps and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If message aggregation is applied to all messages in large-scale HPC collective operations, then the number of messages transmitted is reduced, but the aggregation overhead and latency increase significantly
Solution Approach 1:
The group of processes is divided into multiple sub-groups, and messages are aggregated within each sub-group rather than across the entire group. This segmentation reduces the aggregation overhead and latency by limiting the scope of aggregation to smaller subsets of processes, while still achieving message reduction benefits.
Solution Approach 2:
Different aggregation strategies are applied based on local conditions: small messages within sub-groups are aggregated, while large messages are transmitted directly. This local quality approach optimizes the trade-off between message reduction and aggregation overhead by applying aggregation selectively rather than uniformly across all messages and all groups.
2Productivity
If selective aggregation based on message size is implemented, then communication efficiency is improved, but the complexity of message routing and aggregation logic increases
Solution Approach 1:
A threshold parameter is introduced to automatically determine whether to aggregate or transmit messages directly based on message size. This parameter-based approach simplifies the routing logic compared to complex decision-making algorithms, while still achieving communication efficiency improvements by aggregating only small messages that benefit from aggregation.
3Loss of time
If all messages are transmitted directly without aggregation in large groups, then latency is reduced, but communication bandwidth is wasted due to redundant transmission of small messages
Solution Approach 1:
Aggregation is applied partially rather than completely - only to small messages below a threshold size and only within sub-groups. This partial action approach captures the bandwidth savings from aggregating small messages while avoiding the excessive latency that would result from aggregating all messages, thus optimizing the trade-off between bandwidth utilization and transmission latency.
Data Source
AI summary
A method for collective communications includes invoking a collective operation over a group of computing processes in which the processes in the group concurrently transmit and receive data messages to and from other processes in the group via a communication medium. The processes detect respective sizes of the data messages and transmit the data messages for which the respective sizes are greater than a predefined threshold to respective destination processes in the group without aggregation. The data messages for which the respective sizes are less than the predefined threshold are aggregated, and the aggregated data messages are transmitted to the respective destination processes.