Collective Data Transfer With Single-Step Metadata Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing collective communication methods in high-performance computing (HPC) face inefficiencies due to uncertainty in message sizes and the need for two-stage data exchange operations, leading to increased communication load and latency.
Innovation Solution
A method for collective communications that combines metadata and data transmission in a single operation, using predefined thresholds to determine whether to transmit full messages or only metadata, and employing message aggregation for efficient data handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If message aggregation protocols are used to reduce communication burden, then communication load is reduced, but latency increases due to two-stage data exchange operations
Solution Approach 1:
The patent combines metadata transmission and data transmission into a single unified operation, eliminating the two-stage exchange process. Source processes transmit both metadata (containing data handling instructions) and actual data simultaneously to destination processes, thereby reducing latency while maintaining the communication load reduction benefits of aggregation protocols
Solution Approach 2:
The patent transmits metadata containing data handling instructions in advance within the same communication operation as the data itself. This allows destination processes to prepare for incoming data with proper buffer allocation and processing instructions already in place, eliminating the need for a separate metadata exchange stage and reducing overall latency
2Reliability
If all-to-all-v and all-to-all-w operations are implemented naively with each process transmitting messages to all other processes, then data exchange completeness is ensured, but communication overhead increases significantly
Solution Approach 1:
The patent extracts the data handling instructions (metadata) from the actual data transmission process and transmits them separately as accompanying information. This allows destination processes to understand how to handle incoming data without requiring additional communication rounds, maintaining completeness while reducing overhead
Solution Approach 2:
The patent introduces metadata as an intermediary element that carries instructions about how to process incoming data. This metadata acts as a mediator between source and destination processes, enabling efficient data handling without requiring complex communication protocols or additional message exchanges
Data Source
AI summary
A method for data communications includes composing, for concurrent transmission by multiple source processes in a group of computing processes, messages including metadata indicating how data to be transmitted by the source processes are to be handled by destination processes in the group that are to receive the data and also including in some of the messages the data to be transmitted by one or more of the source processes to one or more of the destination processes, while others of the messages include only the metadata. The composed messages are transmitted concurrently from the source processes to the destination processes in the group over a communication medium. The data are processed by the destination processes in response to the metadata included in the messages received by the destination processes.