One-Sided Communication with NIC Flush for Ordered Data Delivery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing one-sided communication techniques in distributed systems incur additional delays due to waiting for delivery-complete acknowledgments, which accumulate and adversely impact performance, especially in large-scale distributed operations.
Innovation Solution
Implementing transmission-complete acknowledgments and flush instructions to speed up data delivery, where the target NIC sends acknowledgments upon completing data transmission without waiting for delivery to the destination buffer, and using a flush instruction to ensure all data elements are delivered in order.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the target NIC waits for delivery to the destination buffer before sending acknowledgments, then data delivery completeness is ensured, but transmission delays accumulate and performance deteriorates
Solution Approach 1:
The target NIC sends transmission-complete acknowledgments before the data is fully delivered to the destination buffer. This preliminary action allows the source process to continue sending data without waiting for acknowledgment, reducing transmission delays while maintaining reliability through subsequent flush operations that ensure ordered delivery.
2Stability of the object's composition
If the source process waits for delivery-complete acknowledgments before sending the next data element, then data ordering is maintained, but transmission speed decreases
Solution Approach 1:
The system sends transmission-complete acknowledgments preliminarily before actual buffer delivery is complete. This allows the source process to send subsequent data elements without waiting, achieving pipelining effect that increases transmission speed while flush instructions later ensure proper ordering.
Solution Approach 2:
The acknowledgment process is segmented into two types: transmission-complete acknowledgments sent immediately to enable fast retransmission and pipelining, and delivery-complete acknowledgments sent later to ensure ordering. This segmentation allows different phases of data transfer to operate with different acknowledgment strategies.
3Productivity
If multiple data elements are sent simultaneously to multiple target processes, then productivity increases, but incast congestion occurs and performance degrades
Solution Approach 1:
The source process periodically sends flush instructions to target NICs to ensure ordered delivery of data elements. This periodic flushing action prevents incast congestion by controlling the rate at which data is delivered to destination buffers, while still allowing high-speed transmission in between flush operations.
Data Source
AI summary
A computing node in a distributed system is provided. During operation, the node can send, from a source process using direct memory transfer, a first set of data elements from a source buffer to a NIC of a target node executing a target process. These processes may be among a plurality of processes performing a distributed operation across a set of nodes. Subsequent to sending the first set of data elements, the node can receive respective acknowledgments for the first set of data elements from the NIC. The node can then send, to the NIC of the target node, a flush instruction to trigger the delivery of pending data elements to a destination buffer of the target process. The node can send, from the source process, a control message to the target process indicating that the first set of data elements is available to the target process.