Ingress Edge Flush Trapping for GPU Interconnect Congestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High bandwidth interconnects in multi-GPU systems face performance issues due to 'flush storms' caused by the transmission of flush requests on multiple paths and endpoints, leading to network congestion and reduced synchronization performance.
Innovation Solution
Implementing a method that traps flush requests at the ingress edge of the switch fabric, converts posted transactions to non-posted transactions, and monitors acknowledgments to ensure precise flush operations, thereby reducing unnecessary network traffic and congestion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If flush requests are broadcast to every possible path and endpoint to ensure precise flush operations, then flush synchronization precision is improved, but network congestion and performance deteriorate due to flush storms
Solution Approach 1:
The patent applies local quality by making the flush request handling behavior dependent on the transaction type. Posted transactions are tracked locally at the ingress edge without broadcasting flush requests, while non-posted transactions receive full flush broadcasting. This localized differentiation resolves the contradiction by applying precise flush operations only where necessary (non-posted transactions) while avoiding unnecessary network traffic for posted transactions.
Solution Approach 2:
The patent changes the parameter of flush request propagation by introducing a condition based on transaction type. Instead of universally broadcasting flush requests to all endpoints, the system modifies the propagation behavior: tracked posted transactions suppress flush broadcasting, while untracked non-posted transactions trigger full broadcasting. This parameter change resolves the contradiction between precision and performance.
2Speed
If posted transactions are used for high-speed data transfer, then transfer speed is improved, but synchronization precision deteriorates because posted transactions do not require acknowledgments
Solution Approach 1:
The patent introduces feedback mechanisms for posted transactions by tracking them at the ingress edge and generating flush requests based on their completion status. The system monitors whether posted transactions have been completed and uses this feedback information to determine when to issue flush requests, thereby achieving synchronization precision while maintaining the high-speed advantage of posted transactions.
Solution Approach 2:
The patent introduces an intermediary tracking mechanism between the posted transaction sender and receiver. The ingress edge acts as an intermediary that tracks posted transactions, converts them to non-posted transactions with acknowledgment requirements, and manages the flush operation timing. This intermediary layer provides the missing synchronization precision without sacrificing the speed benefits of posted transactions.
3Measurement precision
If flush requests are converted to non-posted transactions with acknowledgment tracking, then synchronization precision is improved, but network overhead increases
Solution Approach 1:
The patent applies partial action by selectively converting only posted transactions to non-posted transactions with acknowledgment tracking, rather than converting all transactions. This partial conversion approach achieves the necessary synchronization precision for posted transactions while avoiding the network overhead of acknowledging every single transaction, thus resolving the contradiction between precision and traffic volume.
Data Source
AI summary
Systems and techniques for synchronizing transactions between processing devices on an interconnection network are provided. Upon receiving a stream of posted transactions followed by a flush transaction from a source processing device connected to the interconnection network, the flush transaction is trapped before it enters the interconnecting network. Subsequently, based on monitoring for responses received from a destination processing device for transactions corresponding to the posted transactions, a flush response is generated and returned to the source processing device. The described techniques enable efficient synchronizing posted writes, posted atomics and the like over complex interconnection fabrics such that a first GPU can write data to a second GPU so that a third GPU can safely consume the data written to the second GPU.