Network Adapter Completion Tracking for InfiniBand Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network request processing in large clusters, particularly with InfiniBand interconnects, faces significant latency and overhead due to the software overhead of managing completion queues and handling network congestion, which can lead to performance bottlenecks and potential system crashes.
Innovation Solution
A system where a network adapter buffers remote data transfer requests, sets a throttle flag to prevent further requests when congestion occurs, and uses completion groups with counters and flush registers to efficiently manage and track the status of network requests, thereby reducing latency and overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the network adapter uses a work queue in host memory to receive requests, then the programming interface is simple, but latency and overhead are added at the IB network adapter and host
Solution Approach 1:
The network adapter pre-allocates a dedicated buffer in its own memory (not host memory) to receive work requests. This preliminary preparation eliminates the need for the adapter to read requests from host memory, thereby reducing latency while maintaining the simple programming interface where applications just write requests to the buffer.
2Ease of operation
If adapter memory is partitioned amongst software threads, then each thread gets a small number of entries, but the total number of concurrent requests is limited
Solution Approach 1:
Instead of partitioning the adapter buffer amongst multiple software threads, the invention merges all requests into a single unified buffer that is shared by all threads. This allows the total number of concurrent requests to be maximized while maintaining thread-safety through proper synchronization mechanisms on the shared buffer.
3Reliability
If the host CPU manages the work queue and tracks available space, then request processing is coordinated, but software overhead increases
Solution Approach 1:
The invention extracts the work queue management function from the host CPU and relocates it to the network adapter itself. The adapter maintains its own buffer and independently manages request processing, thereby reducing the software overhead on the host CPU while maintaining coordinated queue management through standardized interface protocols.
4Reliability
If completion records are written to a completion queue in system memory, then request completion is notified, but software overhead for reaping completions increases
Solution Approach 1:
The network adapter uses completion groups where the adapter itself tracks and manages completion status without requiring host software to actively reap completions. The adapter autonomously handles completion notifications and status tracking, significantly reducing the software overhead and time required for completion processing while maintaining reliable completion notification.
Data Source
AI summary
Techniques for tracking completion of transfer requests. In one embodiment, a compute node connects to a network adapter (NA). In one embodiment, software running on the compute node contains instructions in which some remote data transfer requests belong to (or are associated with) completion groups. These completion groups may be constructed so that the system may more efficiently determine the completion status of remote transfer requests. In one embodiment, The NA includes a hardware counter for each completion group (CG). In one embodiment, the counter is configured to count when each transfer request in the completion group is received and when each request in the completion group is completed. For example, the counter may increment on receipt and decrement on completion such that a zero indicates completion of all requests in the completion group. In one embodiment, the NA includes a flush register to indicate whether the counter is valid.