Shared WQE Memory in Network Interfaces for QP Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Network interface devices face scalability issues when processing a large number of queue pair (QP) connections, leading to bandwidth degradation due to the trade-off relationship between the number of connections and bandwidth, as seen in commercial devices like Nvidia ConnectX-6, which experiences performance drops beyond 256 QP connections.
Innovation Solution
A network interface device with a shared work queue element (WQE) memory and a WQE prefetcher, along with a doorbell controller and response pipeline, allows simultaneous processing of multiple QP connections while maintaining high bandwidth and low latency, supporting up to 10,000 QP connections with minimal CPU intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of QP connections is increased to support more network connections, then the scalability and number of supported connections are improved, but the bandwidth drops sharply causing performance degradation
Solution Approach 1:
The patent segments the WQE storage into a shared WQE memory that is divided into multiple regions, each region being shared by multiple QP connections. This allows many QP connections to access a pooled memory resource rather than each connection requiring dedicated memory, enabling support for thousands of QP connections while maintaining efficient memory utilization and high bandwidth performance.
2Adaptability or versatility
If a shared WQE memory is implemented to support many QP connections, then the scalability is improved, but the memory management complexity increases
Solution Approach 1:
The patent implements a self-service memory management mechanism where the shared WQE memory automatically tracks free regions and allocates WQE storage space to different QP connections as needed. The doorbell controller monitors memory usage and manages allocation without requiring complex external intervention, simplifying the overall system architecture while supporting high scalability.
3Device complexity
If the WQE memory size is reduced to lower device complexity, then the manufacturing cost and device simplicity are improved, but the ability to handle multiple QP connections simultaneously is worsened
Solution Approach 1:
The patent creates a universal shared WQE memory that serves multiple QP connections simultaneously through region-based allocation. This single memory structure performs the function of multiple dedicated memory blocks, allowing the system to support a large number of QP connections with a relatively small total memory size, thereby achieving high scalability without proportionally increasing memory capacity or device complexity.
Data Source
AI summary
A network interface device is configured for simultaneously processing Y (where Y is a positive integer of 2 or more) queue pair (QP) connections for X (where X is a positive integer) send work queue elements (WQEs) or receive WQEs. The network interface device includes: a shared WQE memory having a size corresponding to N (a positive integer less than or equal to 1/n of X*Y, where n is a positive integer of 2 or more) send WQEs; and a first doorbell controller configured to count free regions in the shared WQE memory and control processing of a first doorbell received from an external source.