PCIe Peer-to-Peer RDMA Access for Processor Add-in Cards
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current interconnectivity fabric standards, such as InfiniBand and RDMA Consortium specifications, do not enable peer devices to directly access each other's address space, leading to increased message latency and reduced bandwidth due to the need for data copies through common system memory.
Innovation Solution
Implementing PCIe hardware to enable peer-to-peer data transfers between processor add-in cards and RDMA devices, allowing direct access to RDMA device memory without involving host memory, and using logic and modules to map memory and registers for concurrent I/O operations with user-mode applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If peer devices use common system memory for data transfers, then data access is enabled, but message latency increases and bandwidth decreases
Solution Approach 1:
The patent extracts the data transfer path from the common system memory and creates a direct peer-to-peer connection between RDMA devices. By removing the intermediate system memory stop, data transfers occur directly between devices, eliminating the time penalty and bandwidth loss associated with writing to and reading from common memory.
Solution Approach 2:
The PCIe fabric acts as an intermediary that enables direct device-to-device communication without requiring system memory as the mediator. The fabric provides the communication pathway that allows peer devices to exchange data directly, bypassing the traditional route through system memory while maintaining protocol compatibility.
2Productivity
If peer devices use common system memory for data transfers, then data access is enabled, but processing operations slow down
Solution Approach 1:
The patent removes the bottleneck of system memory involvement from the data transfer path. By extracting this intermediate step, processing operations can proceed directly between peer devices without the time-consuming cycles required to write to and read from system memory, thereby improving overall processing throughput.
Solution Approach 2:
Peer devices perform data transfers autonomously without requiring system memory or host CPU intervention. The RDMA devices independently manage their own data exchange through the PCIe fabric, enabling self-service communication that improves processing efficiency by eliminating external memory access overhead.
3Speed
If direct peer-to-peer access is implemented, then message latency is reduced and bandwidth is increased, but system complexity increases
Solution Approach 1:
The PCIe fabric serves multiple functions: it provides the physical connection for peer-to-peer transfers, implements the address translation mechanism, and maintains compatibility with existing RDMA protocols. By making the fabric universal and multi-functional, the system achieves direct access capabilities without proportionally increasing overall system complexity.
Solution Approach 2:
The PCIe fabric acts as an intelligent intermediary that handles the complexity of direct peer-to-peer communication. It provides address translation, request routing, and protocol management, thereby enabling fast direct access while containing complexity within the fabric layer rather than requiring complex changes to the RDMA devices themselves.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the invention describe systems, apparatuses and methods that enable sharing Remote Direct Memory Access (RDMA) device hardware between a host and a peripheral device including a CPU and memory complex (alternatively referred to herein as a processor add-in card). Embodiments of the invention utilize interconnect hardware such as Peripheral Component Interconnect express (PCIe) hardware for peer-to-peer data transfers between processor add-in cards and RDMA devices. A host system may include modules or logic to map memory and registers to and/or from the RDMA device, thereby enabling I/O to be performed directly to and from user-mode applications on the processor add-in card, concurrently with host system I/O operations.