FPGA Protocol Stack for Low-Latency PCIe Communication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current communication methods between multiple FPGA devices in a server environment suffer from high latency due to memory duplication operations and memory refreshing issues, which are exacerbated by the need for massive physical address spaces when establishing multiple connections.
Innovation Solution
A protocol stack is implemented in each FPGA device to support direct communication between applications, using a PCIe interface with an added FPGA Connect (FC) header in Transaction Layer Packets (TLPs) to differentiate connections and minimize physical address usage, allowing for efficient data transmission without memory duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If memory duplication operations are used for communication between FPGA devices, then data transmission can be achieved, but communication latency increases
Solution Approach 1:
The patent extracts the communication function from the memory subsystem by implementing a direct PCIe communication path between FPGAs. The protocol stack processes communication requests at the application layer and transmits data directly through PCIe interfaces, bypassing the memory duplication operations that previously caused latency. This separation of communication functionality from memory operations resolves the contradiction by enabling reliable data transmission without the time penalty of memory access.
Solution Approach 2:
The patent introduces a protocol stack as an intermediary layer between applications and the PCIe interface. This protocol stack manages connection establishment, data packaging with FC headers, and transmission coordination, enabling reliable communication while maintaining low latency. The intermediary handles the complexity of direct PCIe communication, allowing applications to communicate efficiently without directly managing the low-level transmission details.
2Adaptability or versatility
If multiple connections are established between FPGA devices, then communication versatility improves, but physical address space requirements increase
Solution Approach 1:
The patent adds a new dimension to connection identification by introducing connection identifiers (connection IDs) in the FC header, moving away from relying solely on physical address space to differentiate connections. Instead of using different physical addresses for each connection, the system uses the same physical address space but distinguishes connections through the connection ID field in the protocol header, effectively adding an organizational dimension to connection management.
Solution Approach 2:
The protocol stack implements a universal connection management mechanism that handles multiple connections through a single PCIe interface using connection identifiers. The same physical resources (PCIe lanes, buffers) can be shared across multiple connections by identifying them through connection IDs rather than requiring dedicated physical address spaces for each connection, making the system more versatile while conserving physical address space.
3Loss of time
If direct PCIe communication is implemented between FPGAs, then communication latency is reduced, but protocol complexity increases
Solution Approach 1:
The patent segments the communication protocol into distinct layers: a transport layer handled by the protocol stack (connection management, FC header packaging) and the underlying PCIe layer (physical transmission). This segmentation allows the protocol stack to focus on application-level communication tasks while leveraging the standardized PCIe layer for reliable physical transmission, managing complexity through clear separation of concerns.
Solution Approach 2:
The protocol stack implements self-service mechanisms for connection management, including automatic connection establishment, data packaging with appropriate FC headers, and transmission coordination. The system manages its own communication state and resources without requiring external control, reducing the burden on applications while maintaining low latency through efficient autonomous protocol handling.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The implementations of the subject matter described herein relate to communication between field programmable gate arrays. In some implementations, an FPGA device comprises a first protocol stack configured to: receive, from a source application, a data transmitting request for a destination application; package the data transmitting request into a first packet by adding a header to the data transmitting request, the header indicating the source application and the destination application; and transmit a physical address of a second protocol stack connected with the destination application. The FPGA device further comprises a PCIe interface configured to: package the first packet into a second packet based on the physical address of the second protocol stack received from the first protocol stack so that the first packet serves as a data portion of the second packet, the second packet being a TLP conforming to the PCIe standard; and transmit the second packet.