PCIe Interconnect Redundancy with Ghost Bridging for Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing PCIe systems suffer from single root port limitations that lead to catastrophic connectivity loss and disruptive recovery procedures, particularly in high-performance applications like ML and AI, due to uncorrectable errors causing system reboot and substantial operational overheads.
Innovation Solution
Implement a fully redundant system with an accelerated compute fabric (ACF) that bifurcates PCIe links into multiple paths, using ghost bridging and scatter-gather lists to maintain bandwidth and enable seamless failover without performance penalties.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single root port structure is used to connect endpoint devices to the host, then the PCIe architecture remains simple and scalable, but a single error causes loss of connectivity to all devices in the PCIe hierarchy
Solution Approach 1:
The PCIe link is segmented into multiple independent paths (first PCIe link and second PCIe link) between the host and endpoint device. Each path operates independently, so an error in one path does not affect the other. The system can switch between paths using ghost bridging when one path fails, maintaining connectivity reliability while keeping the overall architecture manageable through structured segmentation.
Solution Approach 2:
Ghost endpoint devices are introduced as intermediary virtual representations of the physical endpoint device. These ghost devices enable the host to maintain communication channels through multiple PCIe links even when one physical link fails. The ghost bridging mechanism acts as a mediator that redirects traffic from failed paths to operational paths, improving reliability without requiring complex hardware redundancy.
2Reliability
If redundant PCIe paths are implemented to provide error robustness, then connectivity reliability improves, but system complexity and overhead increase
Solution Approach 1:
Virtual ghost copies of the endpoint device are created for each PCIe link path. These ghost devices are software/firmware constructs that mirror the physical endpoint's identification and communication protocols. By using copying rather than physical duplication, the system achieves redundancy through virtualization, maintaining reliability while minimizing hardware complexity and overhead.
Solution Approach 2:
The ghost bridging mechanism provides multi-functionality by serving as both a virtual device representation and a failover control mechanism. The same ghost device structure enables normal communication during operation and automatically facilitates path switching when failures occur, reducing the need for separate dedicated failover hardware or software components.
3Reliability
If traditional PCIe error recovery procedures are implemented, then error recovery is achieved, but system operations are disrupted and performance is degraded
Solution Approach 1:
Multiple PCIe communication paths and ghost device representations are established in advance before failures occur. When an error is detected, the system can immediately switch to a pre-configured alternative path without needing to reinitialize or reenumerate devices. This preliminary preparation eliminates disruptive recovery procedures and maintains continuous data flow, preserving productivity during error recovery.
Solution Approach 2:
The ghost bridging mechanism ensures continuous data transmission by maintaining active communication channels through multiple paths. When one PCIe link fails, traffic is seamlessly redirected through alternative paths without interrupting the overall data flow. This continuity of useful action prevents the stop-start nature of traditional error recovery, maintaining system throughput and productivity during error conditions.
Data Source
AI summary
A system and method for achieving peripheral component interconnect express (PCIe) redundancy and recovery are disclosed. In some embodiments, the system comprises an accelerated compute fabric (ACF) comprising a PCIe switch, an application host communicatively coupled to the ACF using one or more upstream PCIe links, and an endpoint device communicatively coupled to the ACF using one or more downstream PCIe links. The application host is configured to send PCIe transaction layer packets (TLP) addressed to the ghosted endpoint devices through the one or more upstream PCIe links, and the ACF is configured to redirect the PCIe TLP packets to the endpoint device through the one or more downstream PCIe links.