NIC Failover Using Health Metrics and Backup Path Rerouting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-throughput, low-latency networks like RoCE and InfiniBand are vulnerable to failures such as packet loss, congestion, and NIC issues, leading to performance degradation and potential system crashes, especially in critical applications like AI tools and real-time data processing.
Innovation Solution
A system that monitors network health to detect payload failures and automatically switches data transmission to backup NICs without centralized control, using proximity and utilization metrics to select backup NICs and spoof progress to upstream applications, enabling spatial retransmission and faster failover.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If NICs operate at high throughput and low latency, then network performance is improved, but network failures and reliability issues worsen
Solution Approach 1:
The system proactively monitors NIC health metrics (temperature, error rates, link status) before failures occur and pre-identifies backup NICs. When degradation is detected, the system initiates failover to backup NICs before complete failure happens, maintaining network throughput while preventing service interruption.
Solution Approach 2:
The system dynamically changes operational parameters by switching from a single active NIC to a standby NIC when health parameters (temperature, error rates) exceed thresholds. This parameter change allows the system to maintain high throughput performance while avoiding failures associated with the degraded NIC.
2Reliability
If the system implements failover to backup NICs, then reliability is improved, but device complexity worsens
Solution Approach 1:
The system implements self-service failover by automatically monitoring NIC health metrics, detecting failures, selecting appropriate backup NICs, and switching data transmission without human intervention. This automation improves reliability while managing complexity through centralized control logic that handles the complexity internally rather than requiring external management.
Solution Approach 2:
The system continuously monitors NIC health metrics (temperature, error rates, link status) and uses this feedback to automatically trigger failover when thresholds are exceeded. This feedback mechanism enables reliable automatic failover while simplifying management by using objective metrics rather than complex manual assessment procedures.
3Reliability
If the system monitors NIC health metrics continuously, then reliability is improved, but use of energy worsens
Solution Approach 1:
The system performs periodic monitoring of NIC health metrics at intervals rather than continuously, checking temperature, error rates, and link status at scheduled times. This periodic approach maintains adequate failure detection capability while significantly reducing energy consumption compared to continuous monitoring.
Data Source
AI summary
A method or system for mitigating payload failures in a networked computing environment. A payload failure is detected at either a sender or receiver NIC based on one or more health metrics, such as timeouts, transmission errors, or latency anomalies. In response, the system spoofs progress to the upstream application by signaling that data transmission is continuing, thereby preventing disruption or termination of the application process. A backup NIC is selected from the available NICs at the sender or receiver host based on one or more efficiency metrics, including a proximity metric (e.g., NUMA locality) and a utilization metric (e.g., current traffic load). Data transmission is then rerouted to the selected backup NIC, maintaining operational continuity.