Distributed Node Fault Containment via Fabric Awareness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, faults in one computing node can propagate errors to shared memory and I/O resources across other nodes, posing a challenge in containing and managing errors effectively due to the shared nature of resources.
Innovation Solution
Implementing a system where an alert indicator is generated upon detecting a fault, triggering actions to restrict operations and communication within the cluster interconnection fabric to halt outbound I/O communications, tag suspect data packets, and allow failure analysis by communicating cached data to other nodes, thereby containing errors within the affected node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If computing nodes share memory and I/O resources across the cluster, then resource utilization and system efficiency are improved, but error propagation from faults in one node to other nodes increases
Solution Approach 1:
The system segments the fault domain by introducing fault domain boundaries that isolate errors to specific computing nodes or resource groups. When a fault is detected in one node, the system contains the error within that node's fault domain, preventing propagation to other nodes while maintaining resource sharing capabilities within safe boundaries.
Solution Approach 2:
The system introduces intermediary components including fabric awareness modules in I/O adapters and computing nodes that act as mediators between faulty nodes and the rest of the system. These intermediaries detect faults, generate alerts, and coordinate error containment actions, enabling resource sharing while blocking error propagation paths.
2Reliability
If I/O transactions are restricted to contain errors, then system reliability is improved, but productivity and resource utilization deteriorate
Solution Approach 1:
The system applies local quality by restricting I/O transactions only to the extent necessary for error containment - specifically blocking transactions that would propagate errors from faulty nodes while allowing normal transactions to continue between healthy nodes. The fabric awareness module selectively blocks or allows transactions based on the specific fault condition and target node status.
Solution Approach 2:
The system implements partial action by applying error containment measures only to affected fault domains rather than the entire system. When a fault is detected, the system generates alerts and restricts transactions only involving the faulty node or resource, leaving other parts of the system fully operational and maintaining overall productivity.
3Reliability
If communication with faulty nodes is halted immediately, then error propagation is prevented, but failure analysis capability is reduced
Solution Approach 1:
The system performs preliminary action by generating fault alerts and tagging data packets from faulty nodes before fully halting communication. This allows the system to prepare for error containment while maintaining limited communication channels necessary for failure analysis. The alert mechanism enables proactive preparation of containment measures while preserving diagnostic capabilities.
Solution Approach 2:
The system implements feedback mechanisms where fabric awareness modules continuously monitor I/O transactions and fault conditions, generating alerts that trigger appropriate containment actions. The system provides feedback about fault status and containment effectiveness, enabling dynamic adjustment of communication restrictions based on real-time system state and analysis needs.
Data Source
AI summary
A technique includes receiving an alert indicator in a distributed computer system that includes a plurality of computing nodes coupled together by cluster interconnection fabric. The alert indicator indicates detection of a fault in a first computing node of the plurality of computing nodes. The technique indicates regulating communication between the first computing node and at least one of the other computing nodes in response to the alert indicator to contain error propagation due to the fault within the first computing node.