Distributed Computing Node Fault Domain Tagging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, faults in one computing node can propagate errors to shared memory and I/O resources across other nodes, posing a challenge in containing and managing these errors effectively due to shared resource configurations.
Innovation Solution
The implementation involves generating an alert indicator to trigger actions within the distributed system to restrict operations, halt I/O communications, and use the cluster interconnection fabric to tag suspect data packets, allowing for error containment and enabling failure analysis by communicating cached data to other nodes while preventing further data corruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If computing nodes share memory and I/O resources across the distributed system, then resource utilization and system efficiency are improved, but error propagation from faults in one node to other nodes increases
Solution Approach 1:
The system segments the distributed computing environment into isolated fault domains by implementing fault domain tags on computing nodes and their associated resources. When a fault is detected, the segmentation mechanism isolates the affected node and its resources from the rest of the system, preventing error propagation while maintaining resource sharing for healthy nodes. This is achieved through tagging mechanisms that associate resources with specific fault domains and filtering operations that prevent cross-domain error transmission.
Solution Approach 2:
The patent introduces intermediary components including fault domain tags, resource filters, and packet tagging mechanisms that act as mediators between computing nodes and shared resources. These intermediaries intercept and examine data packets and resource access requests, applying fault domain information to determine whether to allow or block transmission. This intermediary layer enables resource sharing while simultaneously preventing error propagation by filtering based on fault domain membership.
2Reliability
If communications with a faulty computing node are completely halted to prevent error propagation, then system reliability is improved, but the ability to perform failure analysis and diagnose the fault is reduced
Solution Approach 1:
The system applies different communication policies to different fault domains rather than implementing a blanket halt to all communications with faulty nodes. Healthy fault domains continue normal communications and resource access, while the faulty fault domain is isolated. This local quality approach allows failure analysis to proceed within the isolated fault domain using cached data and diagnostic tools, without affecting the broader system's operational reliability.
Solution Approach 2:
The system performs preliminary actions by caching data from the faulty computing node before complete isolation occurs and by preserving fault domain tag information for later analysis. The cached data and preserved metadata enable post-fault diagnostic activities to reconstruct and analyze the state of the faulty node, facilitating failure analysis even after communications are halted to prevent error propagation.
3Reliability
If fault domain tagging and resource filtering mechanisms are implemented to control error propagation, then error containment is improved, but system complexity and overhead increase
Solution Approach 1:
The fault domain tag structure is designed to be universal and multi-functional, serving multiple purposes within the distributed system. The same tagging mechanism enables error containment, resource filtering, packet routing decisions, and failure analysis all in a single unified framework. This universality reduces overall system complexity compared to implementing separate mechanisms for each function, as the fault domain tag becomes a central metadata element that drives multiple system behaviors simultaneously.
Data Source
AI summary
A technique includes receiving an alert indicator in a distributed computer system that includes a plurality of computing nodes coupled together by cluster interconnection fabric. The alert indicator indicates detection of a fault in a first computing node of the plurality of computing nodes. The technique indicates regulating communication between the first computing node and at least one of the other computing nodes in response to the alert indicator to contain error propagation due to the fault within the first computing node.