Communication Proxy Fault Management in Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computer systems with multiple nodes lack fault tolerance, leading to potential failures in workload execution due to hardware or software faults, especially as the number of nodes increases, as existing communication libraries like MPI and SHMEM do not support resilient operations.
Innovation Solution
Implementing a fault management service with communication proxies that monitor node health, relaunch communication proxies, and roll back programs to a common checkpointed state in case of faults, isolating programs from non-fault-tolerant communication libraries and ensuring continued operation by maintaining synchronization across healthy nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of computer nodes is increased to improve computational capability, then productivity is improved, but reliability deteriorates due to increased likelihood of node faults
Solution Approach 1:
The system segments the communication library functionality into separate communication proxies deployed at each node. These proxies operate independently and can fail without affecting other nodes. The segmentation allows the system to isolate faults at the proxy level while maintaining overall system reliability, resolving the contradiction between scaling up nodes and maintaining reliability.
Solution Approach 2:
The system implements checkpointing mechanisms that allow communication proxies to be discarded upon fault detection and recovered by restarting from saved states. This enables the system to recover from node faults without losing computational progress, maintaining reliability while supporting large-scale deployments.
2Speed
If communication libraries like MPI and SHMEM are used to achieve high-performance communication, then speed is improved, but reliability deteriorates because these libraries do not support fault tolerance
Solution Approach 1:
Communication proxies serve as intermediary components between applications and traditional communication libraries. These proxies wrap the non-fault-tolerant MPI/Shmem calls and implement fault tolerance on top, allowing high-speed communication to coexist with reliability. The proxies act as mediators that isolate applications from the limitations of underlying communication libraries.
Solution Approach 2:
The system implements checkpointing and monitoring mechanisms that detect faults before they propagate through the entire system. By cushioning against potential failures through proactive monitoring and state preservation, the system maintains both high-speed communication capability and fault tolerance.
3Reliability
If fault management mechanisms are added to improve reliability, then reliability is improved, but device complexity increases
Solution Approach 1:
The communication proxies perform multiple functions: they handle communication operations, implement fault detection, manage checkpoints, and coordinate recovery. By making the proxies multi-functional, the system achieves fault tolerance without proportionally increasing complexity, as a single component handles multiple responsibilities.
Solution Approach 2:
The fault management system operates autonomously by monitoring its own state and automatically triggering recovery procedures when faults are detected. This self-service capability reduces the need for external intervention and simplifies the overall system architecture by embedding management functions within the components themselves.
Data Source
AI summary
In some examples, a distributed computer system includes a plurality of computer nodes, where the plurality of computer nodes include respective programs to cooperate to perform a workload. A first computer node includes a communication proxy between the program of the first computer node and a communication library that supports communications between the program of the first computer node and the programs of other computer nodes of the plurality of computer nodes, and a fault management service to monitor a health of the other computer nodes, and in response to a detection of a fault of a second computer node of the plurality of computer nodes, relaunch the communication proxy. The relaunched communication proxy selects, from a plurality of states, a common state to which the programs are to roll back.