MPI Fault Tolerance Mechanism for Cloud HPC Failure Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance computing (HPC) applications in cloud environments face challenges with fault tolerance due to issues like system virtualization, task submission, cloud data I/O, security, and reliability, particularly because current interconnection mechanisms in clouds are not optimal for HPC requirements, leading to inefficiencies in failure detection and recovery.
Innovation Solution
A failure recovery method is implemented using a Message Passing Interface (MPI)-Transmission Control Protocol (TCP) model that extends MPI programs with fault tolerance, allowing failed processes to respawn while the remaining processes continue on their original nodes, minimizing recovery costs and ensuring system reliability through collaborative failure management across hardware, virtual machine, and application layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud interconnection mechanisms (O-Gigabit Ethernet) are used, then system scalability and accessibility are improved, but communication performance and reliability for HPC applications deteriorate
Solution Approach 1:
The patent introduces an intermediary fault tolerance layer between the HPC application and the cloud infrastructure. This layer includes fault detection mechanisms, checkpointing systems, and recovery protocols that mediate between the application's reliability requirements and the cloud's inherent fault-prone nature, allowing the system to maintain high communication reliability despite using standard cloud interconnection mechanisms
Solution Approach 2:
The patent implements prior cushioning through checkpointing mechanisms that periodically save application state before potential failures occur. By establishing these checkpoints in advance and preparing recovery protocols beforehand, the system can quickly restore communication reliability after failures without compromising the use of scalable cloud interconnection mechanisms
2Reliability
If traditional checkpointing protocols are used for fault tolerance, then system reliability is improved, but recovery time and computational overhead increase
Solution Approach 1:
The patent applies partial action by implementing selective checkpointing that focuses only on critical application states and data that are essential for recovery. Instead of checkpointing the entire system state, the mechanism identifies and protects only the necessary portions, significantly reducing the time and computational overhead while maintaining adequate reliability
Solution Approach 2:
The patent segments the checkpointing process into multiple smaller, independent checkpoints rather than requiring a single comprehensive checkpoint. This segmentation allows for faster, more frequent checkpoints of critical components without the overhead of capturing the entire system state, thereby reducing recovery time while maintaining reliability
3Reliability
If full system respawn is used upon failure, then fault recovery is achieved, but system recovery costs and downtime increase
Solution Approach 1:
The patent extracts and isolates the fault tolerance mechanisms from the main application logic, allowing failed components to be independently identified and replaced without affecting the entire system. By separating fault-prone elements from stable elements, the system can recover from failures by respawning only the necessary components rather than the entire system, reducing recovery costs and downtime
Data Source
AI summary
A method of providing failure recovery capabilities to a cloud environment for scientific HPC applications. An HPC application with MPI implementation extends the class of MPI programs to embed the HPC application with various degrees of fault tolerance. An MPI fault tolerance mechanism realizes a recover-and-continue solution. If an error occurs, only failed processes re-spawn, the remaining living processes remain in their original processors/nodes, and system recovery costs are thus minimized.