Node Fault Recovery via Selective Hardware Reset
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Massively parallel supercomputers like the Blue Gene system face inefficiencies due to increased soft and hard failures, leading to system downtime and resource wastage, as failures in one node can render entire partitions unusable and require job abortion and restart.
Innovation Solution
A method and apparatus for fault recovery in a parallel computer system that uses a heartbeat monitor on a service node to detect node failures, allowing for selective resetting of non-network hardware portions of a failed compute node without interrupting executing jobs, enabling continued operation without restarting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a node failure occurs in a massively parallel computer system, then the system must ensure high reliability, but the entire partition becomes unusable and requires job abortion
Solution Approach 1:
The patent segments the compute node into distinct functional components: network hardware portion and remaining portion (processing unit, memory, etc.). The reset hardware can selectively reset only the network hardware portion while leaving the remaining portion operational, allowing the node to continue participating in job execution despite network-related faults. This segmentation enables partial recovery without complete node failure.
2Stability of the object's composition
If a node failure occurs, then system stability is compromised, but resetting the entire node causes job interruption and resource wastage
Solution Approach 1:
The reset hardware is designed to reset only specific portions of the compute node (network hardware portion) rather than the entire node. This selective reset capability stabilizes the system by containing failures to specific components while maintaining overall node functionality and job execution, thereby reducing system downtime and avoiding resource wastage.
Solution Approach 2:
Instead of performing a complete node reset (excessive action), the system performs a partial reset of only the network hardware portion. This partial action is sufficient to recover from network-related faults while avoiding the unnecessary interruption of job execution and minimization of system downtime that would result from a full reset.
3Reliability
If the network hardware portion is reset, then network connectivity is restored, but the entire node reset would interrupt executing jobs
Solution Approach 1:
The compute node is segmented into network hardware portion and remaining portion, with separate reset control. The reset hardware can independently reset the network hardware portion to restore network connectivity while leaving the remaining portion (including executing jobs) undisturbed. This segmentation enables targeted recovery without collateral disruption to job execution.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A method and apparatus for fault recovery of on a parallel computer system from a soft failure without endingan executing job on a partition of nodes. In preferred embodiments a failed hardware recovery mechanism on a service node uses a heartbeat monitor to determine when a node failure occurs. Where possible, the failed node is reset and re-loaded with software without ending the software job being executed by the partition containing the failed node.