Parallel Processing Node Partitioning for Fault-Tolerant Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel-processing systems lack effective fault-tolerance mechanisms, leading to significant overhead and data loss when computing nodes fail, as they require restarting the entire process, which undermines the speedup gains from parallel execution.
Innovation Solution
The system partitions computing nodes into primary and set-aside nodes, proactively monitoring health and transferring tasks from at-risk primary nodes to set-aside nodes, allowing continued execution and ensuring fault-tolerance without the need for frequent checkpointing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If checkpointing is performed frequently to improve fault-tolerance, then reliability improves, but execution time increases due to checkpointing overhead
Solution Approach 1:
The system proactively monitors health metrics of computing nodes and performs task replication to standby nodes before actual failures occur. This preliminary action prevents data loss without requiring frequent checkpointing, thus maintaining reliability while avoiding the time overhead of frequent checkpoints.
Solution Approach 2:
The system divides computing nodes into primary nodes (processing tasks) and standby nodes (receiving replicated tasks). This segmentation allows fault-tolerance to be handled separately from the main computation flow, enabling continuous execution on primary nodes while standby nodes prepare for potential failures, thereby reducing execution time overhead.
2Loss of time
If checkpointing is performed infrequently to reduce overhead, then execution time decreases, but data loss increases when failures occur
Solution Approach 1:
The system creates copies of tasks being processed by primary nodes and transfers them to standby nodes. This copying mechanism ensures that if a primary node fails, the standby node already has the necessary task data to continue execution, eliminating data loss without requiring frequent checkpointing and thus maintaining short execution time.
3Productivity
If all computing nodes are used for processing to maximize throughput, then productivity improves, but fault-tolerance capability deteriorates
Solution Approach 1:
Standby nodes are designed to be multi-functional: they can remain idle during normal operation, process replicated tasks from at-risk nodes, and take over fully when failures occur. This universality allows the system to maintain high throughput with all nodes potentially contributing to computation while ensuring fault-tolerance capability is preserved through the standby mechanism.
Solution Approach 2:
The system assigns different roles to different nodes based on their current state: primary nodes handle main computation while standby nodes monitor and receive replicated tasks. This local differentiation allows maximum throughput during normal operation while maintaining fault-tolerance readiness, with each node optimized for its specific function at any given time.
Data Source
AI summary
One embodiment of the present invention provides a system that enhances throughput and fault-tolerance in a parallel-processing system. During operation, the system first receives a task. Next, the system partitions N computing nodes into M set-aside nodes and N-M primary computing nodes, wherein M≧1. The system then processes the task in parallel across the N-M primary computing nodes. While doing so, the system proactively monitors the health of each of the N-M primary computing nodes. If the system detects a node in the N-M primary computing nodes to be at risk of failure, the system copies the portion of the task associated with the at-risk node to a subset of the M set-aside nodes. The system then processes the portion of the task in parallel across the subset of the M set-aside nodes while the N-M primary computing nodes continue executing.