Worker Node Rebuild via Communication Agents in Distributed Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale parallel processing systems, the non-work-conserving detect/restart model used for fault tolerance reduces the number of worker nodes, leading to decreased system throughput when a worker node fails, as the failed node is removed without replacement, thereby reducing the overall system efficiency.
Innovation Solution
The implementation of communication agents in both worker and redundant nodes to detect failures, spawn new worker nodes from redundant nodes, and rebuild them to maintain system performance, ensuring that the number of active worker nodes remains constant and communication between nodes is uninterrupted.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a non-work-conserving detect/restart model is used for fault tolerance, then system reliability is improved, but system throughput deteriorates due to reduction in worker node count
Solution Approach 1:
The system pre-creates redundant worker nodes before failures occur. When a worker node fails, a pre-existing redundant node is immediately activated and rebuilt to replace it, rather than creating a replacement after the failure. This preliminary preparation eliminates the throughput degradation that would otherwise occur during node replacement.
Solution Approach 2:
The system recovers redundant worker nodes that were previously created but not yet activated. When a failure occurs, instead of discarding the redundant node, the system reuses and rebuilds it to replace the failed worker, thereby maintaining the total worker node count and preserving system throughput while ensuring fault tolerance.
2Device complexity
If failed worker nodes are removed without replacement, then system complexity is reduced, but system efficiency deteriorates
Solution Approach 1:
The system implements self-service through automated detection, rebuilding, and replacement of failed worker nodes using communication agents. The master node automatically manages the replacement process by detecting failures and activating redundant nodes without manual intervention, maintaining system efficiency while keeping node management complexity manageable through automation.
Solution Approach 2:
The system uses communication agents that provide continuous feedback about worker node status to the master node. This feedback mechanism enables automatic detection of failures and triggers the rebuilding process, allowing the system to maintain efficiency through automated responses while managing complexity through structured communication protocols.
3Device complexity
If the number of worker nodes is reduced after failure, then resource allocation is simplified, but system capacity deteriorates
Solution Approach 1:
The system pre-allocates and creates redundant worker nodes before failures occur, maintaining a pool of available nodes. When a failure happens, the redundant node is immediately activated to replace the failed one, ensuring the total number of active nodes remains constant and system capacity is preserved without complex reallocation decisions.
Solution Approach 2:
The system changes the status parameter of redundant nodes from 'inactive' to 'active' when replacements are needed, rather than creating entirely new nodes or removing failed ones. This parameter change approach maintains the total node count and system capacity while simplifying resource management through state transitions.
Data Source
AI summary
A method includes creating communication agents in each worker node in a distributed processing cluster, creating communication agents in nodes other than the distributed processing cluster as redundant nodes, the worker nodes and redundant nodes forming a communicator group, numbering and identifying each worker node in the distributed processing cluster using the communication agents, detecting a distributed processing worker node fault using the communication agent between nodes of the distributed processing cluster, spawning a new worker node via the communication agent from one of the redundant nodes in the communicator group for use as a new worker node, rebuilding the new worker node by using the communication agent to operate with the distributed processing cluster, and starting the new worker node to join the distributed processing cluster to work with the other worker nodes utilizing the communication agents for communications between the worker nodes in the communicator group.