Workload Reassignment Using Generation IDs After Node Failure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In clustered environments, when a communication failure occurs between a master node and a worker node, the worker node may continue processing a workload without the master's knowledge, leading to potential performance issues and interruptions as the workload is reassigned to a new node while the old node continues processing, causing conflicts and delayed processing.
Innovation Solution
Implementing a generation identifier and workload identifier system that allows the storage system to recognize workload reassignment, ensuring the old node stops processing or continues without interruption by using higher or lower priority identifiers to manage access requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the master node reassigns the workload to a new node after communication failure, then the system maintains high availability and fault tolerance, but the old node may continue processing the workload causing conflicts and delayed processing
Solution Approach 1:
The storage system provides feedback to worker nodes about workload reassignment status by comparing generation identifiers. When a worker node's generation identifier does not match the current one stored in the storage system, it receives feedback that the workload has been reassigned, allowing it to stop processing and avoid conflicts.
Solution Approach 2:
The storage system acts as an intermediary between the master node and worker nodes. It stores generation identifiers and uses them to mediate the reassignment process, notifying worker nodes of reassignment status without requiring direct communication between the master node and worker nodes during failure scenarios.
2Duration of action of moving object
If the worker node continues processing the workload without master node knowledge, then processing may complete successfully, but conflicts arise when the master node reassigns the workload to a different node
Solution Approach 1:
The system uses generation identifier feedback to inform worker nodes whether they should continue or stop processing. The storage system compares the worker node's generation identifier with the current one, providing feedback that prevents conflicts by ensuring only the authorized node continues processing.
Solution Approach 2:
The master node performs preliminary action by updating the generation identifier in the storage system before reassigning the workload. This preliminary update ensures that when the old worker node checks the storage system, it detects the mismatch and stops processing, preventing conflicts before they occur.
3Adaptability or versatility
If the master node loses communication with the worker node, then the worker node operates autonomously, but the master node cannot determine the worker node's status leading to potential reassignment conflicts
Solution Approach 1:
The storage system serves as an intermediary that preserves worker node status information through generation identifiers. Even when the master node loses communication with a worker node, the storage system maintains the generation identifier, allowing the worker node to autonomously check its status and determine whether to continue or stop processing.
Solution Approach 2:
The worker node performs self-service by autonomously checking the storage system for its generation identifier and determining its processing status. This self-service mechanism allows the worker node to operate independently without continuous master node communication, reducing information loss while maintaining system consistency.
Data Source
AI summary
A generation identifier is employed with various systems and methods in order to identify situations where a workload has been reassigned to a new node and where a workload is still being processed by an old node during a failure between nodes. A master node may assign a workload to a worker node. The worker node sends a request to access target data. The request may be associated with a generation identifier and workload identifier that identifies the node and workload. At some point, a failure occurs between the master node and worker node. The master node reassigns the workload to another worker node. The new worker node accesses the target data with a different generation identifier, indicating to the storage system that the workload has been reassigned. The old worker node receives an indication from the storage system that the workload has been reassigned and stops processing the workload.