Generation Identifiers for Workload Reassignment After Node Failure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In clustered environments, workload reassignment due to communication failures between master and worker nodes leads to performance issues and interruptions as both old and new nodes continue processing, causing conflicts and delayed processing.
Innovation Solution
Implementing a generation identifier mechanism that allows the storage system to recognize workload reassignment by associating unique identifiers with each node's processing, ensuring that only the new node gains access when the old node has completed its tasks, thereby preventing interruptions and ensuring high availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the master node reassigns the workload to a new worker node after communication failure, then the workload can continue processing on the new node, but the old node may also continue processing without knowledge of the reassignment, causing conflicts and performance issues
Solution Approach 1:
The storage system provides feedback to worker nodes about workload reassignment status. When the master node reassigns a workload to a new worker node, the storage system notifies the old worker node of the reassignment, enabling it to stop processing. This feedback mechanism prevents the old node from continuing to process the workload and avoids conflicts, thereby maintaining both reliability and productivity.
2Duration of action of moving object
If the old worker node continues processing the workload after communication failure, then processing can proceed without interruption, but conflicts arise when the new node also processes the same workload
Solution Approach 1:
The storage system implements a feedback mechanism that notifies worker nodes about workload reassignment. When a workload is reassigned from an old worker node to a new one, the storage system sends a notification to the old node. This feedback allows the old node to stop processing the workload, preventing conflicts while maintaining continuous processing through the new node, thus resolving the contradiction between processing duration and harmful conflicts.
3Adaptability or versatility
If the master node loses communication with the worker node, then the worker node operates autonomously, but the master node cannot determine whether to reassign the workload
Solution Approach 1:
The storage system acts as an intermediary that provides feedback to both the master node and worker nodes about workload status. Even when the master node loses communication with a worker node, the storage system maintains information about which node is processing which workload and can provide this information to the master node. This allows the master node to make informed reassignment decisions without directly communicating with the worker node, preserving autonomous operation while reducing information loss.
Data Source
AI summary
A generation identifier is employed with various systems and methods in order to identify situations where a workload has been reassigned to a new node and where a workload is still being processed by an old node during a failure between nodes. A master node may assign a workload to a worker node. The worker node sends a request to access target data. The request may be associated with a generation identifier and workload identifier that identifies the node and workload. At some point, a failure occurs between the master node and worker node. The master node reassigns the workload to another worker node. The new worker node accesses the target data with a different generation identifier, indicating to the storage system that the workload has been reassigned. The old worker node receives an indication from the storage system that the workload has been reassigned and stops processing the workload.