Computing Node Failure Detection Through Periodic NodeRun Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing environments with multiple nodes, ensuring that a unit of work is run only once becomes challenging when nodes fail unexpectedly, leading to stuck work units due to unanticipated power loss or status updates not being changed, and existing solutions face issues with setting expiry times that either violate the once-per-node constraint or require excessive resource usage.
Innovation Solution
Implementing a node run service that generates a unique identifier (NodeRun) on node startup, periodically updates this identifier in a shared data repository, and checks for node activity by comparing NodeRuns across nodes to reassign tasks to active nodes, thereby ensuring work units are not stuck for long periods even if a node fails unexpectedly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If an expiry time is set with the status field to identify failed nodes, then the time work units are stuck is reduced, but the risk of violating the once-per-node constraint increases when legitimate work exceeds the expiry time
Solution Approach 1:
Instead of using a static expiry time, the patent implements periodic heartbeats where nodes send status updates at regular intervals. The system checks for node failures by detecting the absence of these periodic signals, allowing dynamic adjustment of failure detection timing without fixed constraints on work unit duration
Solution Approach 2:
The patent changes the parameter from a fixed expiry time to a dynamic heartbeat interval. By monitoring whether nodes send heartbeats within expected intervals rather than relying on predetermined time limits, the system adapts to varying work unit durations while maintaining reliable failure detection
2Loss of time
If a short expiry time is set and continuously renewed to decrease stuck time, then the responsiveness to node failures improves, but the resource usage of the shared data store increases
Solution Approach 1:
The patent extracts the failure detection mechanism from the shared data store by implementing local heartbeat monitoring. Each node independently tracks the last known heartbeat time of other nodes, eliminating the need for continuous data store operations and reducing resource usage while maintaining fast failure detection
Solution Approach 2:
Each node autonomously monitors the health of other nodes by checking their heartbeat timestamps locally stored in memory. This self-service approach eliminates dependency on shared data store operations for failure detection, reducing resource consumption while enabling rapid response to node failures
Data Source
AI summary
Systems and methods for identifying failed computing nodes are disclosed. An example system includes a plurality of computing nodes, where each computing node of the plurality of computing nodes includes, or has access to, at least one hardware processor. An example method includes generating a unique identifier particularly corresponding to a particular node. The example method also includes periodically updating a node record in a persistent storage location with the unique identifier, where the node record is associated with the particular node. The example method further includes accessing the persistent storage location at a first time and identifying a most recent update to the node record. The example method also includes determining that the most recent update to the node record occurred more than a threshold amount of time prior to the first time.