Node Management for Pending and Unstable Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data storage systems, node failures can lead to the loss of executed operations that have not been stored in non-volatile memory, disrupting the system's ability to maintain global state consistency and requiring re-execution of operations, which can result in the deletion of dependent operations.
Innovation Solution
Implementing a system with nodes that maintain pending and unstable operations lists, where operations are executed only after all dependencies are completed and stored in non-volatile memory, and in case of a failure, the unstable operations list from a healthy node is used to repopulate the failed node's lists, ensuring no operations are deleted.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If operations are executed in a distributed data storage system, then system functionality and data processing capability are improved, but node failures can cause loss of executed operations that have not been stored in non-volatile memory, requiring re-execution and potentially deleting dependent operations
Solution Approach 1:
The system performs preliminary actions by maintaining pending operations lists and unstable operations lists that record all executed operations before they are permanently stored. When a node fails, these pre-recorded lists allow for recovery without losing any operations, as the system has already documented what was executed but not yet persisted to non-volatile storage.
Solution Approach 2:
The patent implements a cushioning mechanism by creating redundant records of executed operations in pending and unstable operations lists. These lists act as a buffer that protects against data loss during node failures, allowing the system to recover the exact state before failure without deleting any operations or requiring re-execution.
2Stability of the object's composition
If the system re-executes operations after node failure, then consistency is restored, but dependent operations may be deleted and re-execution time is lost
Solution Approach 1:
The system creates copies of operation records in pending operations lists and unstable operations lists that can be transferred between nodes. When a node fails, another node can provide copies of these operation records, allowing the failed node to restore its state without re-executing operations, thereby eliminating re-execution time while maintaining consistency.
Solution Approach 2:
By pre-recording all executed operations in persistent data structures (pending and unstable operations lists) before node failure, the system eliminates the need for re-execution during recovery. The preliminary recording ensures that all operations are captured and can be restored from copies provided by other nodes.
3Reliability
If operations are stored in non-volatile memory, then data persistence is improved, but operations cannot be executed until all dependencies are completed and stored
Solution Approach 1:
The patent segments operations into different categories: pending operations (executed but not persisted), unstable operations (in flight or partially processed), and persisted operations (stored in non-volatile memory). This segmentation allows the system to track dependencies precisely and execute operations as soon as their dependencies are met, without waiting for all operations to be fully persisted to non-volatile storage, thereby improving execution speed while maintaining reliability.
Solution Approach 2:
The system dynamically adjusts the persistence timing of operations based on dependency satisfaction. Once an operation's dependencies are met and the operation is executed, it is recorded in the pending operations list immediately, allowing subsequent operations to proceed without waiting for non-volatile storage completion. The persistence to non-volatile memory occurs asynchronously, balancing reliability and execution speed.
Data Source
AI summary
A system including a plurality of nodes. Each node includes a pending operations list, an unstable operations list, and a hardware controller. The pending operations list can include operations that have not yet been executed on the node. The unstable operations list can include operations that have not yet been stored on non-volatile data storage. Each of the operations listed in the unstable operations list can include a listing of all other operations that must be executed prior to the operation being executed on each respective node. The hardware controller can be configured to execute at least one operation listed in the pending operations list, remove at least one executed operation from the pending operations list, and remove at least one operation from the unstable operations list.