Data Recovery in Distributed In-Memory Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed in-memory systems face data loss due to node failures, and conventional data replication methods like 2PC protocols lead to low resource utilization and performance issues.
Innovation Solution
The method involves utilizing multiple copies of transaction definitions across nodes, with careful coordination for data recovery, allowing failed nodes to restore their state using snapshots and commit logs, without discarding requests until it's safe, thus avoiding the drawbacks of 2PC protocols.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated to multiple nodes using 2PC protocol, then data reliability is improved, but system performance deteriorates due to server rounds of network communication
Solution Approach 1:
The patent applies preliminary action by pre-replicating transaction requests to multiple nodes before the transaction is committed. The coordinator node sends transaction requests to participant nodes in advance, and these requests are queued and held until the transaction commit is confirmed. This allows data to be prepared at multiple locations beforehand, reducing the need for slow network communication during the actual commit phase while maintaining data reliability.
2Reliability
If data is replicated to multiple nodes, then data reliability is improved, but resource utilization deteriorates at 1/k
Solution Approach 1:
The patent applies discarding and recovering by allowing temporary duplication of transaction requests during the preparation phase, then discarding these duplicate requests once the transaction is committed. The coordinator node can determine whether to discard requests based on commit status, and participant nodes can recover and execute committed transactions locally without needing to retain all duplicate copies indefinitely. This reduces resource consumption while maintaining reliability during the critical commit phase.
3Productivity
If requests are discarded early to improve performance, then system performance is improved, but data integrity deteriorates in case of node failure
Solution Approach 1:
The patent applies feedback by implementing a commit status notification mechanism where the coordinator node monitors transaction commit status and provides feedback to participant nodes. When a transaction is successfully committed, the coordinator notifies participant nodes to discard their queued requests. If commit fails or a node fails, the feedback mechanism ensures requests are retained and can be recovered. This feedback loop maintains data integrity while allowing performance optimization through selective discarding of committed transaction requests.
Data Source
AI summary
The present teaching relates to request management and data recovery in a data system. In one example, a failure in connection with first data is detected at a first node in a data system. Information associated with a most recent transaction related to the first node is obtained from the persistent storage. Each of other nodes in the data system is requested to transmit one or more transaction requests previously sent to the first node after the most recent transaction. The one or more transaction requests are received from at least one of the other nodes. A sequence of one or more transactions associated with the one or more transaction requests is determined. The one or more transactions are executed according to the sequence in order to recover the first data at the first node.