Database Cloning via Collector Nodes for Live Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Creating a database clone of a distributed database is complicated by changes applied during the cloning process, leading to inconsistent states and missing write operations in the cloned database, as write operations are typically denied during cloning, resulting in incomplete data replication.
Innovation Solution
A method that ensures data consistency by using collector nodes to transmit and confirm transaction data between source and target nodes, allowing for 'live cloning' where the database remains operational, enabling hot swapping and error analysis without service interruption, by replicating data in a fault-tolerant and distributed manner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If write operations are denied during the cloning process to ensure data consistency, then data consistency is improved, but productivity deteriorates due to service interruption
Solution Approach 1:
The system segments write operations into two categories: snapshot writes (during cloning) and new writes (after cloning). Snapshot writes are directed to both source and collector nodes to ensure consistency, while new writes are directed only to the source database. This segmentation allows the cloning process to maintain data consistency without preventing all write operations, thus resolving the contradiction between reliability and productivity.
Solution Approach 2:
The collector node acts as an intermediary between the source database and the clone database during the cloning process. It receives snapshot writes from the source node and forwards them to the target node, ensuring data consistency without requiring the source database to be offline. This intermediary mechanism enables continuous service availability while maintaining data consistency.
2Productivity
If snapshots are taken from multiple source nodes simultaneously to speed up cloning, then productivity is improved, but manufacturing precision deteriorates due to inconsistent data states
Solution Approach 1:
The system implements a feedback mechanism where the source node tracks which data blocks have been written to the collector node and communicates this information back to itself. This feedback allows the source node to wait for confirmation that snapshot writes have been successfully received by the collector before considering the cloning of those blocks complete, ensuring data consistency even when cloning from multiple source nodes simultaneously.
Solution Approach 2:
The source node performs preliminary actions by preparing snapshot writes and tracking them before the actual cloning process begins. It maintains a record of pending snapshot writes and their status, allowing it to coordinate with multiple collector nodes and ensure that all snapshot writes are properly received and applied before completing the cloning operation, thus maintaining manufacturing precision.
3Reliability
If the cloning process waits for all source nodes to complete snapshots before allowing writes, then data consistency is improved, but time is lost due to delayed write operations
Solution Approach 1:
The system dynamically adjusts the handling of write operations based on the cloning state. During the cloning process, write operations are allowed to proceed with snapshot writes to both source and collector nodes. After cloning is complete, the system transitions to a new state where only new writes to the source database are allowed. This dynamic adaptation eliminates unnecessary delays while maintaining data consistency at each stage.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
The invention relates to a method for creating a database clone of a distributed database, wherein the distributed database is stored on source nodes of a computer network, wherein the distributed database comprises a set of data items being distributed across the source nodes, wherein a first data item of the set of data items is stored on a first node of the source nodes, wherein the method comprises the steps of: -- coupling collector nodes to the computer network, wherein the collector nodes comprise a second node, -- starting a cloning process, wherein the cloning process includes: -- creating, on the second node, a copy of the first data item of the first node, -- updating, in a transaction step, the first data item of the first node, -- transmitting, to the second node storing the copy of the first data item, transaction data related to the transaction step, wherein the transaction data is stored on the second node, and/or updating the copy of the first data item being stored on the second node, -- completing the cloning process such that a complete copy of the set of data items of the source nodes is stored on the collector nodes.