Data Consistency in Distributed Nodes via Bit-Vector Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication methods in distributed computing architectures are inefficient due to high network communication costs and latency, particularly when dealing with varying data formats and sizes, and lack flexibility in transmitting only the necessary data changes.
Innovation Solution
A method utilizing a consistency acceleration engine that classifies data fields into volatile and stable categories, generates bit-vectors for volatile data, and transmits these along with updated data structures, allowing for efficient data packetization and compression to reduce bandwidth and latency, thereby optimizing data replication across computing nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete data-set replication is performed using the force approach, then data consistency is maintained, but network transmission volume and latency increase significantly
Solution Approach 1:
The patent extracts only the changed portions of data-sets rather than transmitting complete data-sets. The system identifies and transmits only the specific data blocks that have changed since the last replication, significantly reducing network transmission volume while maintaining data consistency across computing nodes.
Solution Approach 2:
The patent divides data-sets into discrete blocks and tracks changes at the block level. By segmenting the data and identifying which specific blocks have changed, the system transmits only the necessary segments rather than complete data-sets, optimizing network efficiency.
2Quantity of substance
If change detection is implemented to transmit only modified blocks, then network transmission volume is reduced, but the complexity of tracking and detecting changes increases
Solution Approach 1:
The patent pre-divides data-sets into blocks and establishes a block change tracking mechanism in advance. By preparing the data structure and change tracking infrastructure beforehand, the system can efficiently detect and transmit only changed blocks without requiring complex real-time analysis during replication.
3Reliability
If multiple-master database replication with change event logging is used, then data changes are tracked accurately, but message overhead increases due to including key and format information
Solution Approach 1:
The patent extracts only the essential change information (the actual changed data blocks) without including redundant metadata such as keys and format descriptions in each transmission message. This approach maintains accurate change tracking while significantly reducing message overhead compared to traditional change event logging.
4Productivity
If serialization schemes like Google protocol-buffer are used for data transmission, then transmission efficiency is improved, but flexibility in handling diverse data formats and sizes is reduced
Solution Approach 1:
The patent applies serialization selectively only to the changed data blocks that require format standardization for efficient transmission, rather than serializing entire data-sets or all data. This partial application of serialization maintains transmission efficiency for the transmitted portions while preserving flexibility in handling diverse data formats and sizes in the overall system.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention relates to a method and a system for maintaining consistency of data between data-sets stored in a master database of a master computing node and corresponding data-sets stored in a replication database of at least one replication computing node. Each time an updated version of a stored data set is received, the master computing node is configured for updating a corresponding data-set stored in the master database and transmitting replication data relating to the updated data-set version to the at least one replication computing node for replication. The replication data comprising a bit-vector storing data values extracted from data fields of the data-set classified as volatile data fields. The data values stored in the bit-vector representing the changes in the data-set to be replicated in the at least one replication node.