Database Change Synchronization Using Bit-Vector Indicators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Synchronizing changes, deletions, and additions in large datasets between databases leads to excessive bandwidth and processing resource usage, wasting network and processing resources due to the need for frequent data replication in distributed architectures.
Innovation Solution
A method using a change indication bit-vector to efficiently communicate changes by indicating positions of deletions and additions in a data-set representation structure, allowing for compressed transmission and updating of data fields in replica databases, thereby reducing unnecessary data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete datasets are replicated between databases to ensure data consistency, then data freshness and consistency are improved, but network bandwidth usage and processing resource consumption increase significantly
Solution Approach 1:
The patent extracts only the changed portions of data (specific fields that have been added, deleted, or modified) from the complete dataset and transmits only these extracted changes to the replica database. This is achieved through change detection mechanisms that identify modified fields and generate compact change indicators (such as bit vectors) that represent only the necessary updates, thereby significantly reducing network bandwidth consumption while maintaining data consistency.
Solution Approach 2:
The patent segments the complete dataset into individual fields or small groups of fields, allowing independent tracking and transmission of changes. By dividing the dataset into discrete units (fields with associated keys and positions), the system can identify and transmit only the segmented portions that have changed, rather than replicating the entire dataset, thus reducing processing overhead and network usage.
2Reliability
If frequent data replication is performed to maintain real-time consistency, then data freshness is improved, but processing overhead and system load increase
Solution Approach 1:
The patent performs preliminary actions by pre-defining the data structure schema (DSR - Data Set Representation) that organizes fields with unique keys and positions before replication occurs. This preliminary structuring enables efficient change detection and indexing, allowing the system to quickly identify modified fields without scanning entire datasets during replication events, thereby reducing processing overhead while maintaining real-time data freshness.
Solution Approach 2:
The patent changes the parameter representation of data changes by using compact bit vectors instead of traditional verbose change logs. Each bit in the vector corresponds to a specific field position in the DSR structure, providing a highly compressed parameter representation that reduces processing requirements for tracking and applying changes, thus lowering system load while enabling frequent replication.
3Manufacturing precision
If detailed change information is transmitted to indicate field additions and deletions, then replication accuracy is improved, but data transmission size increases
Solution Approach 1:
The patent transforms the parameter representation of change information by encoding field positions, keys, and change types (addition, deletion, modification) into compact bit vectors. Each bit vector uses individual bits to represent specific field positions, with additional metadata (such as key-value pairs for added fields) transmitted only when necessary. This parameter transformation maintains complete replication accuracy while dramatically reducing the volume of transmitted data compared to traditional change logging methods.
Data Source
AI summary
A device, system and method for synchronizing databases changes is provided. A change-indication bit-vector indicates a position of a change to a stored data-set and/or a respective key position defined by a data-set representation (DSR) that defines a data-set structure. When the change comprises a data-field deletion, an indication of the change-indication bit-vector is provided to a replication node managing a replica database storing a replica data-set; the replication node updates the replica and a respective DSR to show the deletion at the indicated position. When the change comprises a data-field addition, the indication of the change-indication bit-vector and a record of a key value of a new data field and a data value of the new data field, is provided to the replication database; the replication node updates the replica and the respective DSR to show the addition at the indicated position.