Invertible Bloom Filter Sync for Low-Bandwidth Database Reconciliation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data synchronization methods between databases involve costly operations such as scanning and copying data records, leading to high bandwidth consumption.
Innovation Solution
A centralized database management system uses invertible bloom filters to encode data tables, perform subtraction operations on these filters to identify differences, and generate instructions for data synchronization, reducing the need for raw data copying and minimizing bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional methods scan and copy data records between databases, then data synchronization is achieved, but bandwidth consumption increases and operation cost rises
Solution Approach 1:
The patent extracts only the essential identification information (primary keys and checksums) from complete data records to create row representations. This extracted minimal representation is then used to generate bloom filters, allowing synchronization without transmitting full data records, thus reducing bandwidth consumption while maintaining data consistency.
Solution Approach 2:
Instead of copying actual data records, the patent creates and transmits bloom filter data structures that represent sets of row representations. These bloom filters enable identification of synchronized rows without copying the underlying data, achieving synchronization with minimal bandwidth usage.
2Reliability
If complete data records are copied between databases, then data consistency is maintained, but operation complexity and cost increase
Solution Approach 1:
The patent introduces bloom filters as an intermediary data structure between the source and destination databases. These bloom filters serve as mediators that enable comparison and identification of synchronized rows without requiring direct manipulation of complete data records, simplifying the synchronization operation while maintaining consistency.
Solution Approach 2:
The patent replaces the mechanical process of scanning and copying complete data records with a more efficient computational approach using bloom filters. Instead of physically moving data, the system uses probabilistic data structures to identify synchronization status, reducing operational complexity.
3Measurement precision
If row representations include primary keys and checksums, then data verification is improved, but data structure complexity increases
Solution Approach 1:
The patent changes the parameters of row representations by including not only primary keys but also checksums of specific columns. This parameter enhancement improves data verification accuracy by enabling detection of modifications, while the structured approach to selecting which columns to checksum keeps the complexity manageable.
Data Source
AI summary
A centralized database management system performs data synchronization with lower bandwidth consumption and higher efficiency. The system manages data synchronization and data reconciliation across multiple databases managed by multiple DBMS across different client servers. The system generates and sends instructions that encode each data table into an invertible bloom filter and identifies differences between the two databases by performing a subtraction operation on the two invertible bloom filters. The system may generate a third invertible bloom filter comprising information associated with differences between the two data tables. The system may send instructions to the source and the destination databases, where a first and a second invertible bloom filters are encoded for the source and the destination databases, respectively. The system may decode the third invertible bloom filter, identify the different elements, and generate instructions to the source and/or the destination database.