Resizable Invertible Bloom Filters for Low-Bandwidth Database Sync

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional database synchronization methods involve costly operations such as scanning and copying data records, leading to high bandwidth consumption.

Innovation Solution

A centralized database management system using resizable invertible bloom filters to encode data tables and perform subtraction operations to identify differences between databases, optimizing data synchronization by reducing bandwidth consumption and improving efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional methods scan and copy data records between databases, then data consistency is maintained, but bandwidth consumption increases

Engineering Contradiction:

Improvedata consistencyVSAvoidbandwidth consumption

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent extracts only the essential information needed for synchronization by encoding data tables into bloom filters, which represent the data in a compressed probabilistic format. This allows comparing databases without transmitting or copying actual data records, significantly reducing bandwidth consumption while maintaining the ability to identify and sync differences.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Instead of copying actual data records, the patent creates and transmits compact bloom filter representations that serve as proxies for the data. These bloom filters can be subtracted and decoded to identify differences, achieving synchronization without high-bandwidth record copying.

Inventive Principle:

Principle #26Copying

2Reliability

If data records are copied between databases, then synchronization is achieved, but operation cost increases

Engineering Contradiction:

ImprovesynchronizationVSAvoidoperation cost

Core Design Contradiction:

ReliabilityVSEase of manufacture

Solution Approach 1:

The patent replaces the mechanical process of scanning and copying data records with a mathematical approach using bloom filters. By encoding data into probabilistic data structures and performing set operations (subtraction) on these filters, the system identifies differences without the costly mechanical operations of record-by-record scanning and copying.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent changes the representation parameter of data from full data records to compact bloom filter encodings. This parameter transformation reduces the size and complexity of data being processed, making synchronization operations less costly while maintaining the ability to detect and transfer only necessary changes.

Inventive Principle:

Principle #35Parameter changes

3Loss of energy

If invertible bloom filters are used to identify differences, then bandwidth consumption is reduced, but device complexity increases

Engineering Contradiction:

Improvebandwidth consumptionVSAvoidsystem complexity

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The patent introduces bloom filters as an intermediary data structure between the source and destination databases. These filters serve as mediators that enable difference identification through mathematical operations without requiring direct comparison or copying of actual data records, simplifying the synchronization process while reducing bandwidth usage.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Measurement precision

If full data tables are compared, then accuracy is maintained, but processing time increases

Engineering Contradiction:

Improvedifference identification accuracyVSAvoidsynchronization time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent segments the data comparison process into two stages: first, using compact bloom filter representations to quickly identify potential differences through fast set operations, and second, decoding only the identified differences to retrieve actual changed records. This segmentation avoids the time-consuming process of comparing full data tables while maintaining accuracy in difference identification.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12602399B2Database synchronization using resizable invertible bloom filters with database snapshots

Publication Date: 2026.04.14 FIVETRAN INC

AI summary

A centralized database management system performs data synchronization with lower bandwidth consumption and higher efficiency using a resizable invertible bloom filter. The system may include a resizable invertible bloom filter module that constructs and maintains invertible bloom filters that are resizable based on a number of differences between different snapshots. The resizable invertible bloom filter module may maintain a list of possible sizes for a resizable invertible bloom filter. The resizable invertible bloom filter module may determine and maintain a list of applicable partition sizes, each partition size being a product of a divisor and a resizing factor. If the number of differences exceeds the number of expected differences and results in failure in decoding, the system may retry a larger size in a set of predetermined sizes. The system may continue to try larger sizes until a minimal size required for successful decoding is found.