Check Vector Compression for Low-Latency Database Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anti-entropy protocols for distributed databases face challenges in achieving efficient and accurate data base consistency with minimal overhead and latency, particularly when dealing with large databases and multiple copies, as they either require substantial overhead in single-round protocols or increased latency in multi-round protocols.
Innovation Solution
The method employs Slepian-Wolf encoding to compress and transmit a check vector from a transmitter unit to a receiver unit, using hash values to efficiently detect discrepancies in distributed data sets, combined with multi-dimensional partitioning to enhance accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If complete Merkle trees are exchanged via data net in a single communication round, then latency time is short, but overhead is large
Solution Approach 1:
The patent extracts only the essential information needed for consistency checking by computing check vectors from Merkle tree root hash values. Instead of transmitting complete Merkle trees, the system extracts and transmits only the condensed check vectors that contain the necessary consistency verification information, thereby reducing overhead while maintaining single-round communication for low latency
Solution Approach 2:
The patent transforms the representation of database consistency information by changing from transmitting complete tree structures to transmitting condensed check vectors derived from root hash values. This parameter change in data representation enables significant overhead reduction while preserving the ability to detect inconsistencies efficiently
2Quantity of substance
If a multi-round protocol is used, then overhead is small, but latency time is long
Solution Approach 1:
The patent performs preliminary computation of check vectors from Merkle tree root hash values before transmission. By pre-computing these condensed representations, the system enables single-round communication where the receiver can immediately verify consistency without requiring multiple iterative rounds of exchange, thus reducing latency while keeping overhead minimal
3Measurement precision
If very large Merkle trees are transmitted, then differences between databases are precisely identified, but overhead is substantial
Solution Approach 1:
The patent uses cryptographic hash functions to create condensed copies (check vectors) of the database states represented by Merkle trees. These hash-based copies preserve the essential consistency information needed for detection while occupying minimal space, enabling precise difference identification with very small overhead compared to transmitting actual tree structures
Data Source
AI summary
The invention relates to a method for transmitting a check vector from a transmitter unit to a receiver unit check vector being provided for checking the consistency of a data set. The method includes dividing a data set into a plurality of sections; calculating a hash value for each of the sections, using a hash function; combining the check vector from the calculated hash values; compressing the check vector, the compression being performed using a Slepian-Wolf encoding method; and transmitting the compressed check vector from the transmitter unit to the receiver unit.