Differential Encoding for Network Replication Metadata Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication methods for disaster recovery over low bandwidth links are inefficient due to the high transmission of metadata, even after achieving significant compression through deduplication, as they require transmitting full fingerprints of data chunks, which can be redundant.
Innovation Solution
The method involves selecting a representative fingerprint and its representations for each batch of data chunks, allowing the target system to recover missing fingerprints through differential encoding, reducing the amount of metadata transmitted over the network by using bitmask or bit vector information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full fingerprints of data chunks are transmitted during network replication, then the target system can accurately identify and retrieve data chunks, but the amount of metadata transmitted increases significantly
Solution Approach 1:
The patent segments the fingerprint transmission process by dividing fingerprints into batches and selecting only representative fingerprints from each batch to transmit. This segmentation allows the system to reduce metadata transmission while maintaining accurate data chunk identification through the representative fingerprints that point to the actual data chunk locations.
Solution Approach 2:
The patent extracts only the essential information from the full fingerprints by selecting representative fingerprints and their associated metadata (such as offset and length information). This extraction allows the target system to reconstruct the necessary data chunk identification information without transmitting the complete fingerprint data for all chunks, thereby reducing metadata transmission volume.
2Loss of energy
If differential encoding is used to reduce metadata transmission, then bandwidth usage decreases, but the complexity of the replication process increases
Solution Approach 1:
The patent applies preliminary action by pre-organizing fingerprints into batches and pre-determining representative fingerprints before the actual replication process. This preliminary organization allows the differential encoding to work more efficiently, as the system already has the fingerprint batches structured and ready for comparison, reducing the computational complexity during replication.
Solution Approach 2:
The patent implements feedback mechanisms where the source system transmits metadata about which fingerprints are representative and their corresponding data chunk locations. The target system uses this feedback information to accurately identify and retrieve only the necessary data chunks, managing the replication process complexity through informed decision-making based on transmitted metadata.
3Quantity of substance
If only representative fingerprints are transmitted, then metadata transmission volume is reduced, but the target system must perform additional processing to recover missing fingerprints
Solution Approach 1:
The patent introduces intermediary information in the form of representative fingerprints and associated metadata (offset, length, and indexing information) that act as mediators between the transmitted data and the actual data chunks. This intermediary allows the target system to reconstruct the necessary fingerprint information and identify data chunks without receiving all full fingerprints, balancing transmission reduction with processing requirements.
Data Source
AI summary
According to one embodiment, in response to a list of fingerprints representing data chunks of a first batch for replication, a second batch previously transmitted to the target storage system is identified based on at least a portion of the fingerprints of the first batch. Differential encoding information is generated representing a difference between fingerprints of the first batch and the second batch. The differential encoding information is transmitted to the target storage system, without transmitting all full fingerprints of the first batch, to allow the target storage system to determine which of the data chunks are missing at the target storage system. In response to information received from the target storage system indicating one or more data chunks that are missing at the target storage system, the missing data chunks are then transmitted to the target storage system.