Fingerprint Segmentation for Network Replication Metadata Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication methods for disaster recovery face challenges in reducing metadata transmission over low bandwidth links, leading to inefficiencies in data compression and increased data transfer during backup processes.
Innovation Solution
The method involves analyzing fingerprints of data chunks to select a representative fingerprint, generating a shorter fingerprint representation, and transmitting these representations instead of full fingerprints, allowing the target system to recover missing fingerprints and reduce network traffic through differential encoding and confirmation processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full fingerprints are transmitted for each data chunk, then the target system can accurately identify and reconstruct all files, but the amount of metadata transmitted over the network increases significantly
Solution Approach 1:
The fingerprint is segmented into multiple components: a shared prefix portion and a unique suffix portion. Only the unique suffix is transmitted for each data chunk, while the shared prefix is transmitted once for the entire batch. This segmentation reduces metadata transmission volume while maintaining the ability to accurately reconstruct full fingerprints at the target system.
Solution Approach 2:
Multiple fingerprints are merged by identifying and extracting their common prefix portion. The shared prefix is combined with individual unique suffixes to reconstruct complete fingerprints at the target. This merging approach reduces the total metadata transmitted while preserving identification accuracy for all data chunks in the batch.
2Productivity
If all unique data chunks are transmitted to achieve high compression ratios, then storage efficiency improves, but the complexity of tracking and managing metadata increases
Solution Approach 1:
The unique identifying portion of each fingerprint is extracted and transmitted separately from the shared prefix. This extraction allows the system to maintain high compression ratios by transmitting only necessary unique data while simplifying metadata management through differential encoding rather than managing complete fingerprint sets for all chunks.
Solution Approach 2:
The fingerprint representation is changed from complete fixed-length hashes to variable-length differential representations. This parameter change reduces the average metadata size transmitted and simplifies management by leveraging the statistical properties of fingerprint similarity across related data chunks.
3Loss of energy
If differential encoding is used to reduce metadata transmission, then network bandwidth efficiency improves, but the risk of fingerprint reconstruction errors increases
Solution Approach 1:
The target system sends feedback to the source system regarding the successful reconstruction of fingerprints from the transmitted differential data. This feedback mechanism allows the source to verify reconstruction accuracy and retransmit any missing or erroneous unique portions, ensuring reliability while maintaining bandwidth efficiency through selective retransmission rather than redundant initial transmission.
Solution Approach 2:
The source system prepares and transmits the shared prefix portion of fingerprints in advance before transmitting the unique suffix portions. This preliminary action ensures that when unique portions are received, the target system can immediately reconstruct complete fingerprints without error, as the foundation data is already in place and verified.
Data Source
AI summary
A target storage system receives a representative fingerprint and fingerprint representations from a source storage system. Each fingerprint representation contains only a portion of a corresponding fingerprint and the representative fingerprint is a full fingerprint. The fingerprints of the data chunks are missing at the target storage system are identified based on the fingerprint representative and the fingerprint representations. A bitmap is transmitted to the source storage system, each bit indicating whether one of the fingerprints is missing. One or more fingerprints are received from the source storage system that are missing at the target storage system based on the bitmask. One or more missing data chunks are identified based on at least the one or more fingerprints received from the source storage system. The missing data chunks are then received from the source storage system to be stored at the target storage system.