Seeding Replication Using Bloom Filters for Deduplicated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplicated systems, efficiently seeding replication for a portion of data stored on an originating system to a replica system is hindered by excessive traffic and communication overhead due to the need to check each reference for segment presence, especially when dealing with many references compared to actual data segments.
Innovation Solution
A data structure, such as a Bloom filter, is generated on the originating system to track referenced segments, allowing for efficient identification and transfer of only unique segments to the replica system, reducing communication to the segment level and minimizing bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If each reference is checked individually to determine segment presence on the replica system, then accurate identification of segments to be transferred is achieved, but substantial communication traffic and time are generated
Solution Approach 1:
The patent combines multiple individual reference checks into a single batched communication process. Instead of checking each reference separately (which would generate substantial traffic), the system collects multiple reference checks and processes them in batches, significantly reducing the number of communication round-trips between the originator and replica systems while maintaining accurate segment identification.
Solution Approach 2:
The system performs preliminary actions by collecting and preparing multiple reference checks before initiating communication with the replica system. By pre-processing the reference list and organizing checks in advance, the system minimizes the actual communication time required while ensuring accurate identification of segments that need to be transferred.
2Reliability
If each reference is checked individually to determine segment presence, then complete data replication is ensured, but communication bandwidth is substantially consumed
Solution Approach 1:
The patent merges multiple individual reference verification operations into consolidated batch operations. By combining multiple reference checks into single communication transactions, the system maintains complete and accurate data replication while dramatically reducing the total communication bandwidth consumed during the seeding replication process.
3Measurement precision
If all references are processed to identify segments for replication, then accurate segment selection is achieved, but excessive traffic is generated between systems
Solution Approach 1:
The system performs preliminary processing of the reference list locally at the originator system before communication occurs. By pre-organizing and pre-processing reference data into batches, the system achieves accurate segment selection while minimizing the quantity of communication traffic required, as most processing is done locally rather than through repeated system-to-system queries.
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
Seeding replication is disclosed. One or more but not all files stored on a deduplicated storage system are selected to be replicated. One or more segments referred to by the selected one or more but not all files are determined. A data structure is created that is used to indicate that at least the one or more segments are to be replicated. In the event that an indication based at least in part on the data structure indicates that a candidate segment stored on the deduplicating storage system is to be replicated, the candidate segment is replicated.