Object Replication Using Reduced Metadata Hashes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication and replication technologies are inadequate for distributed computing environments, leading to inefficient use of network bandwidth and resources, as they fail to effectively manage data replication across geographically dispersed systems.
Innovation Solution
Implementing a local metadata repository to accelerate block data replication using deduplication technologies, which involves calculating and transmitting metadata maps to identify and transfer only missing data blocks or objects, thereby reducing the amount of data sent over the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data replication technologies are used in distributed computing environments, then data redundancy is achieved, but network bandwidth utilization is excessive and resource consumption is high
Solution Approach 1:
The patent extracts only the essential metadata components (hash values, object identifiers, size information) from complete object data and transmits only these extracted elements for replication. This allows the system to achieve data redundancy verification without transmitting full object data, significantly reducing network bandwidth consumption while maintaining reliability.
Solution Approach 2:
The patent creates and transmits simplified copies of object metadata (hash values and identifiers) instead of complete object data. These metadata copies enable the destination site to verify data redundancy and identify missing objects without requiring transmission of the actual object content, reducing resource consumption while preserving data integrity.
2Reliability
If complete object data is transmitted for replication, then data synchronization is achieved, but data transfer time is excessive
Solution Approach 1:
The patent extracts critical identification and verification elements (object hashes, identifiers, size metadata) from complete object data and transmits only these extracted components. This extraction approach enables rapid synchronization verification at the destination site without waiting for lengthy complete object data transfers, significantly reducing data transfer time while maintaining synchronization accuracy.
Solution Approach 2:
The patent performs preliminary computation of object hashes and metadata generation at the source site before transmission. By preparing verification data in advance, the destination site can immediately begin synchronization verification upon receipt, eliminating waiting time and accelerating the overall data synchronization process.
3Measurement precision
If comprehensive metadata is transmitted for each object, then replication accuracy is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent extracts only the essential metadata elements required for replication verification (object hashes, identifiers, size information) and transmits only these extracted components. This selective extraction maintains replication accuracy by including all necessary verification data while eliminating redundant metadata, achieving precise replication with minimized network bandwidth consumption.
Solution Approach 2:
The patent applies partial metadata transmission by sending only the specific metadata components necessary for replication verification rather than complete object data or comprehensive metadata sets. This partial action approach provides sufficient information for accurate replication while consuming significantly less network bandwidth than complete data transmission.
Data Source
AI summary
Embodiments for, in an object storage environment, managing data replication between first and second sites of a distributed computing environment by one or more processors. A first pass metadata hash is calculated for each of the objects in an object-set that is subsequently transferred from the first to the second site. Responsive to the second site, a second pass metadata hash is calculated for remaining objects of the object-set that are identified by the second site at a sub-object level using a predetermined size.