Network Efficient Deduplication via Segment Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication systems require sending entire data sets over networks to identify duplicates, leading to inefficient use of network resources and increased costs, as they do not effectively determine which data segments are already deduplicated before transmission.
Innovation Solution
Implementing a system that determines segment identifiers (IDs) for files and compares them with existing IDs in a deduplication system, allowing only new segments to be transmitted and updating references and reference counts, thereby reducing redundant data transmission and optimizing deduplication processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional deduplication systems send entire data sets over networks to identify duplicates, then complete data verification is achieved, but network bandwidth consumption increases and backup time extends
Solution Approach 1:
The patent extracts only the essential identifying characteristics (hash values) of data segments from the complete data sets. Instead of transmitting entire data sets for comparison, the system extracts and transmits only hash values, which are sufficient for duplicate identification. This extraction principle resolves the contradiction by maintaining verification reliability through hash comparison while dramatically reducing network bandwidth consumption.
Solution Approach 2:
The patent performs preliminary hashing of data segments before network transmission. By pre-computing hash values locally and transmitting only these compact identifiers for comparison, the system avoids the need to transmit complete data sets. This preliminary action enables efficient duplicate detection while minimizing network resource consumption.
2Measurement precision
If conventional deduplication systems transmit all data segments for processing, then accurate duplicate identification is achieved, but backup window duration increases
Solution Approach 1:
The patent uses hash values as compact copies or representations of the actual data segments. Instead of transmitting and processing complete data segments, the system transmits and compares hash values, which are computational copies that preserve the essential identifying characteristics. This copying approach maintains duplicate identification accuracy while significantly reducing the time required for backup operations.
Solution Approach 2:
The patent changes the parameter being transmitted from complete data segments to condensed hash values. This parameter transformation maintains the ability to identify duplicates accurately (since hash functions preserve uniqueness) while reducing the volume of data requiring transmission and processing, thereby shortening backup window duration.
3Quantity of substance
If data compression is applied to reduce storage requirements, then storage efficiency improves, but processing complexity increases
Solution Approach 1:
The patent segments data into fixed-size blocks and applies hashing to each segment independently. This segmentation approach enables efficient storage utilization through deduplication while keeping processing complexity manageable by working with uniform, fixed-size segments rather than attempting to compress entire files. The segmentation principle allows parallel processing and simplifies the compression pipeline.
Data Source
AI summary
Mechanisms are provided for performing network efficient deduplication. Segments are extracted from files received for deduplication at a host connected to a target over one or more networks and/or fabrics in a deduplication system. Segment identifiers (IDs) are determined and compared with segment IDs for segments already deduplicated. Segments already deduplicated need not be transmitted to a target system. References and reference counts are modified at a target system. Updating references and reference counts may involve modifying filemaps, dictionaries, and datastore suitcases for both already deduplicated and not already deduplicated segments.