Data Block Signature Storage for Backup Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication techniques are computationally intensive, leading to significant performance overhead during large backup operations due to the need to read and compare data blocks for redundancy, which burdens storage systems with high data volumes.
Innovation Solution
Storing signature values alongside data blocks during backup operations, allowing for efficient retrieval and communication by reading only the signature values when data blocks are already present in the destination storage system, thereby reducing the need for redundant data reads and computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication techniques are used to eliminate redundant data, then storage utilization is improved, but computational overhead increases significantly
Solution Approach 1:
The patent segments data into fixed-size blocks and computes signatures only for these segments rather than processing entire files. This segmentation allows the system to apply deduplication at a granular level, improving storage utilization while reducing the computational burden compared to processing complete files.
Solution Approach 2:
The patent introduces signature values (hashes) as intermediaries between the original data and the deduplication comparison process. Instead of directly comparing large data blocks, the system computes and compares compact signature values, which significantly reduces computational overhead while maintaining deduplication effectiveness.
2Measurement precision
If signature computation is performed for every data block during backup, then deduplication accuracy is improved, but processing speed decreases
Solution Approach 1:
The patent creates compact copies (signatures) of data blocks instead of working with the full data blocks during comparison. These signature copies retain the essential identifying characteristics needed for deduplication accuracy while being much smaller and faster to process, thus resolving the trade-off between accuracy and speed.
Solution Approach 2:
The patent transforms the data representation from full data blocks to condensed signature values, changing the parameter from raw data size to compact hash size. This parameter transformation maintains the unique identification capability (accuracy) while dramatically reducing the processing burden (speed).
3Reliability
If all data blocks are read and compared during auxiliary copy operations, then data integrity is ensured, but network bandwidth consumption increases
Solution Approach 1:
The patent extracts only the necessary signature information from data blocks during auxiliary copy operations, rather than transferring complete data blocks. This extraction approach allows the system to verify data presence and integrity through signature comparison while minimizing network bandwidth consumption by transferring only essential metadata.
Solution Approach 2:
The patent applies partial action by performing signature comparison instead of full data block comparison during auxiliary copy operations. This partial approach is sufficient to determine data deduplication opportunities while consuming significantly less network bandwidth, reserving full data transfer only when necessary.
Data Source
AI summary
A system according to certain embodiments associates a signature value corresponding to a data block with one or more data blocks and a reference to the data block to form a signature/data word corresponding to the data block. The system further logically organizes the signature/data words into a plurality of files each comprising at least one signature/data word such that the signature values are embedded in the respective file. The system according to certain embodiments reads a previously stored signature value corresponding to a respective data block for sending from a backup storage system having at least one memory device to a secondary storage system. Based on an indication as to whether the data block is already stored on the secondary storage system, the system reads the data block from the at least one memory device for sending to the secondary storage system if the data block does not exist on the secondary storage system, wherein the signature value and not the data block is read from the at least one memory device if the data block exists on the secondary storage system.