Content-Based Block Alignment for Shift-Aware Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing block-based data storage systems struggle to detect and reduce redundant data at a fine granularity, such as small-offset shifts in file contents, due to their constraint of operating at page or block/sector boundaries, leading to inadequate storage savings in scenarios where files differ only by a single byte or small offset.
Innovation Solution
A content-aware method that selects a starting position in a block using a deterministic function, generates a block digest, and compares it with stored digests to identify matches, allowing for the storage of references to identical blocks rather than their full content, thereby reducing physical storage utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If block-based storage systems operate at page or block/sector boundaries, then device complexity is reduced and operation is simplified, but measurement precision of data identicality detection deteriorates
Solution Approach 1:
The patent divides a data block into multiple segments and computes hash values for each segment independently. This segmentation enables fine-grained comparison of data portions, allowing the system to detect identicality at a precision level finer than traditional block boundaries without requiring complex full-block processing for every comparison.
Solution Approach 2:
The patent introduces hash values as intermediary representations of data blocks. Instead of directly comparing raw block data, the system computes and compares hash values, which serve as compact intermediaries that preserve identicality information while reducing computational complexity and enabling efficient precision detection.
2Productivity
If block-based storage systems use fixed block boundaries, then storage management is simplified, but data reduction efficiency deteriorates
Solution Approach 1:
The patent segments data blocks into smaller units and computes hash values for each segment. This allows the system to identify and deduplicate only the identical segments rather than entire blocks, significantly improving data reduction efficiency while maintaining manageable storage operations through systematic segment processing.
Solution Approach 2:
The patent applies different processing qualities to different parts of the data structure. By computing hash values for specific segments rather than uniformly processing entire blocks, the system achieves local optimization where identical segments are efficiently detected and deduplicated, improving overall data reduction efficiency without complicating storage management.
3Quantity of substance
If traditional block alignment is used, then storage structure simplicity is maintained, but storage space utilization deteriorates
Solution Approach 1:
The patent extracts and stores only the hash values of data segments rather than the entire block structures. This extraction approach enables the system to identify identical segments and store references instead of duplicates, significantly improving storage space utilization while maintaining relatively simple alignment structures through selective information retention.
Solution Approach 2:
The patent uses hash values as compact copies or representations of the actual data segments. Instead of storing and comparing full block copies, the system stores and compares these digest copies, enabling efficient identification of identical segments for deduplication purposes, thereby improving storage utilization without requiring complex alignment mechanisms.
Data Source
AI summary
A method of data reduction in a block-based data storage system includes selecting a starting position in a block based on a deterministic function of block data content. Then for an unaligned block beginning at the selected starting position, a block digest (e.g., block hash) is generated and compared with stored block digests of stored data blocks. If there is a match, and the stored block matches the unaligned block, then a reference to the stored block is stored in place of the unaligned block, and otherwise the unaligned block and a corresponding digest are stored. The storing of references to already stored blocks, without the constraint of observing aligned-block boundaries, realizes increased savings of physical storage space.