Block Deduplication Using Sequence Classifications
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data block deduplication methods do not distinguish between long and short block sequences, leading to fragmentation of long sequences and inefficient read operations, resulting in longer latencies and operational deficiencies.
Innovation Solution
Implementing block deduplication using sequence classifications based on criteria such as exact long match, disrupted long match, loosely long match, and short match, allowing users to selectively apply deduplication processes to specific categories of block sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If conventional block deduplication is applied to all data blocks without distinction, then storage space is conserved through deduplication, but long block sequences become fragmented leading to inefficient read operations and longer latencies
Solution Approach 1:
The patent segments block sequences into different categories (long sequences vs. short sequences) and applies different deduplication strategies to each segment. Long block sequences are identified and excluded from deduplication to maintain their integrity and sequential access efficiency, while short sequences and individual blocks are subjected to deduplication to conserve storage space. This segmentation resolves the contradiction by applying deduplication selectively rather than universally.
Solution Approach 2:
The patent applies different quality characteristics to different parts of the data set by implementing location-aware deduplication. Specifically, blocks belonging to long sequential regions are treated differently (excluded from deduplication) compared to blocks in short sequences or isolated locations. This local quality approach ensures that deduplication is applied where beneficial (space conservation) while preserving sequential access patterns where critical (latency reduction).
2Loss of substance
If block deduplication is applied to all candidate blocks, then storage efficiency is improved, but the complexity of managing different sequence types and access patterns increases
Solution Approach 1:
The patent introduces a length parameter (block sequence length) as a criterion for deduplication decision-making. By changing the parameter from a binary deduplication/not-deduplication approach to a length-based classification approach, the system simplifies management while maintaining storage efficiency. Long sequences (above a threshold) are automatically excluded, while shorter sequences are deduplicated, providing a clear and manageable rule set that balances efficiency and complexity.
3Productivity
If single data blocks are deduplicated independently, then maximum deduplication ratio is achieved, but sequential access patterns are disrupted causing defragmentation over time
Solution Approach 1:
The patent performs preliminary identification and classification of block sequences before applying deduplication. By first identifying long sequential patterns and marking them for exclusion, the system prevents fragmentation before it occurs. This preliminary action ensures that sequential integrity is maintained proactively rather than requiring corrective defragmentation operations later, thus preserving both sequence stability and achieving reasonable deduplication ratios.
Data Source
AI summary
A computer-implemented technique performs block deduplication. The technique involves generating a list of candidate blocks and adding classifications to the list of candidate blocks. The classifications classify the candidate blocks on the list into different categories based on a set of block sequence matching criteria. The technique further involves performing, based on the classifications added to the list of candidate blocks, further block deduplication processing on first candidate blocks having a first classification while not performing further block deduplication processing on second candidate blocks having a second classification. Examples of the categories include an “exact long match” category, a “disrupted long match” category, a “loosely long match” category, and a “short match” category.