Deduplication Entropy Filtering for Storage IOPS Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Variable deduplication in fixed block storage arrays with sector-level deduplication increases resource requirements, impacting IOs per second (IOPS) performance due to the need for additional memory and CPU resources, especially when calculating partial hashes for non-duplicate blocks.
Innovation Solution
A new distance function is introduced to identify 'very similar' blocks with minimal hash distance, allowing for efficient evaluation of duplicate sectors and reducing resource usage by calculating digital entropy values, which determines if blocks are candidates for deduplication based on a threshold, and using byte comparison to validate hash distances and prevent collisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If sector-level deduplication is implemented with variable deduplication in fixed block storage arrays, then data reduction capability is improved, but resource consumption (memory and CPU) increases and IOPS performance deteriorates
Solution Approach 1:
The patent applies partial action by implementing selective deduplication based on entropy thresholds. Instead of performing full deduplication on all blocks, the system calculates entropy values and only performs expensive hash comparisons on blocks with entropy below the threshold (indicating potential duplicates). This partial application of deduplication logic maintains data reduction benefits while avoiding unnecessary CPU and memory operations on non-duplicate blocks, thereby preserving IOPS performance.
Solution Approach 2:
The system performs preliminary entropy calculation on all blocks before committing to full deduplication processing. This preliminary action filters out blocks that are unlikely to be duplicates (high entropy blocks), preventing wasted computational resources. By pre-screening blocks using the lightweight entropy metric, the system prepares only the necessary subset of blocks for expensive hash-based comparison, optimizing both data reduction efficiency and performance.
2Measurement precision
If full hash calculation is performed on all blocks to ensure accurate duplicate detection, then deduplication accuracy is improved, but CPU and memory resource consumption increases
Solution Approach 1:
The patent implements partial hash calculation based on entropy filtering. Instead of calculating full hashes for all blocks, the system first computes entropy values and only performs partial or full hash calculations on blocks with entropy below the threshold. This selective approach maintains duplicate detection accuracy for actual duplicates while avoiding unnecessary computational expenditure on unique blocks, thereby resolving the contradiction between accuracy and resource consumption.
Solution Approach 2:
The entropy calculation serves as an intermediary filtering mechanism between block identification and full hash comparison. This intermediary step provides a low-cost approximation of duplicate likelihood, allowing the system to screen blocks before committing expensive hash-based verification. The entropy metric acts as a gatekeeper that preserves accuracy for true duplicates while eliminating wasteful computations on non-duplicates.
Data Source
AI summary
A method, computer program product, and computer system for identifying a plurality of blocks. At least one heuristic associated with at least a portion of the plurality of blocks may be determined. It may be determined whether at least the portion of the plurality of blocks is a candidate for deduplication based upon, at least in part, the at least one heuristic. At least the portion of the plurality of blocks may be deduplicated based upon, at least in part, the at least one heuristic.