Inline Storage Data Reduction Using Entropy-Based Block Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression and deduplication techniques in storage systems are often ineffective or costly when data lacks redundancy, as they do not assess the likelihood of data sequences being previously stored, leading to inefficient resource allocation and performance degradation.
Innovation Solution
The method identifies data blocks based on entropy to determine whether to apply compression or deduplication, grouping chunks with similar entropy to enhance data reduction, and using entropy thresholds to decide on the block size for compression or deduplication, thereby optimizing resource allocation and storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data compression and deduplication techniques are applied to data lacking redundancy, then storage efficiency is improved, but system performance deteriorates due to ineffective processing
Solution Approach 1:
The patent calculates entropy of data blocks before applying compression or deduplication operations. This preliminary assessment allows the system to identify data blocks with sufficient redundancy (low entropy) and avoid processing blocks with high entropy that would not benefit from these operations, thereby preventing performance degradation while maintaining storage efficiency
Solution Approach 2:
The patent introduces entropy as a decision parameter to dynamically determine whether to apply compression or deduplication. By changing the approach from blanket application of these techniques to selective application based on entropy thresholds, the system optimizes both storage efficiency and performance
2Ease of operation
If data blocks are processed without assessing redundancy likelihood, then processing simplicity is maintained, but resource allocation becomes inefficient
Solution Approach 1:
The patent replaces complex heuristic assessments of data redundancy with an entropy calculation-based decision mechanism. This substitution provides a mathematically rigorous yet computationally efficient method for determining whether data blocks are suitable for compression or deduplication, improving resource allocation without significantly complicating the processing workflow
Data Source
AI summary
A method is used in managing inline data compression and deduplication in storage systems. A block of data from data stored in a cache of a storage system is identified based on entropy. Entropy of the block of data is compared with a first threshold value. Based on the comparison, the block of data is either deduplicated or compressed without deduplication.