Data Deduplication Using Wavelet Band Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data deduplication methods in storage systems are inefficient in identifying and reducing redundant data, especially in cloud storage environments, where they lack insight into file structures and rely on raw block comparisons, leading to suboptimal storage capacity utilization and increased costs.
Innovation Solution
The method involves segmenting data into bands using wavelet transforms like Haar or Daubechies 5/3, removing non-identical bands to enhance matching likelihood, and applying hash functions to structurally identical bands for deduplication, storing only the transformed data and hash indices in a database, which allows for efficient data reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional block-based deduplication is used, then the system operates without file system structure insight, but deduplication effectiveness is reduced
Solution Approach 1:
The patent segments data into fixed-size blocks and further divides each block into variable-length runs of identical bytes. This hierarchical segmentation allows the system to identify redundant data at multiple levels: block level for structural matching and run level for precise duplicate detection, thereby improving deduplication effectiveness without requiring complex file system structure analysis
Solution Approach 2:
The patent changes the parameter representation by transforming raw block data into run-length encoded formats. Instead of comparing raw byte sequences, the system compares structured run representations that capture both the value and length of identical byte sequences. This parameter transformation enables more effective duplicate detection while maintaining a relatively simple system architecture
2Productivity
If all original data is stored for post-process deduplication, then deduplication can be performed, but storage capacity is increased
Solution Approach 1:
The patent extracts only the essential components needed for deduplication: run-length encoded representations of data blocks and their corresponding hash values. Instead of storing complete original data, the system stores compressed run structures that can be used for both deduplication identification and potential data reconstruction, significantly reducing the quantity of stored data while maintaining full deduplication functionality
Solution Approach 2:
The patent creates virtual copies of data through run-length encoded representations rather than storing physical duplicates. The run structure serves as a compact representation that can be used to reconstruct original data when needed, allowing the system to reference data efficiently without maintaining large quantities of actual stored data
3Adaptability or versatility
If fixed-length segments are used, then data is divided uniformly, but structural similarities are missed
Solution Approach 1:
The patent applies hierarchical segmentation: first dividing data into fixed-size blocks for uniform processing, then further segmenting each block into variable-length runs of identical bytes. This two-level segmentation strategy enables the system to recognize structural similarities at the run level while maintaining the simplicity of fixed-block processing, capturing both uniform and variable patterns in the data
Solution Approach 2:
The patent introduces dynamic element length within the run structure. While blocks are fixed-size, the runs within blocks have variable lengths that adapt to the actual data patterns. This dynamic length adjustment allows the system to efficiently represent both long sequences of identical bytes and short variations, improving structural recognition without requiring complex adaptive block sizing
4Adaptability or versatility
If variable-length segments are used, then structural similarities are detected, but processing overhead increases
Solution Approach 1:
The patent uses hierarchical segmentation where fixed-size blocks are first processed uniformly, then each block is segmented into variable-length runs. This approach limits the complexity of variable-length processing to within-block operations only, rather than requiring variable-length blocks throughout the entire data set, thereby reducing overall processing overhead while maintaining structural detection capability
Solution Approach 2:
The patent applies variable-length characteristics locally within blocks rather than globally across the entire data set. Each block is processed with uniform fixed-size boundaries, but internally contains variable-length runs that capture local structural patterns. This localized application of variable-length segmentation reduces processing overhead compared to global variable-length approaches while preserving structural similarity detection
Data Source
AI summary
A method for data deduplication includes the following steps. First, segmenting an original data set into a plurality of data segments. Next, transforming the data in each data segment into a transformed data representation that has a band-type structure for each data segment. The band-type structure includes a plurality of bands. Next, selecting a first set of bands, grouping them together and storing them with the original data set. The first set of bands includes non-identical transformed data for each data segment. Next, selecting a second set of bands and grouping them together. The second set of bands includes identical transformed data for each data segment. Next, applying a hash function onto the transformed data of the second set of bands and thereby generating transformed data segments indexed by hash function indices. Finally, storing the hash function indices and the transformed data representation of one representative data segment in a deduplication database.