Block-Data Clustering Deduplication via Locality Hashes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication techniques in storage systems are inefficient due to their inability to effectively eliminate duplicate data copies and manage data variations, leading to suboptimal storage and processing performance.
Innovation Solution
An information handling system that employs a deduplication agent to calculate cryptographic and locality-preserving hashes for data blocks, performs clustering based on these hashes, and stores data blocks as cluster members with references to cluster centers and lists of differences, allowing for efficient deduplication and data manipulation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data deduplication techniques are used to eliminate duplicate data copies, then storage space is reduced, but storage overhead increases and processing efficiency decreases due to inability to effectively manage data variations
Solution Approach 1:
The patent segments data into fixed-size blocks and further divides each block into variable-length runs of identical bytes. This hierarchical segmentation allows the system to identify and deduplicate data at multiple levels, reducing storage overhead while maintaining processing efficiency by working with smaller, more manageable units.
Solution Approach 2:
The patent changes the parameter representation by using run-length encoding to transform sequences of identical bytes into compact (byte, count) pairs. This parameter transformation enables more efficient storage and comparison of data blocks, directly addressing the contradiction between storage space reduction and processing efficiency.
2Reliability
If data blocks are stored as complete copies to ensure data integrity and retrieval, then data reliability is maintained, but storage overhead increases
Solution Approach 1:
The patent uses selective copying by storing only the run-length encoded representation of data blocks rather than complete copies. When a data block is retrieved, the system reconstructs it from the stored runs, maintaining data integrity while significantly reducing storage overhead through this partial copying approach.
Solution Approach 2:
By transforming data blocks into run-length encoded parameter sequences, the patent enables more efficient storage of data representations. This parameter change allows the system to maintain full data integrity through reversible transformation while reducing the actual storage overhead by eliminating redundant byte sequences.
3Measurement precision
If clustering is performed on all data blocks to improve deduplication accuracy, then deduplication effectiveness increases, but computational effort increases
Solution Approach 1:
The patent segments data blocks into runs of identical bytes before clustering, reducing the amount of data that needs to be processed during clustering operations. This segmentation allows the system to achieve accurate deduplication by comparing compressed representations rather than full data blocks, thereby reducing computational effort while maintaining deduplication effectiveness.
Solution Approach 2:
The patent transforms data blocks into run-length encoded parameters before performing clustering operations. This parameter transformation reduces the computational complexity of clustering by working with smaller, compressed representations of the data, thereby achieving accurate deduplication with reduced computational effort and energy consumption.
Data Source
AI summary
A method may include calculating a cryptographic hash for each of a plurality of data blocks stored on a storage resource, calculating a locality-preserving hash for each of the plurality of data blocks, and performing clustering based on the locality-preserving hashes. The method may also include, for each particular cluster setting a cluster center data block for the particular cluster and for each particular data block of the particular cluster, storing the particular data block as a cluster member data block including a reference to the cluster center data block for the particular cluster, a list of differences of the particular data block from the cluster member data block, the cryptographic hash for the particular data block to identify the particular data block, and the locality-preserving hash for the particular data block for manipulation of the particular data block.