Sparse Hash Indexing for Online Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data indexing methods in storage systems face challenges in efficiently handling metadata updates, particularly in low-performing media, leading to performance degradation due to random IO operations, and fail to effectively reduce duplication without impacting system performance.
Innovation Solution
A method involving dividing data into large blocks, calculating strong hash values for small blocks, selecting representative hash values, and compiling a sparse index to enable deduplication without changing underlying data, allowing for online deduplication with reduced disk access and network traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If metadata is stored in low-performance media such as spinning disk or SSD, then storage capacity is increased, but random IO performance deteriorates significantly
Solution Approach 1:
The patent segments metadata into two distinct layers: a compact sparse index layer stored in fast-access memory for quick lookups, and a full metadata layer stored in low-performance media for complete data. This segmentation allows the system to maintain high-speed access for common operations while preserving storage capacity in low-performance media.
Solution Approach 2:
The patent introduces a prediction filter as an intermediary layer between the sparse index and the full metadata. The prediction filter anticipates metadata access patterns and pre-loads relevant data, reducing the frequency of slow random IO operations to the underlying storage media while maintaining accurate similarity detection.
2Productivity
If prediction filter is used to increase efficiency of metadata access, then successful similarity lookup probability is improved, but performance in hotspot scenarios deteriorates sharply
Solution Approach 1:
The patent implements a dynamic metadata update mechanism where the sparse index and prediction filter are automatically adjusted based on access patterns. When hotspot scenarios are detected (repeated overwrites of the same data segments), the system dynamically adapts by updating only the necessary portions of the sparse index and invalidating relevant prediction filter entries, maintaining performance stability throughout workload variations.
3Quantity of substance
If layered indexing is used to group data and metadata by time and location, then metadata footprint is reduced, but performance impact in overwrite scenarios is not reduced
Solution Approach 1:
The patent extracts only the essential identifying elements (hash values and location pointers) from the full metadata and stores them in the sparse index layer. This extraction creates a minimal representation that enables fast lookups without storing complete metadata, significantly reducing the metadata footprint in fast-access memory while maintaining the ability to handle overwrites efficiently through selective updates.
Data Source
AI summary
A computer-implemented method for indexing a data item in a data storage system includes: dividing the data item into one or more large blocks; dividing each large block into one or more small blocks; calculating a strong hash value for each of the small blocks and storing a list of strong hash values with a pointer to a location of the large block; from the list of strong hash values calculated for each large block, selecting one or more representative hash values for the large block; and compiling a sparse index including an entry for each large block. Each entry is based on the representative hash values and a pointer to the list of strong hash values for each large block.