Deduplication Data Structure Reducing Metadata Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication methods require significant storage for metadata, leading to storage overhead and performance issues due to the inclusion of complete hash values in block metadata, especially at high deduplication ratios, and raise security concerns with SHA-1 being cast into doubt.
Innovation Solution
A new data structure design for deduplication that eliminates the need for storing complete hash values in block metadata by using probabilistically unique segment identifiers and references, directly referencing containers in the deduplication index, reducing storage requirements and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete hash values are stored in block metadata, then data deduplication can be achieved, but metadata storage size increases significantly
Solution Approach 1:
The patent extracts only the necessary portion of the hash value (first 16 bits) for storage in block metadata, while storing the complete hash value separately in a hash table. This separation allows deduplication functionality to be maintained with minimal metadata overhead, resolving the contradiction between deduplication accuracy and metadata size.
Solution Approach 2:
The patent moves the complete hash value storage to a different dimension (separate hash table structure) rather than embedding it directly in block metadata. This dimensional separation enables the system to maintain full hash functionality while keeping metadata compact, effectively resolving the storage size issue.
2Reliability
If SHA-2 hash is used instead of SHA-1, then security is improved, but metadata storage requirement increases to 33%
Solution Approach 1:
The patent extracts only the essential 16 bits of the SHA-2 hash value for block metadata storage, while storing the complete hash separately. This extraction approach allows using the more secure SHA-2 algorithm without proportionally increasing metadata storage requirements, thus resolving the contradiction between security and storage percentage.
3Measurement precision
If conventional deduplication index with complete hash values is used, then data segment identification is accurate, but I/O performance deteriorates due to indirection
Solution Approach 1:
The patent extracts a compact 16-bit signature from the complete hash value for use in block metadata. This extracted signature is sufficient for accurate data segment identification while being much more compact, thereby improving I/O performance by reducing the amount of data that needs to be read and processed, while maintaining identification accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention relates to an apparatus (100) for storing received data blocks (300) as deduplicated data blocks (104), the apparatus (100) being configured to maintain a plurality of containers (101), wherein a reference (105) to a container is unique within the apparatus (100) and each container (101) includes one or more data segments (102) and segment metadata (103) for each data segment (102), the segment metadata (103) including a segment identifier (110) and a segment reference (108), wherein the segment identifier (110) is unique within the container (101) and the segment reference (108) is unique within the apparatus (100), maintain a plurality of deduplicated data blocks (104) storing received data blocks (300), wherein each deduplicated data block (104) includes a plurality of identified container references (105), wherein a container reference identifier (105i) is unique within the deduplicated data block (104), and an ordered list of one or more segment indicators (106).