Storage Deduplication Signature Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in determining whether new data chunks are duplicates or singles, leading to unnecessary comparisons and increased processing resources during deduplication, especially in handling collisions where the first portion of signatures matches but the second portion differs.
Innovation Solution
The proposed method uses a metadata storage system with a first and second portion of signatures, where the first portion addresses a metadata location, and if a collision occurs, the second portion is compared to determine if the data chunks are the same, thereby reducing the need for bit-by-bit comparisons and optimizing storage by avoiding frequent data chunk comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the storage system performs bit-by-bit comparison of data chunks to determine duplicates, then deduplication accuracy is improved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent segments the signature comparison process into two distinct stages: first comparing only the first portion of signatures (fast path), and only if they match, then comparing the second portion (slow path). This segmentation allows the system to quickly eliminate non-duplicate chunks without performing full comparisons, significantly reducing average processing time while maintaining accurate duplicate detection
Solution Approach 2:
The patent introduces an intermediary structure (the two-portion signature system with metadata storage) that mediates between the need for accurate duplicate detection and the need for fast processing. The first portion of signatures acts as a filtering intermediary, quickly identifying potential duplicates before the more expensive second portion comparison is performed
2Measurement precision
If the storage system stores complete signatures for all data chunks, then duplicate detection accuracy is improved, but metadata storage requirements increase
Solution Approach 1:
The patent segments the signature data into two portions with different storage requirements and access patterns. The first portion is stored in a compact metadata structure for rapid comparison, while the second portion is stored separately and only accessed when needed. This segmentation reduces the amount of data that must be continuously maintained and accessed in the hot path
Solution Approach 2:
The patent applies partial action by only performing the second portion of signature comparison when the first portion matches. This means the system does the minimum necessary work (first portion comparison) for most data chunks, and only does the more expensive second portion comparison for the small subset of chunks that are potential duplicates
3Reliability
If the storage system performs frequent data chunk comparisons to ensure deduplication accuracy, then reliability of duplicate detection is improved, but processing throughput decreases
Solution Approach 1:
The patent segments the comparison process into a fast filtering stage (first portion comparison) and a reliable verification stage (second portion comparison). This segmentation allows the system to maintain high throughput by quickly processing most chunks through the first stage, while ensuring reliability by performing the more accurate second stage comparison only when necessary
Solution Approach 2:
The patent applies partial action by performing only the necessary level of comparison for each data chunk. Most chunks require only the first portion comparison (partial action), while potential duplicates receive the full two-portion comparison (excessive action to ensure reliability). This differential approach maintains both throughput and reliability
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An example storage system may include storage media and a storage controller. The storage controller may be to establish virtual volumes, private data stores, and a deduplication data store, each being a virtual storage space of the storage media, wherein each of the private data stores is associated with one of the virtual volumes and the deduplication data store is shared among the virtual volumes. The storage controller may, in response to receiving input data that is to be stored in a given one of the virtual volumes, determine a signature for the input data and select between storing the input data in the private data store associated with the given one of the virtual volumes and storing the input data in the deduplication data store. The storage controller may select where to store the input data based on whether a metadata storage location addressed by a first portion of the signature includes valid metadata, and, if so: whether the metadata includes a back-pointer to one of the virtual volumes, and whether a second portion of the signature matches a key included in the metadata.