Hash-to-UID Table Duplicate Elimination in Storage Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage clusters face inefficiencies due to duplicate digital objects, which waste disk space, consume CPU time, and slow down processing as the number of files increases, especially when using hash values or random numbers as unique identifiers, making it difficult to eliminate duplicates without impacting input rates.
Innovation Solution
A technique that detects and eliminates duplicate digital objects by using a hash-to-UID table and page mapping table to identify and redirect unique identifiers, allowing for the deletion of duplicates while maintaining the validity of original identifiers, and storing records for future duplicate detection, thereby optimizing storage cluster efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If in-line elimination of duplicates using hash values is implemented, then duplicate elimination is achieved, but input rate to the cluster is reduced due to the need to calculate hash values using the entire object before storage
Solution Approach 1:
The system performs preliminary actions by calculating hash values and storing objects in a hash-to-UID table before actual storage occurs. This allows the cluster to recognize duplicates in advance and handle them appropriately, eliminating the need to recalculate hashes during storage operations and thus maintaining high input rates while achieving reliable duplicate elimination.
2Reliability
If multiple copies of the same digital object are stored in the cluster, then data redundancy is achieved, but disk space is wasted and CPU time is consumed
Solution Approach 1:
The system creates a copy of the hash value and stores it in the hash-to-UID table along with the unique identifier. This copy mechanism allows the system to track all references to the same digital object without storing multiple physical copies of the object itself, thereby maintaining data redundancy information while minimizing disk space consumption.
Solution Approach 2:
The system discards redundant copies of digital objects by identifying them through the hash-to-UID table and eliminating unnecessary duplicates. The unique identifier and hash value are recovered and reused to point to the original object, thereby freeing up disk space while maintaining the ability to access the object through its identifier.
3Measurement precision
If a hash value is calculated using the entire object to determine uniqueness, then accurate duplicate detection is achieved, but processing time increases and input rate decreases
Solution Approach 1:
The system performs the hash calculation as a preliminary action during the object storage process. By calculating the hash value once and storing it in the hash-to-UID table, the system avoids repeated hash calculations for duplicate detection, thereby achieving accurate duplicate detection while minimizing processing time and maintaining high input rates.
Data Source
AI summary
Digital objects within a fixed-content storage cluster use a page mapping table and a hash-to-UID table to store a representation of each object. For each object stored within the cluster, a record in the hash-to-UID table stores the object's hash value and its unique identifier (or portions thereof). To detect a duplicate of an object, a portion of its hash value is used as a key into the page mapping table. The page mapping table indicates a node holding a hash-to-UID table indicating currently stored objects in a particular page range. Finding the same hash value but with a different unique identifier in the table indicates that a duplicate of an object exists. Portions of the hash value and unique identifier may be used in the hash-to-UID table. Unneeded duplicate objects are deleted by copying their metadata to a manifest and then redirecting unique identifiers to point at the manifest.