Fingerprint Datastore Segmentation for Stale Entry Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication systems face inefficiencies due to the large size of fingerprints datastores, which consume more time and resources, especially when dealing with stale entries and the need to overwrite the entire datastore during each operation, leading to increased write costs and customer response time issues.
Innovation Solution
The system divides the fingerprints datastore into a primary and secondary datastore, with the primary containing unique fingerprints and the secondary containing identical fingerprints, allowing for efficient identification and removal of duplicate blocks and stale entries, and manages verify operations as background processes to reduce downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the fingerprints datastore is sorted and overwritten with stale-free entries using conventional verify operations, then stale entries are removed, but the entire datastore must be sorted twice and overwritten completely, consuming excessive time and resources
Solution Approach 1:
The fingerprints datastore is divided into multiple segments, allowing the verify operation to process only the segment containing stale entries rather than the entire datastore. This segmentation enables targeted verification, reducing the time and resources needed while maintaining complete stale entry removal.
Solution Approach 2:
Instead of performing a complete sort and overwrite of the entire fingerprints datastore, the system performs partial verification by identifying and processing only the portion of the datastore that contains stale entries. This partial action achieves the necessary cleanup without the overhead of full datastore processing.
2Reliability
If the fingerprints datastore is sorted twice during verify operations, then stale entries can be identified and removed, but the double sorting increases operational complexity and time consumption
Solution Approach 1:
By segmenting the fingerprints datastore, the system eliminates the need for double sorting. Only the specific segment containing stale entries requires sorting, while other segments remain in their original order. This reduces operational complexity from two full sorts to one partial sort.
Solution Approach 2:
The system performs preliminary identification of stale entries using metadata or reference information before executing the verify operation. This preliminary action allows the system to target only the necessary portion of the datastore, avoiding unnecessary sorting operations and reducing overall complexity.
3Reliability
If the entire fingerprints datastore is overwritten with stale-free entries, then data consistency is maintained, but write costs increase significantly
Solution Approach 1:
The fingerprints datastore is segmented so that only the portion containing stale entries is overwritten, rather than the entire datastore. This selective overwrite maintains data consistency in the affected region while dramatically reducing the total write volume and associated energy costs.
Solution Approach 2:
Instead of overwriting the entire datastore, the system identifies and discards only the stale entries while preserving the valid entries. This approach recovers write capacity by maintaining the majority of the datastore without modification, reducing energy consumption while ensuring consistency.
4Reliability
If verify operations are performed as blocking operations, then data consistency is ensured, but de-duplication operations cannot execute simultaneously, reducing system productivity
Solution Approach 1:
By segmenting the fingerprints datastore and performing verify operations on individual segments rather than the entire datastore, multiple verify operations can execute in parallel with de-duplication operations on different segments. This maintains data consistency within each segment while enabling concurrent operations across the system, thereby increasing overall productivity.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
A storage server is coupled to a storage device that stores blocks of data, and generates a fingerprint for each data block stored on the storage device. The storage server creates a fingerprints datastore that is divided into a primary datastore and a secondary datastore. The primary datastore comprises a single entry for each unique fingerprint and the secondary datastore comprises an entry having an identical fingerprint as an entry in the primary datastore. The storage server merges entries in a changelog with the entries in the primary datastore to identify duplicate data blocks in the storage device and frees the identified duplicate data blocks in the storage device. The storage server stores the entries that correspond to the freed data blocks to a third datastore and overwrites the primary datastore with the entries from the merged data that correspond to the unique fingerprints to create an updated primary datastore.