Two-Stage Extent Map Database for Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication techniques in storage systems suffer from substantial overhead due to the need to read and write entire fingerprint databases, and lack efficient lookup mechanisms, leading to increased storage costs and resource utilization.
Innovation Solution
Implementing a B+ tree data structure for the extent map database, where each extent ID is divided into checksum and duplicate bits, allowing for efficient identification and removal of duplicate data blocks through hashing and incremental duplicate bit tracking, along with a two-stage update process and partial key cuckoo hash lookup interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a flat file fingerprint database is used for data de-duplication, then the database structure is simple, but substantial overhead occurs due to reading and writing entire databases for each de-duplication operation
Solution Approach 1:
The patent segments the fingerprint database into a hierarchical B+ tree structure with internal nodes and leaf nodes. This segmentation allows the system to access only relevant portions of the database during de-duplication operations rather than reading the entire flat file, thereby improving operational efficiency while maintaining manageable complexity through structured organization.
Solution Approach 2:
The patent transitions from a one-dimensional flat file structure to a multi-dimensional B+ tree structure with multiple levels (root node, internal nodes, leaf nodes). This dimensional change enables efficient indexing and selective access to fingerprint entries, reducing the I/O overhead associated with traversing entire databases during de-duplication operations.
2Ease of manufacture
If a flat file fingerprint database is used, then implementation is straightforward, but no efficient lookup mechanisms are available within the file
Solution Approach 1:
The patent implements preliminary action by pre-organizing fingerprint entries into a B+ tree structure with calculated checksums stored in leaf nodes. This preliminary organization enables efficient lookup operations during de-duplication, as the system can directly navigate to relevant entries using the tree structure rather than scanning through a flat file, thereby improving ease of operation while maintaining implementation feasibility.
3Reliability
If the entire fingerprint database is read and overwritten for each de-duplication operation, then comprehensive de-duplication is achieved, but substantial storage overhead and time consumption occur
Solution Approach 1:
The patent extracts only the necessary portions of the fingerprint database during de-duplication operations by utilizing the B+ tree structure to navigate directly to relevant leaf nodes containing checksums. This extraction approach maintains de-duplication completeness by accessing all necessary fingerprint entries while avoiding the time-consuming process of reading and writing entire databases, thereby reducing operational time while preserving reliability.
4Reliability
If duplicate data is stored at multiple locations, then data redundancy is maintained, but storage space consumption increases and costs rise
Solution Approach 1:
The patent implements feedback mechanisms through checksum calculation and comparison in the B+ tree structure. When new data is written, the system calculates its checksum and queries the B+ tree to check for existing duplicates. This feedback loop enables the system to identify and eliminate duplicate data blocks, maintaining data redundancy for reliability while preventing unnecessary storage space consumption by storing only unique data blocks.
Data Source
AI summary
Multiple key-value stores may be employed to smooth out random updates (based on the extent ID) to the EMAP database. The updates to the EMAP database occur in a two-stage manner: (i) using an append-only log store for the first stage and (ii) using an on-disk hash store for the second stage. The append-only log store is used to convert the random updates to sequential write operations on the EMAP database. Once full, the contents of the log store are sorted and moved to the on-disk hash store, which holds the updates for a transient period of time to enable batching of the updates. Once sufficient batching of the extent map entries are accumulated, those entries are sorted and moved to the EMAP database. Thereafter, the EMAP database can be scanned to find extent map entries having identical checksum bits to perform data deduplication.