Dense Tree Data Structure for Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication techniques in storage systems suffer from substantial overhead due to the need to read and write entire fingerprint databases and lack indexing capabilities, leading to inefficient duplicate data removal.
Innovation Solution
The implementation of a dense tree data structure with multiple levels, allowing for efficient data deduplication by merging fingerprint entries between levels and facilitating real-time lookup and deduplication operations through indexing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a flat file fingerprint database is used for data deduplication, then the system can store and compare fingerprints, but the overhead of reading and writing the entire database for each deduplication operation is substantial
Solution Approach 1:
The patent divides the flat file fingerprint database into a hierarchical tree structure with multiple levels. Each node in the tree contains a subset of fingerprints, organized such that the root node contains all fingerprints and child nodes contain partitions of their parent's fingerprints. This segmentation allows the system to search only relevant portions of the database rather than reading the entire flat file, significantly reducing I/O overhead and improving deduplication operation efficiency.
2Ease of operation
If a flat file fingerprint database is used, then the database can be implemented simply, but there is no means to facilitate lookup operations within the file
Solution Approach 1:
The patent transitions from a one-dimensional flat file structure to a multi-dimensional hierarchical tree structure. Each level of the tree represents a dimension of organization, with nodes at different levels providing progressively finer partitioning of the fingerprint data. This dimensional transformation enables efficient lookup operations by allowing the system to navigate through hierarchical levels rather than scanning a flat file, while the tree structure itself provides the indexing mechanism needed for fast access.
Data Source
AI summary
In one embodiment, as new blocks of data are written to storage devices of a storage system, fingerprints are generated for those new blocks and inserted as entries into a top level (L0) of a dense tree data structure. When L0 is filled, the contents from L0 may be merged with level 1 (L1). After the initial merge, new fingerprints are added to L0 until L0 fills up again, which triggers a new merge. Duplicate fingerprints in L0 and L1 are identified which, in turn, indicates duplicate data blocks. A post-processing deduplication operation is then performed to remove duplicate data blocks corresponding to the duplicate fingerprints. In a different embodiment, as new fingerprint entries are loaded into L0, those new fingerprints may be compared with existing fingerprints loaded into L0 and/or other levels to facilitate inline deduplication to identify duplicate fingerprints and subsequently perform the deduplication operation.