Three-Level Metadata Hierarchy for Data Deduplication Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face inefficiencies in index search as the number of unique blocks increases, requiring significant RAM and compromising performance, and lack transactional security in metadata management.
Innovation Solution
A three-level hierarchy is implemented, where Level 1 stores metadata on disk with user data, Level 2 uses low-latency storage like RAM and SSDs for caching, and Level 3 employs a Trie data structure in RAM to organize fingerprints, optimizing memory usage and performance while ensuring transactional security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional index search methods are used in data deduplication systems, then the system can handle basic deduplication operations, but memory usage increases significantly and performance degrades as the number of unique blocks increases
Solution Approach 1:
The patent segments the index search problem into two distinct parts: a filter index stored in RAM that provides quick rejection of duplicates, and a persistent index stored on disk that provides complete deduplication verification. This segmentation allows the system to maintain high performance with reduced memory usage by only loading essential filter data into RAM while storing the complete index on disk.
Solution Approach 2:
The patent introduces a new dimension of storage hierarchy by moving from a single-layer RAM-based index to a multi-layer architecture spanning RAM and disk storage. This dimensional change enables the system to leverage both the speed of RAM and the capacity of disk, resolving the contradiction between performance and memory usage.
2Speed
If more RAM is allocated for metadata storage, then search performance improves, but the system loses transactional security and metadata may be lost on system crashes
Solution Approach 1:
The patent implements preliminary action by pre-computing and storing filter indexes in RAM before they are needed for search operations, while maintaining the ability to reconstruct the complete index from persistent disk storage. This allows the system to have fast search performance with minimal RAM while ensuring that metadata can be recovered from disk if the system crashes.
Solution Approach 2:
The patent introduces an intermediary persistent storage layer between RAM and the complete index, acting as a mediator that provides both speed benefits (through RAM caching) and reliability benefits (through disk persistence). This intermediary layer resolves the contradiction by allowing the system to leverage both fast storage and secure persistence.
Data Source
AI summary
A method and system of optimizing the memory usage and performance of data deduplication storage systems includes organizing the metadata of data blocks needed by deduplicating storage systems. A three level hierarchy is used. Level 1 stores the metadata on disk along with the user data. Level 2 uses low latency storage (e.g. RAM and Solid State Disks) to cache the on-disk meta data for faster direct access. Level 3 organizes the fingerprints using a Trie and is entirely resident in RAM. Thus, the search, to determine whether a data block is unique or not and a candidate for transfer, can be more efficiency executed and to ensure that the meta data is transactionally secure.