Log-with-index structure for deduplication metadata management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data deduplication introduces complexities in managing large and frequently accessed databases, particularly in identifying and removing corrupted data patterns and maintaining efficient storage management, as existing methods lack effective reverse mapping information for deduplicated data structures.
Innovation Solution
The implementation of a log-with-index (LWI) data structure that includes a sequential log for inserting and deleting records, backed by a tree-based index, which supports atomic merging of tablets and maintains a tablet index using copy-on-write storage to preserve data integrity during merges, and employs deduplication keys to manage data deduplication transactions efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is implemented to reduce storage capacity, then storage efficiency is improved, but the complexity of managing reverse mapping information increases
Solution Approach 1:
The patent segments the reverse mapping information into fixed-size tablets that are stored in a tablet library. Each tablet contains a portion of the deduplication metadata, allowing the system to manage large datasets by dividing the reverse mapping information into manageable, independently addressable units. This segmentation reduces the complexity of managing entire datasets while maintaining deduplication efficiency.
Solution Approach 2:
The patent introduces a new dimensional organization by implementing a tablet library with a specific data structure that adds an intermediate layer between the deduplication data and the reverse mapping information. This dimensional change allows efficient access to reverse mapping data through a structured hierarchy, reducing the management complexity while preserving storage efficiency.
2Productivity
If a log structure is used for sequential insertion of metadata, then insertion performance is improved, but querying and access efficiency deteriorates
Solution Approach 1:
The patent merges the advantages of both log structures and index structures by combining sequential logging for insertions with indexed tablets for efficient querying. The log structure maintains insertion performance through sequential writes, while the indexed tablet library provides efficient random access for queries, achieving both high insertion performance and querying efficiency simultaneously.
3Stability of the object's composition
If tablets are merged frequently to maintain data organization, then data organization is improved, but the risk of data loss during merges increases
Solution Approach 1:
The patent applies preliminary actions by performing validation checks and creating backup references before executing tablet merge operations. The system verifies data integrity and maintains transaction logs that can recover from merge failures, reducing the risk of data loss while maintaining organized data structure through frequent merges.
4Quantity of substance
If the tablet library grows to accommodate large datasets, then data capacity is improved, but access time and processing overhead increase
Solution Approach 1:
The patent segments the growing tablet library into fixed-size, independently addressable units with a hierarchical structure. This segmentation allows the system to access specific tablets without scanning the entire library, maintaining constant or logarithmic access time even as data capacity grows to accommodate large datasets.
Data Source
AI summary
An information handling system maintains a log-with-index (LWI) structure that includes a log for inserting and deleting records, each of which includes a key, and a binary index for maintaining an index of log records. The log may be stored to disk for persistence. Insertions of new records into the log may occur sequentially to ensure adequate performance. The index tree may be maintained in memory for performance. When the log is full, log records may be written in key-sorted order to a new tablet in a tablet library. Two tablets may be merged from time to time. Merging may include iteratively performing a plurality of atomic merges for each of a plurality of atomic portions. During atomic merges, tablet index data may be modified in copy-on-write fashion to preserve existing data until the atomic merge completes.