Deduplication Reference Mapping With Log-Indexed Tablet Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data deduplication introduces complexities in managing large and frequently accessed databases, particularly in identifying and handling corrupted data patterns and maintaining efficient storage management, as existing methods lack effective reverse mapping and efficient data reference identification.
Innovation Solution
The implementation of a log-with-index (LWI) data structure that includes a processor-executable program for maintaining a binary tree index and sequential insertion/deletion of records, with features like atomic merging of tablets and copy-on-write storage to ensure data integrity and performance, along with data deduplication methods that use deduplication keys to manage and merge data efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is implemented to reduce storage capacity, then storage efficiency is improved, but system complexity increases due to N:1 ratio between data references and data patterns
Solution Approach 1:
The patent introduces reverse mapping information as an intermediary data structure that maps data patterns back to their references. This intermediary enables efficient identification of all references to a corrupted data pattern without scanning every reference, thus reducing system complexity while maintaining deduplication benefits
Solution Approach 2:
The patent replaces the mechanical scanning approach (linear search through all references) with an indexed lookup system (reverse mapping). This substitution transforms an O(n) operation into an O(1) operation, significantly reducing computational complexity for identifying corrupted data references
2Reliability
If reverse mapping information is maintained for each data pattern, then data integrity is improved by enabling efficient corruption identification, but storage overhead increases
Solution Approach 1:
The patent merges reverse mapping information with the existing deduplication metadata structure. By combining these functions into a unified data structure, the patent reduces total storage overhead while maintaining both deduplication capabilities and corruption identification functionality
Solution Approach 2:
The reverse mapping structure serves multiple functions: it enables efficient corruption identification, supports reference counting for data pattern lifecycle management, and provides a foundation for various query operations. This multi-functionality reduces the need for separate data structures, thereby reducing overall storage overhead
3Reliability
If scanning of every data reference is performed to identify corrupted data patterns, then data integrity is maintained, but processing time increases significantly
Solution Approach 1:
The patent performs preliminary organization of reference information into reverse mapping structures during normal deduplication operations. This preliminary action ensures that when corruption detection is needed, the system can immediately query the pre-organized indexes rather than scanning all references, dramatically reducing processing time
Data Source
AI summary
A data deduplication method includes detecting a deduplication transaction including a data pattern associated with a data pattern address (DPA) and a reference, to the pattern, associated with a data reference address (DRA). A deduplication key may be determined based on the DPA and the DRA by concatenating the DPA and the DRA with the DPA as the most significant bits. The key may be stored in a key field of a record in a persistent and sequentially-accessed log, which is part of a log-with-index (LWI) structure that also maintains, in RAM or SSD, a binary index of the log records. When full, the log is cleared by writing the records in key-sorted order to the new tablet. From time to time, two tablets in the tablet library are merged. Tablet merging may include two or more atomic merges, each atomic merge corresponding to a portion of the tablet.