Metadata Deduplication via Chunk Identifier Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face inefficiencies due to the storage of duplicative metadata, which wastes storage resources as they back up data from primary systems over time.
Innovation Solution
The implementation of a technique that deduplicates metadata by identifying and consolidating common chunk identifier sequences across multiple entries, using a modulo operation to determine chunk identifier markers and generating new data chunks for shared sequences, thereby reducing redundant storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the storage system stores all backup data and metadata over time, then the data can be located and recovered, but duplicative metadata consumes excessive storage resources
Solution Approach 1:
The patent merges identical chunk identifier sequences from multiple metadata entries into a single shared sequence. When duplicate sequences are detected, they are consolidated into one master sequence that multiple entries can reference, thereby reducing overall metadata storage while preserving complete data location information for recovery operations.
Solution Approach 2:
The patent creates a universal chunk identifier sequence structure that serves multiple metadata entries simultaneously. A single deduplicated sequence acts as a shared resource that can be referenced by multiple different data chunks and metadata entries, eliminating the need for each entry to store its own complete sequence and thus reducing storage resource consumption while maintaining universal access capability.
2Loss of information
If the storage system maintains complete metadata for each backup, then data location information is preserved, but redundant metadata increases storage overhead
Solution Approach 1:
The patent creates lightweight references or pointers to the shared chunk identifier sequences instead of storing complete duplicate sequences in each metadata entry. Each metadata entry contains a reference to the deduplicated sequence rather than the full sequence data, significantly reducing metadata storage overhead while preserving the ability to locate data through these references.
3Quantity of substance
If the storage system performs deduplication of metadata, then storage efficiency is improved, but the complexity of metadata management increases
Solution Approach 1:
The patent performs deduplication of chunk identifier sequences during the metadata generation process itself, before the metadata is fully written to storage. By identifying and consolidating duplicate sequences in advance, the system reduces the burden of managing redundant data later while maintaining relatively simple metadata management structures.
Data Source
AI summary
A first group of chunk identifiers associated with a first content identifier structure of a first metadata element and a second group of chunk identifiers associated with a second content identifier structure of a second metadata element are determined. A common sequence of chunk identifiers across at least a portion of the first group of chunk identifiers associated with the first content identifier structure and the second group of chunk identifiers associated with the second content identifier structure is determined. A portion of the first group of chunk identifiers associated with the first content identifier structure and a portion of the second group of chunk identifiers associated with the second content identifier structure is updated to reference a common sequence identifier in place of the determined common sequence of chunk identifiers.