Self-Describing Object Names for Deduplication Metadata Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication systems face challenges in efficiently storing and updating deduplication metadata in immutable object storage systems, such as cloud object storage, which limits disaster recovery and data reconstruction capabilities.
Innovation Solution
The system utilizes deduplication metadata stored locally and stores backup data and metadata in cloud object storage with self-describing object names to enable efficient reconstruction and restore of deduplicated data streams, employing sparse indexes and virtual container indexes for efficient chunk matching and data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deduplication metadata is stored in immutable object storage systems, then data durability and cloud backup capability are improved, but the ability to update and reconstruct metadata efficiently deteriorates
Solution Approach 1:
The deduplication metadata is divided into multiple versioned snapshots stored in immutable object storage. Each snapshot represents a point-in-time state that can be independently retrieved and applied, enabling efficient reconstruction without requiring updates to existing metadata structures.
Solution Approach 2:
Multiple versions of deduplication metadata are pre-stored in the immutable object storage system. When reconstruction is needed, the appropriate pre-stored version can be directly retrieved and applied without performing complex update operations, thus maintaining both reliability and productivity.
2Adaptability or versatility
If local deduplication metadata is lost, then disaster recovery capability is tested, but data reconstruction time and complexity increase
Solution Approach 1:
The system maintains copies of deduplication metadata in both local storage and remote immutable object storage. When local metadata is lost, the remote copy serves as a backup that can be retrieved and applied to reconstruct the deduplication state, enabling rapid disaster recovery without lengthy reconstruction processes.
Solution Approach 2:
Backup copies of deduplication metadata are pre-stored in remote object storage before any potential data loss occurs. This preliminary action ensures that when disaster strikes, the metadata is already available for immediate retrieval and application, minimizing reconstruction time and complexity.
3Speed
If all deduplication metadata is stored locally, then access speed is improved, but system reliability and backup capability deteriorate
Solution Approach 1:
Deduplication metadata is segmented into local working copies and remote backup copies. The local copies provide fast access for active deduplication operations, while the remote copies in immutable object storage provide reliable backup capability. This segmentation allows the system to simultaneously achieve high access speed and strong reliability.
Solution Approach 2:
The system uses an intermediary architecture where local metadata storage serves as a fast-access cache for active operations, while remote object storage serves as a reliable backup repository. The deduplication system can switch between local and remote metadata sources depending on operational needs, optimizing both speed and reliability.
Data Source
AI summary
Examples include the acquisition of objects names for portion index objects. Some examples include acquisition, from a remote object storage system, of a list of object names for a plurality of portion index objects, stored in the remote object storage system. In some examples, for each of the portion index objects, the acquired object name includes an identifier of an associated deduplicated backup item and information identifying a data range of the associated deduplicated backup item that is represented by metadata of the portion index object.