Backup Metadata Range Indexing for Remote Object Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems face challenges in efficiently deduplicating data across remote storage systems, particularly in disaster recovery scenarios, due to issues with data consistency and the risk of overwriting existing object names, especially in eventually consistent storage systems.
Innovation Solution
A deduplication system that generates and manages deduplication metadata, including a sparse index and container indexes, to identify and store unique chunks of data, and uses hash values in object names to ensure uniqueness, while replicating metadata to a remote object storage system for efficient recovery and to prevent overwriting of existing data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If data is stored in remote backup system with eventual consistency, then storage capacity and accessibility are improved, but data consistency and risk of overwriting increase
Solution Approach 1:
The system performs preliminary actions by generating and storing deduplication metadata (including sparse indexes and container indexes) before actual data backup operations. This metadata contains information about used object names and chunk locations, enabling the system to prevent overwriting and maintain consistency in eventually consistent storage systems.
Solution Approach 2:
The system implements feedback mechanisms by continuously tracking and recording used object names in the deduplication metadata. Before storing new data chunks, the system queries the metadata to check for name conflicts, creating a feedback loop that ensures data consistency and prevents overwriting in remote backup systems.
2Reliability
If deduplication metadata is replicated to remote storage, then disaster recovery capability is improved, but network bandwidth consumption increases
Solution Approach 1:
The system extracts only the essential deduplication metadata (sparse indexes and container indexes) for replication to remote storage, rather than replicating entire data sets. This selective extraction minimizes network bandwidth consumption while maintaining disaster recovery capability, as the metadata is sufficient to reconstruct data relationships without requiring full data duplication.
3Reliability
If hash values are used in object names, then object name uniqueness is improved, but computational overhead increases
Solution Approach 1:
The system changes the parameter of object names by incorporating hash values (such as SHA-1 or MD5 hashes) into the naming scheme. This transformation ensures global uniqueness of object names across distributed storage systems. The computational overhead is acceptable because hash functions are efficient and the uniqueness guarantee enables reliable deduplication and prevents naming conflicts in distributed environments.
Data Source
AI summary
In some examples, in response to an event at the deduplication system, a system accesses item metadata of a backup item that is backed up to a remote object storage system, the item metadata of the backup item including range information indicating a range of identifier values for portion objects of the backup item stored in the remote object storage system. The system issues, based on the range information, requests to obtain respective attribute information of the portion objects of the backup item stored in the remote object storage system. The system determines, based on the attribute information, a name of a given portion object of the backup item already used.