Range-Based Snapshot Deletion in Cloud Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud/object storage services face challenges in providing strong consistency for data backup and restore systems due to eventual consistency guarantees, high latency, and limited network bandwidth, making it difficult to efficiently archive and manage large volumes of data.
Innovation Solution
Implementing local metadata staging and range-based deletion mechanisms, along with batch-based deletion, to efficiently manage snapshot data and metadata in cloud/object storage, ensuring consistent data views and optimizing bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If cloud/object storage services are used for data archival, then storage scalability and cost-effectiveness are improved, but data consistency and reliability deteriorate due to eventual consistency guarantees
Solution Approach 1:
The patent introduces an intermediary consistency verification mechanism that mediates between the cloud storage's eventual consistency model and the backup system's requirement for strong consistency. The system uses metadata staging and verification protocols to ensure data integrity without requiring changes to the underlying cloud storage consistency model.
2Quantity of substance
If large volumes of data are archived to cloud storage, then storage scalability is improved, but network bandwidth consumption and latency increase
Solution Approach 1:
The patent segments the data archival process into multiple components: metadata operations are separated from data operations, and deletion operations are segmented into marking and actual removal phases. This segmentation allows optimization of network usage by minimizing metadata transfers and batching data operations.
Solution Approach 2:
The system performs preliminary actions by staging metadata locally before actual data operations. Deletion markers are prepared and validated before initiating data removal, reducing unnecessary network traffic and optimizing bandwidth utilization during actual data transfers.
3Adaptability or versatility
If individual snapshot deletion is implemented, then data management flexibility is improved, but system complexity and operational overhead increase
Solution Approach 1:
The patent merges individual deletion operations with batch deletion capabilities. The same metadata-driven deletion mechanism used for individual snapshots is extended to handle batches efficiently, allowing the system to maintain flexibility for individual operations while gaining the efficiency of batch processing without increasing complexity.
Data Source
AI summary
According to one set of embodiments, a computer system can receive a request/command to delete a snapshot from among a plurality of snapshots of a dataset that are stored in cloud/object storage. In response, the computer system can determine a first minimum chunk identifier (ID) and a first maximum chunk ID of a range of data chunks referred to by the first snapshot, and a second minimum chunk ID and a second maximum chunk ID of a range of data chunks referred to by an immediate child snapshot of the first snapshot. The computer system can then, for each data chunk identified by a chunk ID spanning from the second minimum chunk ID to the first maximum chunk ID, check whether the data chunk is referred to by the immediate child snapshot and if not, delete the data chunk from the cloud/object storage.