Batch Snapshot Deletion in Cloud Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud/object storage services face challenges in providing strong consistency for data backup and restore systems due to eventual consistency guarantees, high latency, and limited network bandwidth, making it difficult to manage data archiving and deletion efficiently.
Innovation Solution
Implementing local metadata staging and range-based deletion mechanisms, where metadata is staged on-premises and uploaded in fixed-sized chunks, allowing for efficient snapshot management and deletion by leveraging the range of data chunks referred to by each snapshot, and using batch-based deletion to optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If cloud/object storage services are used for data archiving, then scalability and storage capacity are improved, but consistency guarantees and network reliability deteriorate
Solution Approach 1:
The patent segments the metadata management by introducing version identifiers and dividing the metadata into multiple versions (current and previous). This allows the system to maintain consistency by tracking and managing different versions of metadata independently, resolving the contradiction between scalable storage and consistency guarantees.
Solution Approach 2:
The patent implements preliminary actions by pre-defining versioning strategies and deletion policies before data archiving operations. The system pre-establishes metadata version chains and deletion rules, ensuring consistency is maintained proactively rather than reactively, thus addressing the reliability concern while preserving storage scalability.
2Adaptability or versatility
If individual snapshot deletion is implemented, then data management flexibility is improved, but resource overhead and processing time worsen
Solution Approach 1:
The patent merges individual deletion operations into batch-based deletion processes. By combining multiple snapshot deletions into a single batch operation, the system reduces the cumulative overhead of metadata updates and network operations, thereby decreasing processing time while maintaining the flexibility to delete specific snapshots through selective batch composition.
Solution Approach 2:
The patent applies partial action by implementing range-based deletion that deletes only the necessary portion of snapshots within a specified range rather than requiring complete sequential processing. This approach reduces processing time by performing exactly the needed deletions without unnecessary overhead, balancing flexibility with efficiency.
3Quantity of substance
If metadata overwriting is used for snapshot updates, then storage space utilization is improved, but data consistency and recovery capability deteriorate
Solution Approach 1:
The patent implements preliminary versioning where previous metadata versions are preserved before overwriting. This preliminary action ensures that if overwriting causes consistency issues, previous versions can be recovered. The system pre-establishes version chains that maintain data consistency while allowing efficient storage through controlled overwriting of superseded versions.
Solution Approach 2:
The patent uses copying by creating and maintaining copies of metadata across different versions rather than directly overwriting. This copying mechanism preserves historical metadata states, enabling consistency verification and recovery while still achieving storage efficiency by allowing older copies to be garbage collected when no longer needed.
4Productivity
If network bandwidth is increased for faster backup, then productivity is improved, but cost and energy consumption worsen
Solution Approach 1:
The patent segments the backup data into fixed-size chunks with associated metadata. This segmentation allows for efficient parallel processing and optimized network utilization, improving backup productivity without requiring proportionally increased network bandwidth. The chunked approach enables better traffic management and reduced energy consumption per unit of data transferred.
Solution Approach 2:
The patent applies partial action by implementing incremental backup strategies that transfer only the necessary portions of data (new or modified chunks) rather than complete datasets. This reduces the total network bandwidth required and associated energy costs while maintaining high productivity for the actual backup operations performed.
Data Source
AI summary
According to one set of embodiments, a computer system can receive a request or command to delete a snapshot from among a plurality of snapshots of a dataset, where the plurality of snapshots are stored in cloud/object storage. In response, the computer system can add the snapshot to a batch of pending snapshots to be deleted and can determine whether the size of the batch has reached a threshold. If the size of the batch has not reached the threshold, the computer system return a response to an originator of the request or command indicating that the snapshot has been deleted, without actually deleting the snapshot from the cloud/object storage.