Cloud Backup Metadata Reuse for Failed Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based storage systems face inefficiencies and increased costs due to re-initiating failed backup operations, which require similar resource allocation as new backups, leading to unnecessary data transfer and storage expenses.
Innovation Solution
A system that leverages metadata to identify and reuse non-committed backup data from failed operations, reducing the amount of data transferred during subsequent backups by committing retained data from object storage, thus optimizing backup efficiency and lowering costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a failed backup operation is re-initiated, then the backup can be completed, but resource requirements and storage costs increase significantly
Solution Approach 1:
The system performs preliminary identification of successfully transferred backup data before re-initiating a failed backup operation. By examining metadata and tracking which data objects were successfully transmitted to cloud storage during the failed backup, the system prepares a list of data to be excluded from the retry, thereby avoiding redundant transfers and reducing resource consumption.
Solution Approach 2:
The system applies different handling strategies to different portions of backup data based on their transfer status. Successfully transferred data is identified and marked for exclusion, while only the failed portions are re-transmitted. This localized approach ensures that resources are consumed only where necessary, rather than re-transmitting entire backup sets.
2Loss of energy
If incremental backups are used to limit data transfer, then storage costs are reduced, but managing failed backup data becomes more complex
Solution Approach 1:
The system introduces metadata as an intermediary layer between the backup data and the management system. This metadata tracks the transfer status of each data object, enabling the system to identify successfully transferred data without complex analysis of the actual backup data. The metadata serves as a simple index that reduces the complexity of managing incremental backup failures.
Solution Approach 2:
The backup data is divided into discrete data objects, each with its own transfer status tracked in metadata. This segmentation allows the system to handle individual objects independently during failed backup recovery, rather than treating the entire backup as a single unit. Each object can be identified, tracked, and re-transmitted separately if needed.
3Reliability
If cloud storage retention policies preserve transmitted data, then data is retained during failed backups, but tracking and managing this retained data becomes challenging
Solution Approach 1:
The system implements a feedback mechanism that continuously updates metadata about the transfer status of backup data objects. When data is successfully transmitted to cloud storage, the metadata is updated to reflect this status. During failed backup recovery, the system queries this metadata to identify which objects were successfully retained, ensuring accurate tracking without losing information about backup status.
Data Source
AI summary
Described is a system (and method) for leveraging data previously transferred to a cloud-based object storage as part of a failed backup when performing a subsequent backup operation. The system may leverage the native configuration of an object storage that may retain certain data even when a backup operation is classified as a failed operation. To provide such capabilities, the system may store specialized metadata that tracks which backups have failed and metadata indicating which backup data has not been committed. Accordingly, upon a subsequent backup operation, the system may expediently refer to the stored metadata to identify retained backup data that may be reused for the subsequent backup. As a result, the system may improve the efficiency of a backup procedure by reducing the amount of data required to be transferred from the backup source.