Synthetic Full Backup Generation via Snapshot Differencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating full backups of large data sets, such as those in a Hadoop Distributed File System, is resource-intensive and time-consuming, making it expensive to maintain point-in-time full backup copies.
Innovation Solution
The method involves obtaining snapshots of previous and current data states, generating a snapshot difference report to identify deleted, renamed, and created files, and performing delete, rename, and copy operations on a previous backup to create a synthetic full backup, reducing the need to re-copy all data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full backups of all data are performed, then complete data protection is achieved, but the process becomes expensive (time-consuming and resource intensive)
Solution Approach 1:
The backup process is segmented into incremental changes tracking and synthetic full backup generation. Instead of backing up all data each time, the system segments the backup into: (1) tracking only changed blocks using snapshots, (2) maintaining a change log of modifications, and (3) generating synthetic full backups on-demand by applying changes to a base backup. This segmentation reduces regular backup time while maintaining complete data protection.
Solution Approach 2:
The system uses copying to create synthetic full backups by applying incremental changes to a base backup image rather than copying all source data. The synthetic full backup is generated by copying and applying only the necessary change blocks from the change log to the base backup, significantly reducing the time and resources required compared to traditional full backups.
2Reliability
If full backups of all data are performed, then complete data protection is achieved, but resource consumption increases
Solution Approach 1:
The backup system segments resource usage by separating the heavy full backup operation from routine backups. Routine backups only consume minimal resources to track changes in snapshots and update change logs. Synthetic full backups are generated on-demand by applying segmented change blocks to a base backup, distributing resource consumption over time and avoiding peak resource usage during regular backup intervals.
Solution Approach 2:
Instead of performing complete full backups at regular intervals, the system performs partial actions by tracking only the incremental changes that occurred since the last backup. This partial action approach consumes minimal resources during routine operations while maintaining the ability to generate complete synthetic full backups when needed.
3Use of energy by moving object
If incremental backups are used instead of full backups, then resource consumption is reduced, but backup complexity increases
Solution Approach 1:
The system introduces a change log as an intermediary data structure that simplifies the backup process. The change log acts as a mediator between snapshots and synthetic full backups, automatically tracking and organizing all data modifications. This intermediary layer abstracts the complexity of incremental tracking, making the overall process simpler to manage while maintaining low resource consumption.
Solution Approach 2:
The system uses copying to simplify the backup process by creating synthetic full backups through a straightforward application of incremental changes to a base backup. This copying approach is simpler than traditional incremental backup methods because it avoids complex reconstruction algorithms and directly applies recorded changes, reducing operational complexity while maintaining resource efficiency.
Data Source
AI summary
Techniques described herein relate to a method for generating synthetic full backups. The method may include obtaining a snapshot associated with a previous backup stored in a data domain; obtaining a second snapshot associated with a current state of the source data; generating a snapshot difference report using the first snapshot and the second snapshot and including a delete list, a rename list, and a copy list; storing a copy of the previous backup in the data domain; performing a delete operation on the copy based on the delete list; performing a rename operation on the copy based on the rename list; and performing a copy operation based on the copy list. In one or more embodiments, performing the delete operation, the rename operation, and the copy operation on the copy of the previous backup generates a first synthetic full backup of the source data.