Incremental Backup Consolidation for Database Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high cost and inefficiency of making entire copies of large databases for backup, particularly in systems like Cassandra, where incremental backups create overhead and generate many small files, leading to expensive data transfer and storage.
Innovation Solution
Identifying a list of only new data files since the last backup, creating incremental backups by comparing lists, and including metadata to reference the preceding backup, thereby reducing the size of backups by only storing new data files and allowing efficient restoration by traversing through previous backups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If incremental backups create separate files for each data file, then backup granularity is improved, but device complexity and storage overhead increase due to generating many small files
Solution Approach 1:
The patent merges multiple small data files into a single consolidated backup file. Instead of creating separate backup files for each data file modification, the system combines all changes into one unified backup file, reducing the number of files while maintaining incremental backup functionality and granularity.
Solution Approach 2:
The backup file structure is designed to be universal, containing metadata that identifies which data files are included and their locations. This single backup file serves multiple purposes: storing data file contents, maintaining references to original files, and enabling restoration without requiring multiple separate files.
2Reliability
If entire database copies are made for backup, then data completeness is improved, but transfer cost and time increase
Solution Approach 1:
The system extracts only the necessary portions of the database for backup by identifying and copying only the data files that have been modified, created, or deleted since the last backup. This extraction approach maintains data completeness for recovery purposes while significantly reducing the volume of data that needs to be transferred and stored.
Solution Approach 2:
Instead of performing a complete database copy, the system performs a partial backup action by selectively copying only the changed data files. This partial action is sufficient to maintain the ability to restore the complete database state while minimizing transfer time and cost.
3Ease of operation
If all data files are copied in each backup, then restoration simplicity is improved, but storage cost increases
Solution Approach 1:
The backup system implements a nested structure where a single backup file contains multiple data files and their metadata in a hierarchical organization. This nesting allows the backup to maintain simplicity in restoration (treating as one file) while efficiently packing only the necessary data files within, reducing overall storage requirements.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between the backup file and the original data files. This metadata contains references and identification information that enables simple restoration operations while the actual data files are selectively included based on changes, reducing backup size without complicating the restoration process.
Data Source
AI summary
Systems and methods are provided for database or data file backup. The system may comprise one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the system to identify a list of data files required for restoring the database or data files, create a backup comprising copies of a subset of the data files that are created or modified after a preceding backup, and back up the database, wherein the backup includes an identification to the preceding backup.