Coalescing Storage Log Entries for Incremental Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inefficiency of traversing the entire directory structure to identify changed files and directories in incremental backups of large storage systems, which is resource and time intensive, and can lead to data integrity issues due to clock skew between snapshot and log clocks.
Innovation Solution
A selective directory traversal approach using snapshots and storage logs to identify changed objects within a threshold time window, deduplicating directory traversals, and verifying object presence to ensure accurate incremental backups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire directory structure is traversed to identify changed files and directories, then all changed objects are accurately identified, but the process becomes resource and time intensive
Solution Approach 1:
The system performs preliminary actions by creating snapshots of the storage system at specific points in time and maintaining storage logs that record changes between snapshots. This preliminary structuring of data allows the incremental backup process to query for changed objects using the log information rather than traversing the entire directory structure, thereby reducing time while maintaining accuracy.
Solution Approach 2:
The storage log acts as an intermediary mechanism between the snapshot data and the backup process. Instead of directly traversing the directory structure, the backup process queries the storage log to identify changed objects, which mediates the information retrieval process and eliminates the need for full directory traversals.
2Measurement precision
If the entire directory structure is traversed to identify changed files and directories, then all changed objects are accurately identified, but the process becomes resource intensive
Solution Approach 1:
By pre-creating snapshots and pre-recording changes in storage logs, the system eliminates the need for resource-intensive full directory traversals during the backup process. The preliminary data structuring allows efficient querying of changed objects using minimal computational resources.
Solution Approach 2:
The storage log serves as an intermediary that provides efficient change information without requiring direct traversal of the directory structure. This intermediary approach reduces computational resources by enabling indexed queries rather than sequential scans of the entire file system.
3Productivity
If storage logs are used to identify changed objects, then resource and time requirements are reduced, but data integrity issues may arise due to clock skew between snapshot and log clocks
Solution Approach 1:
The system incorporates feedback mechanisms where the backup process queries not only the storage log but also verifies changes against the actual snapshot data and file system state. This feedback loop allows the system to detect and correct inconsistencies caused by clock skew, ensuring data integrity while maintaining efficiency.
Solution Approach 2:
The system applies beforehand cushioning by using multiple verification mechanisms and tolerance thresholds in the change detection algorithm. These pre-built safeguards compensate for potential clock skew issues without requiring exact timestamp synchronization, thereby protecting data integrity while maintaining the efficiency benefits of log-based change identification.
Data Source
AI summary
An identification of a new primary snapshot created for a primary storage system is received. A change tracking time window that is at least a portion of a period between a first capture time associated with a previous primary snapshot and a second capture time associated with the new primary snapshot is determined. Entries of a storage log of the primary storage system occurring within the change tracking time window are analyzed to coalesce changes identified in the entries of the storage log occurring within the change tracking time window into a change tracking result set. The change tracking result set is used to identify at least a portion of data changes between the previous primary snapshot and the new primary snapshot to capture in a new backup snapshot stored at a secondary storage system.