Incremental Backup Using Storage Log Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inefficiency of traversing the entire directory structure to identify changed files and directories during an incremental backup process, which is resource and time-intensive, especially when only a small number of changes have occurred, leading to potential data integrity issues due to clock skew between snapshot and storage log timestamps.
Innovation Solution
Selective directory traversal of primary snapshots, combined with analysis of a storage log to identify changed objects within a threshold time window, and deduplication of directories to reduce traversal efforts and prevent data corruption, ensuring accurate incremental backups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire directory structure is traversed to identify changed files and directories, then all changes are accurately identified, but the process becomes resource and time intensive
Solution Approach 1:
The patent segments the backup process into two distinct phases: (1) a fast filtering phase that uses storage log analysis to identify candidate changed objects, and (2) a verification phase that performs selective directory traversal only on relevant subdirectories. This segmentation allows the system to quickly eliminate most unchanged files while thoroughly verifying only those that may have changed, thus resolving the contradiction between comprehensive change detection and backup efficiency.
Solution Approach 2:
The patent performs preliminary analysis of storage logs before executing the full directory traversal. By examining the storage log first to identify candidate changed objects based on timestamps and event records, the system prepares a filtered list of directories that actually need to be traversed. This preliminary action prevents unnecessary traversal of unchanged directories, significantly improving backup efficiency while maintaining accuracy.
2Manufacturing precision
If the entire directory structure is traversed to identify changed files, then complete backup accuracy is achieved, but the time required for incremental backups increases significantly
Solution Approach 1:
The patent applies partial action by performing directory traversal only on subdirectories that contain candidate changed objects identified from storage log analysis, rather than traversing the entire directory structure. The system selectively applies the computationally expensive traversal operation only where necessary, achieving sufficient backup accuracy for incremental backups while dramatically reducing the time required compared to full directory traversal.
Solution Approach 2:
The patent substitutes the mechanical directory traversal process with a more efficient information retrieval process. Instead of physically traversing every directory and file to check for changes, the system uses storage log analysis to retrieve information about changed objects directly from logged events and timestamps. This substitution replaces the inefficient mechanical traversal with a faster information-based identification method.
3Productivity
If storage log timestamps are directly compared with snapshot timestamps, then changed objects can be identified, but clock skew causes data integrity issues
Solution Approach 1:
The patent implements beforehand cushioning by introducing a time window mechanism that anticipates and compensates for clock skew before it causes problems. Instead of directly comparing timestamps that may differ due to clock synchronization issues, the system defines a time window around the snapshot timestamp and includes all objects whose logged changes fall within this window. This cushioning approach ensures that even if timestamps differ slightly due to clock skew, the correct set of changed objects is still captured, maintaining data integrity while preserving the efficiency of log-based identification.
Data Source
AI summary
An identification of a new primary snapshot created for a primary storage system is received. A threshold time window based on a capture time associated with the new primary snapshot is determined. Entries of a storage log of the primary storage system occurring within the threshold time window are analyzed to identify any objects of the primary storage system that have changed during the threshold time window. Changed objects to capture in a new secondary backup stored at a secondary storage system and corresponding to the new primary snapshot are identified. Metadata of the new primary snapshot and metadata of a previous secondary backup are compared to determine for each of the objects of the primary storage system identified as having changed during the threshold time window whether a change to an object since the previous secondary backup is captured in the new primary snapshot.