Summary Change Log Inode Graph for Backup Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional incremental backup systems for large data storage systems face inefficiencies and performance degradation due to the need to traverse and process entire file systems or temporal and transactional change logs, which are resource-intensive and time-consuming, especially for systems with millions of files.
Innovation Solution
A new summary change log structure with enhanced inode records that include pointers to related file system objects, allowing for a single pass preprocessing and depth-first search to efficiently identify changed files for backup, reducing the need for full system traversal and hierarchical sorting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If incremental backup systems examine all data files by trawling the entire file system to identify changed files, then they can identify modified files for backup, but this consumes significant resources and time, degrading system performance
Solution Approach 1:
The system pre-processes and stores file change information in summary change logs during normal file operations, organizing data by parent directory hierarchies before backup operations. This preliminary organization allows backup systems to quickly retrieve only relevant changed files without traversing the entire file system, resolving the contradiction between accurate change identification and system performance.
2Reliability
If conventional summary change logs record all file changes sequentially in time order, then they maintain complete change history, but the randomness of entries and need for hierarchical sorting degrades performance
Solution Approach 1:
The summary change log is segmented and organized by parent directory hierarchies rather than maintaining a single sequential time-ordered list. Each parent directory maintains its own change log entries, allowing the backup system to process only relevant directory segments. This segmentation preserves complete change history reliability while eliminating the need to sort through all entries hierarchically, reducing processing time significantly.
3Reliability
If a file is created, changed multiple times, and then deleted between successive backups, then the change log records all these changes, but processing all entries to determine the file was deleted is time-consuming
Solution Approach 1:
The system extracts and stores only the essential final state information (whether a file ultimately exists or was deleted) from the sequence of changes in the summary change log. By organizing logs by parent directory and extracting the net result of all changes rather than processing every individual change entry, the system maintains accurate change tracking while dramatically improving backup processing efficiency by ignoring intermediate changes to files that were ultimately deleted.
Data Source
AI summary
A backup system and method for a large data store employs a new and enhanced summary change log data structure that records all file system changes comprises inode entry records augmented with additional fields to indicate inodes representing related child and parent file system objects. A single preprocessing pass through the summary change log augments all of the appropriate inode fields with the appropriate inode numbers of family members to create a graph structure that can be traversed in hierarchical order starting at a root inode to quickly identify all created and modified file system objects for inclusion in an incremental backup.