Metabase Generation via Change Journal Traversal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage management systems face inefficiencies in scanning file systems for metadata updates, particularly in large volumes with high change rates, leading to time-consuming processes and resource utilization issues, as cached information from change journals may be insufficient and require restarting scans.
Innovation Solution
A system generates a metabase by traversing the file system and recording change notifications, allowing for simultaneous scanning and updating of metadata, reducing the need for iterative data access and minimizing network resource usage, enabling efficient data object information retrieval and reducing the frequency of file system scans.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the system uses change journal caching to reduce file system scans, then the frequency of scans is reduced, but the metadata may become insufficient when volumes are large or change rates are high
Solution Approach 1:
The system performs preliminary scanning of the file system to capture initial metadata before changes occur. This preliminary action ensures that the baseline metadata is established, allowing subsequent change journal updates to build upon this foundation rather than starting from scratch, thus maintaining accuracy while reducing scan frequency.
Solution Approach 2:
The system implements feedback mechanisms where change journal information is continuously monitored and fed back into the metadata storage structure. This feedback loop allows the system to update metadata dynamically based on actual file system changes, ensuring accuracy even when full scans are performed less frequently.
2Measurement precision
If the system performs frequent file system scans to update metadata, then metadata accuracy is maintained, but client resources are consumed and production application tasks are delayed
Solution Approach 1:
The change journal acts as an intermediary between the file system and the metadata storage system. Instead of directly scanning the file system for every metadata update, the system uses the change journal as a mediator that pre-processes change information, reducing the burden on client resources while maintaining metadata accuracy.
Solution Approach 2:
The system performs preliminary processing of file system changes through the change journal mechanism, capturing change information before it needs to be reflected in metadata. This preliminary action offloads the resource-intensive scanning operation from the critical path of metadata updates, allowing accurate metadata maintenance with reduced client resource consumption.
3Reliability
If the system restarts scans when paused, then completeness is ensured, but time is lost and processing is repeated
Solution Approach 1:
The system performs preliminary scanning to establish an initial metadata state before pausing operations. This preliminary action creates a known starting point that can be resumed from, ensuring that when the scan is restarted, the system can continue from the exact point where it paused rather than restarting from the beginning, thus maintaining completeness while minimizing time loss.
Solution Approach 2:
The system maintains feedback information about the scan progress and current position in the file system traversal. This feedback mechanism allows the scan process to remember where it left off during pauses, enabling reliable resumption without repeating already processed sections, thus ensuring completeness while reducing redundant processing time.
Data Source
AI summary
Aspects of the present invention are generally concerned with systems and methods for generating a database of metadata that describes system data and storage operations. The database of metadata may be referred to herein as a “metabase.” For example, to generate a metabase, a data agent may traverse a file system to obtain certain characteristics of data managed by the file system while substantially simultaneously detecting and recording change notifications. These actions may be performed even if the actions of the data agent are interrupted one or more times during the traversal of the file system. The data agent may process the characteristics and change notifications to generate and update a metabase. Once formed, the metabase may be consulted to determine changes in system data rather than determining the changes by scanning the data files themselves.