Snapshot Change List File System Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As the number of files in a distributed file system grows, data services jobs become computationally intensive and time-consuming, especially when traversing billions of files to apply backup or analytics policies, leading to performance degradation.
Innovation Solution
The system creates snapshots of the file system namespace at different times, generates a change list representing differences between these snapshots, and updates an index with this information, allowing for efficient data services operations without traversing the entire file system namespace.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the file system namespace is traversed to select files matching data services policy criteria, then complete file evaluation is achieved, but computing resources and time become excessively large
Solution Approach 1:
The patent creates snapshots of the file system namespace at different points in time and generates change lists that record modifications between snapshots. This preliminary action captures file system state changes without requiring full traversals when data services jobs need to execute, thereby reducing computing resources and time while maintaining evaluation completeness for changed files.
Solution Approach 2:
The patent extracts only the changed files and their metadata from the full file system namespace by comparing snapshots and generating change lists. This extraction allows data services jobs to operate on a reduced subset of files that actually changed, rather than traversing and evaluating all files in the namespace, thus improving job execution speed while maintaining precision for relevant files.
2Adaptability or versatility
If billions of files are evaluated to apply backup or analytics policies, then comprehensive data services are provided, but performance degradation occurs
Solution Approach 1:
The system performs preliminary snapshot creation and change list generation to identify only the files that have changed since the last snapshot. This allows data services jobs to be applied selectively to changed files rather than evaluating all billions of files, maintaining comprehensive data services coverage for relevant files while preventing performance degradation through reduced processing scope.
Solution Approach 2:
The patent applies partial action by executing data services jobs on only the subset of files that have changed, as identified through snapshot comparison and change lists. This partial evaluation approach maintains adaptability and versatility for files that actually require processing while avoiding the performance degradation associated with evaluating all files in the namespace.
3Measurement precision
If metadata evaluations are performed for each file, then accurate policy criterion assessment is achieved, but overall file system performance is adversely impacted
Solution Approach 1:
The patent extracts and stores metadata information about changed files in change lists generated from snapshot comparisons. This extraction allows data services jobs to access and evaluate metadata for only the changed files without performing full metadata evaluations across all files in the namespace, thereby maintaining accurate policy criterion assessment while improving file system throughput by reducing the scope of metadata processing.
Data Source
AI summary
The described technology is generally directed towards creating snapshot change lists that describe differences between snapshots that capture file system states at different times. Also described is the use of snapshot change lists for updating an index that represents the objects in the file system at a given time. Data services jobs may be run against the index to select files, instead of running the jobs against the actual file system namespace, which is typically far more efficient when dealing with large file system namespaces.