Storage System Change Detection Using Pre-computed Hierarchy Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face inefficiencies in identifying renames and deletions in storage systems, particularly when changes to parent directories affect numerous child directories, leading to high processing overhead and latency due to the need for numerous API calls and file scans.
Innovation Solution
The implementation of a more efficient approach using metadata from change list information and intermediate data structures such as tables and stacks to identify renamed or deleted directories and files, allowing for the update of path identifiers and generation of an updated index file, which reduces the need for extensive API calls and file scans.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional methods are used to identify renames and deletions by scanning all files and making API calls, then completeness of change detection is improved, but processing overhead and latency increase significantly
Solution Approach 1:
The system performs preliminary actions by maintaining pre-computed data structures (parent-child relationship tables, directory stacks) that capture the storage system hierarchy before changes occur. When changes are detected, these pre-computed structures enable rapid identification of affected files without scanning the entire filesystem, thus achieving complete change detection with reduced processing overhead
Solution Approach 2:
The patent introduces intermediary data structures (parent directory tables, child directory stacks, path mapping tables) that act as mediators between the storage system and the backup system. These intermediaries cache hierarchical relationships and enable efficient propagation of change information from parent directories to child files without direct scanning, resolving the contradiction between detection completeness and processing efficiency
2Measurement precision
If extensive API calls and file scans are performed to detect changes in parent directories, then accuracy of change identification is improved, but processing time and latency increase
Solution Approach 1:
The system pre-computes and stores parent-child directory relationships, file-to-directory mappings, and hierarchical path structures before changes occur. When a parent directory change is detected, these pre-computed structures enable immediate identification of all affected child files without time-consuming scans, maintaining high accuracy while minimizing processing time
Solution Approach 2:
The patent creates and maintains copied representations of the storage hierarchy (in-memory data structures, indexing tables, path maps) that mirror the actual filesystem structure. These copies allow rapid querying and change propagation without accessing the actual filesystem, achieving accurate change identification with significantly reduced access time
3Reliability
If the system processes changes in storage systems with many files and directories, then completeness of backup is improved, but processing overhead increases prohibitively
Solution Approach 1:
The patent segments the storage system hierarchy into manageable units (parent directories, child directories, affected files) and processes changes in a structured cascade manner. By dividing the large-scale backup problem into hierarchical segments and using pre-computed relationship tables, the system achieves complete backup coverage without prohibitively high processing overhead
Solution Approach 2:
Intermediary data structures (change propagation queues, affected file lists, path mapping tables) act as mediators that efficiently manage the complexity of processing changes across thousands of files. These intermediaries organize and streamline the backup process, maintaining completeness while reducing overhead by avoiding redundant operations
Data Source
AI summary
A data management system (DMS) may receive entries corresponding to files and directories of a storage system that changed within a time period. The received entries may include metadata associated with the changed files and directories. The DMS may identify one or more directories that were renamed or deleted within the time period by analyzing the metadata from the received entries. The DMS may identify descendent directories and descendent files corresponding to the one or more renamed or deleted directories. The DMS may update path identifiers associated with the descendent directories and the descendent files based on the metadata from the received entries. The DMS may generate an updated index file based on updating the path identifiers for the descendent directories and the descendent files. The described techniques may enable the DMS to identify renames and deletions with greater efficiency, among other benefits.