Block-Order File Traversal Using Snapshot Extent Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data management systems face challenges in efficiently traversing and reading large numbers of files due to the non-sequential order in which files are discovered in user-level filesystems, leading to suboptimal performance and increased scanning times, particularly in backup environments.
Innovation Solution
Implementing a block-order traversal technique using a reverse map generated from metadata, which maps extents to user-level file paths, allowing the system to read files in a sequential order that maximizes disk performance by using a sliding-window technique and a merged journal file.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If files are traversed in traditional filesystem order, then file discovery is simple, but scanning time increases significantly
Solution Approach 1:
The system performs preliminary indexing of files and their physical block locations during backup operations. This index structure is built in advance and stored with the backup data, enabling the system to retrieve file paths and block locations without performing full filesystem traversal during analysis operations, thus significantly reducing scanning time.
Solution Approach 2:
The patent introduces an intermediary index structure that maps logical file paths to physical block locations on disk. This intermediary layer (the index) mediates between the filesystem layer and the storage layer, allowing the system to skip directly to relevant data blocks without traversing the entire filesystem hierarchy, thereby improving scanning speed.
2Speed
If files are read sequentially from disk, then disk performance is optimized, but filesystem traversal complexity increases
Solution Approach 1:
The system pre-computes and stores the mapping between file paths and physical block locations during the backup process. This preliminary indexing action eliminates the need for complex real-time filesystem traversal during analysis, as the mapping information is already available in the backup data structure, simplifying the read operation while maintaining sequential disk access patterns.
Solution Approach 2:
The patent creates a copy of the filesystem metadata and indexing information within the backup data structure itself. This copied index structure allows the system to replicate the filesystem hierarchy and block mappings in the backup environment, enabling sequential reads from the backup storage without needing to traverse the original complex filesystem structure, thus reducing operational complexity.
3Reliability
If all files are scanned to ensure comprehensive analysis, then detection accuracy improves, but processing time increases
Solution Approach 1:
The system performs comprehensive indexing of all files and their block locations during backup operations, creating a complete map of the filesystem state at the time of backup. This preliminary action ensures that when analysis is performed, the system has complete information about all files and their physical locations, enabling accurate detection while reducing analysis time through direct access to the index rather than full scanning.
Solution Approach 2:
The backup process provides feedback about file locations and metadata to the analysis system through the index structure stored in the backup data. This feedback mechanism enables the analysis system to quickly locate and access specific files without performing comprehensive scanning, thereby maintaining detection accuracy while significantly reducing processing time through the pre-provided spatial information.
Data Source
AI summary
Methods, systems, and devices for data management are described. A data management system (DMS) may access a snapshot of a target client object to perform a block-order traversal of a set of files included in the snapshot. The DMS may generate an index file for the set of files based on a metadata associated with the snapshot, the index file including a mapping between a file path for each file and one or more properties of each file. The DMS may then read the set of files included in the snapshot using a sliding window in accordance with the mapping between the file path for each file and the one or more properties of each file, and may determine whether to generate an alert based on reading the set of files included in the snapshot.