Multi-Threaded Indexed File System Metadata Scanning Bottleneck
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file systems face performance degradation due to overhead associated with loading and scanning metadata for large-scale file operations, especially in caching servers handling billions of files with irregular storage patterns and complex multi-dimensional operations, leading to bottlenecks and inability to scale effectively.
Innovation Solution
The Multi-Threaded Indexed (MTI) file system indexes desired file attributes independently and simultaneously executes storage operations based on these indexes, allowing direct identification and management of files without scanning metadata, thereby reducing overhead and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the file system loads and scans metadata to identify files for operations, then file identification accuracy is improved, but file system performance and device throughput deteriorate
Solution Approach 1:
The patent pre-computes and stores file attribute values (such as file size, modification time, access time) in an optimized data structure before operations are executed. This preliminary indexing allows the file system to quickly retrieve pre-processed attribute data without scanning raw metadata during operations, thus maintaining identification accuracy while improving performance.
Solution Approach 2:
The patent extracts frequently queried file attributes from the complete metadata and stores them in a separate, optimized index structure. By separating these commonly accessed attributes from the full metadata, the system avoids loading and scanning entire metadata records during operations, reducing I/O overhead and improving query performance while maintaining accurate file identification.
2Measurement precision
If the file system scans metadata for complex multi-dimensional operations, then operation accuracy is improved, but processing time and resource consumption worsen
Solution Approach 1:
The patent pre-computes composite indices for multi-dimensional operations by combining multiple file attributes during the preliminary indexing phase. For example, it pre-calculates composite keys based on combinations of file size ranges, time intervals, and other attributes that are commonly used together in multi-dimensional operations. This allows the system to perform accurate multi-dimensional queries by simply looking up pre-computed composite indices without performing multiple separate scans or complex filtering during operation execution.
3Speed
If the file system loads all metadata into memory, then file access speed is improved, but memory consumption and overhead worsen
Solution Approach 1:
The patent implements a selective loading strategy where only the specific file attributes required for the current operation are loaded into memory, rather than loading all metadata. The optimized data structure allows the file system to efficiently retrieve and load only the necessary attribute columns for the given operation type, reducing memory consumption while maintaining fast access speed for the required data.
Solution Approach 2:
The patent extracts and stores only the essential file attributes in an optimized in-memory index structure, leaving non-essential or rarely accessed metadata on storage. This selective extraction reduces the amount of data that needs to be loaded into memory during operations, decreasing memory overhead while preserving fast access performance for the most commonly used attributes.
4Measurement precision
If the file system performs multiple scans for multi-dimensional operations, then comprehensive file identification is improved, but operation throughput and scalability worsen
Solution Approach 1:
The patent merges multiple attribute indices into a unified optimized data structure that can simultaneously support queries on multiple dimensions. Instead of performing separate scans for each attribute dimension, the system combines the indices for file size, time attributes, and other dimensions into an integrated structure that allows single-pass or single-query retrieval of files matching complex multi-dimensional criteria, thereby improving both identification comprehensiveness and operation throughput.
Data Source
AI summary
A Multi-Threaded Indexed (“MTI”) file system may use a first set of threads, processes, or executable instances to index desired file attributes in a database while simultaneously but independently executing file operations with a second set of threads, processes, or executable instances. In response to receiving a file operation, the second set of threads, processes, or executable instance may query the database to directly identify files that are indirectly implicated by the file operation with a wildcard, regular expression, and/or other expression that indirectly identifies the files based on different file attributes, paths, name expressions, or combinations thereof. The second set of threads, processes, or executable instances are therefore able to identify the files implicated by the file operation based solely on the indexed file attributes already entered in the database without the need to load and scan the metadata of files in directories targeted by the file operation.