File System Metadata Management for Large Directory Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file systems face challenges with large directories, where metadata operations become a bottleneck due to high IO requests and read-write amplification, leading to poor performance, especially in high-performance computing environments.
Innovation Solution
The solution involves organizing all directories and files in a tree structure with a continuous address space, updating metadata as a log, and using multi-threading, vector processing, and many-core acceleration technologies to ensure each metadata operation triggers only one IO request, reducing directory path parsing delay and IO requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional tree structure metadata storage is used, then directory organization is simple, but access performance deteriorates significantly when directories contain large numbers of subdirectories or files
Solution Approach 1:
The patent segments the metadata storage by separating directory entries from file data, and further segments large directories by distributing entries across multiple storage nodes. Each directory is divided into entry blocks that can be independently accessed, avoiding the need to traverse entire directory trees and reducing path parsing delays.
Solution Approach 2:
The patent introduces a new dimension to metadata storage by implementing a hierarchical index structure that adds an intermediate lookup layer. Instead of direct linear search through directory entries, the system uses index blocks that provide rapid navigation to specific directory entries, transforming the access pattern from sequential to logarithmic time complexity.
2Productivity
If distributed metadata management is implemented, then concurrent access capability improves, but system complexity increases
Solution Approach 1:
The patent segments metadata into discrete entry blocks that can be independently distributed across multiple storage nodes. Each node manages a subset of directory entries, allowing parallel access without requiring complex coordination protocols. This segmentation enables simple replication and distribution strategies that maintain low system complexity.
Solution Approach 2:
The patent implements local quality by allowing each storage node to independently manage and cache directory entry blocks locally. This enables local access to frequently used directory entries without remote communication, while maintaining global consistency through versioning. The local quality approach reduces network overhead and simplifies distributed coordination.
3Stability of the object's composition
If tree index structure is used for large directories, then organization is maintained, but IO requests and calculation operations increase significantly
Solution Approach 1:
The patent extracts the directory entry metadata from the file data storage, placing them in separate, optimized structures. Directory entries are taken out and stored in fixed-size blocks that can be efficiently indexed and searched, while file data remains in its original storage location. This extraction eliminates the overhead of managing large directory structures within the file system hierarchy.
Solution Approach 2:
The patent changes the parameters of directory storage by using fixed-size entry blocks with predetermined capacity limits. Instead of variable-length directory structures that grow with the number of files, the system uses fixed blocks that can be pre-allocated and efficiently managed. This parameter change enables constant-time access operations and predictable performance regardless of directory size.
Data Source
AI summary
A metadata management method, system and medium are provided. All directories and files in a file system are organized in a tree structure, and all directories and subfiles under a single directory are saved as a whole on a continuous address space of an underlying storage device in the order of creation in sequence. When metadata is updated, update operations on all subdirectories/subfiles under a directory are appended to the tail of the directory in the form of log in order, so that each metadata operation triggers only one IO request. In the process of metadata retrieval, parsing a subdirectory under a parent director requires reading all data on the continuous address space corresponding to the parent directory, and then searching for a match by means of multiple acceleration methods.