Distributed Directory Metadata Sharding for Higher File-System QPS
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing distributed file systems face inefficiencies in reading and writing due to the use of a single index shard for directory metadata, leading to reduced scalability and Queries-per-second (QPS) performance, which cannot meet the demands of large data volumes.
Innovation Solution
Implement a distributed file system architecture with multiple index shards, where directory metadata is stored across these shards based on full path hashing, allowing for partial loading and adjustment of metadata and index information to improve efficiency and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single index shard is used to store all directory metadata, then the system structure is simple, but the reading and writing efficiency deteriorates due to the need to load the entire index shard
Solution Approach 1:
The index cell is divided into multiple index shards (first index shard, second index shard, etc.), each storing index information for a portion of the directory metadata. This segmentation allows the system to load only relevant index shards instead of the entire index cell, improving reading and writing efficiency while maintaining manageable system complexity
Solution Approach 2:
The patent introduces a hierarchical dimension to the index structure by organizing index shards at different levels (first level index shards containing metadata pointers, second level index shards containing actual index information). This multi-dimensional organization enables efficient navigation and partial loading, resolving the contradiction between structural simplicity and operational efficiency
2Ease of operation
If the entire index shard is loaded for every read/write operation, then data access is straightforward, but the Queries-per-second (QPS) performance deteriorates
Solution Approach 1:
The patent extracts only the necessary index shard (first index shard) containing the metadata pointer into memory, rather than loading the entire index cell. This selective extraction significantly reduces memory I/O operations and improves QPS performance while maintaining straightforward data access through the extracted metadata
Solution Approach 2:
The system performs preliminary loading of the first index shard containing metadata pointers before actual data access operations. This preliminary action prepares the necessary routing information in advance, enabling fast subsequent access to specific index shards and improving overall QPS performance
3Device complexity
If a single index shard stores all directory metadata, then the system architecture is simple, but the storage capacity and scalability deteriorate
Solution Approach 1:
The index cell is segmented into multiple index shards that can be distributed across different storage locations. This segmentation enables the system to scale storage capacity by adding more index shards without fundamentally changing the overall system architecture, as each shard independently stores a portion of the directory metadata
Solution Approach 2:
The patent creates a universal index shard structure that can serve multiple functions: storing index information, enabling distributed storage, and supporting scalable expansion. Each index shard follows the same format and can be independently managed, allowing the system to universally handle growing storage requirements through simple replication and distribution
Data Source
AI summary
The present disclosure provides a method for processing directory metadata in a distributed file system, an apparatus and a device, which are related to the field of computer technologies, in particular to the field of big data and distributed file systems. The specific solution is as follows: when a processing request is acquired, a first index shard corresponding to directory metadata indicated by the processing request may be determined according to the processing request; the first index shard may be loaded, and a metadata shard corresponding to the directory metadata indicated by the processing request is determined according to the first index shard. Information in the determined metadata shard is adjusted. A second index shard related to the directory metadata indicated by the processing request is determined according to the processing request, and index information corresponding to the directory metadata in the second index shard is adjusted.