Dynamic Striping for Distributed File Storage Small Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file storage systems face performance bottlenecks when handling small files due to the increased burden of disk and network I/O operations caused by a large number of empty strips generated during striping, leading to inefficient use of resources and reduced access performance.
Innovation Solution
A method that dynamically adjusts the number of actual strips based on the size of the file by determining the quantity of valid and parity strips, allowing for reduced empty strips and optimized I/O operations through an N′+M redundancy mechanism, ensuring data integrity and performance by ensuring N′ > M.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If file data is divided into a large number of data strips using traditional striping algorithms, then space utilization of the distributed file storage system is improved, but the quantity of disk I/O operations and network I/O operations increases, leading to degraded access performance for small files
Solution Approach 1:
The patent implements dynamic striping by adjusting the number of data strips based on file size. For small files, fewer data strips are used to reduce I/O operations, while for large files, more data strips are utilized to improve space utilization. This dynamic adjustment resolves the contradiction between space utilization and access performance by adapting the striping configuration to the specific file being stored.
Solution Approach 2:
The patent changes the parameter of striping configuration (number of data strips) based on file size characteristics. By modifying this parameter dynamically, the system optimizes both space utilization and access performance for different file types, particularly improving small file access while maintaining efficient space usage.
2Quantity of substance
If the cluster size increases to improve space utilization, then more data strips are generated during striping, but the burden on access performance in small file scenarios increases due to increased I/O operations
Solution Approach 1:
The patent introduces dynamic adjustment of striping parameters based on file size, allowing the system to handle small files with reduced I/O complexity while maintaining the benefits of large cluster configurations for space utilization. This resolves the contradiction by making the system adaptable rather than static.
Solution Approach 2:
The patent applies different striping strategies to different file sizes within the same distributed file storage system. Small files receive optimized treatment with fewer data strips, while large files utilize the full capacity of the expanded cluster. This local optimization resolves the contradiction between cluster expansion benefits and small file access burden.
Data Source
AI summary
A data processing method and a device in a distributed file storage system, where the method includes receiving, by a client agent, a data processing request which carries a file identifier, an offset address, a file length, and other information of a target file; obtaining, by the client agent, redundancy information according to the file identifier carried in the data processing request, where the redundancy information includes a quantity of data strips, N, of the distributed file storage system and a quantity of parity strips, M, of the distributed file storage system; determining a quantity of valid strips, DSC, of the target file according to the offset address and the length information; determining a quantity of actual strips, N′, of the target file according to the DSC and the M; and determining corresponding strips according to the N′ and processing the corresponding strips.