Distributed File System Hot File Migration to High-Performance Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face performance issues due to varying capacities and IOPS (input/output operations per second) among storage devices, leading to reading delays and reduced overall performance when high-frequency accessed 'hot' files are stored on low-performance devices, causing a 'short slab' effect.
Innovation Solution
A method and apparatus that migrate files with high access frequencies to storage devices with higher performance, balancing IOPS pressure by determining access times and migrating files to devices with idle blocks and higher performance, setting an access enthusiasm attribute for 'hot' files, and managing IOPS limits and ratios to optimize storage usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If files are stored on storage devices using random polling and selection strategy or disk space balancing strategy, then storage space utilization is improved, but reading delay occurs on low performance storage devices and overall system performance deteriorates
Solution Approach 1:
The patent applies local quality by differentiating storage devices into high-performance and low-performance categories, and differentiating files into hot files (high access frequency) and cold files (low access frequency). Hot files are specifically directed to high-performance storage devices while cold files can be stored on low-performance devices, creating localized optimization for different data types rather than uniform storage allocation
Solution Approach 2:
The patent changes the parameter of file selection criteria from random or space-balancing metrics to access frequency-based metrics. By monitoring access times and identifying hot files, the system dynamically adjusts which files are placed on which storage devices based on performance characteristics, transforming the storage allocation strategy from static to dynamic parameter-based selection
2Productivity
If high performance storage devices are used for all files, then reading speed is improved, but storage cost and device complexity increase
Solution Approach 1:
The patent segments the storage system into high-performance storage devices and low-performance storage devices, and segments files into hot files and cold files. This segmentation allows the system to allocate resources appropriately - using expensive high-performance devices only for files that require fast access, while using cheaper low-performance devices for less frequently accessed files, thereby reducing overall system complexity and cost
3Productivity
If files are migrated frequently to balance load, then IOPS pressure is balanced, but system overhead and time consumption increase
Solution Approach 1:
The patent implements feedback mechanisms by monitoring access times of files and IOPS metrics of storage devices. The system continuously collects data on file access patterns and storage device performance, then uses this feedback to make informed migration decisions. This feedback loop ensures migration occurs only when necessary (when hot files are identified and IOPS balance is needed) rather than following fixed schedules, reducing unnecessary migration overhead
Solution Approach 2:
The patent applies partial action by selectively migrating only hot files to high-performance storage devices rather than migrating all files. The system identifies specific files that exceed access time thresholds and migrates only those files, leaving cold files on their current storage devices. This partial migration approach balances IOPS pressure without the excessive time consumption of comprehensive file migration
Data Source
AI summary
Provided are a storage method and apparatus for a distributed file system. The method includes: determining access times to a same file in preset time; and migrating the file to a storage device with higher performance than that of a current storage device according to the access times. Through the technical solution, tiered storage is performed on files in the distributed file system according to access enthusiasm of the files, that is, a file with a high access frequency in the preset time is migrated to a storage device with higher performance, and a file fragment storage location is associated with file fragment access enthusiasm through differential storage, so that IOPS pressure between storage devices is balanced and the overall performance is improved.