Metadata Server Storage Tiering for Flash and Disk Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cluster file systems, such as Lustre, face challenges in balancing storage capacity and IO throughput requirements, leading to suboptimal performance or excessive costs due to mismatched storage devices in object storage servers.
Innovation Solution
Implementing storage tiering control functionality in metadata servers to dynamically balance storage capacity and IO throughput by transparently incorporating a flash storage tier alongside disk storage tiers, without requiring significant changes to clients or object storage servers, using a metadata server to control data movement between different storage tiers based on application hints and policy engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional Lustre implementations use object storage servers with back-end storage arrays, then storage capacity can be scaled, but IO throughput performance becomes suboptimal due to mismatched storage devices
Solution Approach 1:
The storage system is segmented into multiple storage tiers with different performance characteristics. Object storage servers are divided into groups associated with different storage tiers (e.g., flash storage tier, disk storage tier), allowing the system to provide both high capacity and high performance pathways simultaneously.
Solution Approach 2:
The system dynamically selects which storage tier to use for each IO operation based on requirements. The metadata server controls data placement and movement between tiers, enabling flexible adaptation to different performance and capacity needs without static configuration constraints.
2Productivity
If high-performance flash storage devices are deployed to improve IO throughput, then performance increases, but system costs become excessive
Solution Approach 1:
Different storage tiers with different quality characteristics are deployed in different locations within the storage system. Flash storage provides high-performance local quality for specific data sets, while disk storage provides cost-effective local quality for other data, avoiding the need to deploy high-performance storage system-wide.
Solution Approach 2:
The system changes the performance parameter of storage resources by moving data between storage tiers. When high IO throughput is needed, data is placed in the flash storage tier; when capacity is needed at lower cost, data is placed in the disk storage tier, dynamically adjusting performance characteristics.
3Productivity
If storage tiering control functionality is implemented in metadata servers, then transparent acceleration of IO operations is achieved, but device complexity increases
Solution Approach 1:
The metadata server is given multiple functions: traditional metadata management plus storage tiering control. By making the metadata server universal, the system avoids adding separate complex control components, as the metadata server already exists and can be extended to handle tiering decisions.
Solution Approach 2:
The metadata server autonomously makes decisions about data placement and migration between storage tiers based on policies and application hints, without requiring external complex control systems. The system serves itself by using existing metadata server infrastructure to manage storage tiering.
Data Source
AI summary
A cluster file system comprises a metadata server coupled to a plurality of object storage servers via a network. The metadata server comprises a controller configured to implement storage tiering control functionality for at least first and second storage tiers comprising respective disjoint subsets of the plurality of object storage servers. The metadata server is thereby configured to control movement of data between the first and second storage tiers. The object storage servers in the first storage tier may be configured to interface with object storage targets of a first type and the object storage servers in the second storage tier may be configured to interface with object storage targets of a second type different than the first type. For example, the object storage targets of the first type may comprise non-volatile electronic storage devices such as flash storage devices, and the object storage targets of the second type may comprise disk storage devices.