Data Mover Module for Cluster File System Storage Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Lustre cluster file systems face challenges in balancing storage capacity and IO throughput, leading to suboptimal performance and excessive costs, particularly when using scale-out network attached storage in high-performance computing environments.
Innovation Solution
Implementing a cluster file system with a front-end Lustre file system, a back-end scale-out network attached storage cluster, and an intermediate data mover module that splits files for storage across different tiers of storage devices, allowing dynamic balancing of performance and cost through hierarchical storage management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If scale-out network attached storage is used in conventional Lustre file systems, then storage capacity and cost efficiency are improved, but IO throughput and performance deteriorate
Solution Approach 1:
The file system is segmented into two distinct layers: a front-end Lustre file system optimized for high-performance IO operations, and a back-end scale-out network attached storage cluster optimized for capacity and cost efficiency. The data mover module further segments data into smaller units that can be distributed and managed across the back-end storage cluster, allowing each layer to operate in its optimal performance regime.
Solution Approach 2:
A data mover module is introduced as an intermediary component between the front-end and back-end file systems. This mediator translates high-performance IO requests from the front-end into appropriate storage operations for the back-end, managing data movement, caching, and coordination to bridge the performance gap between the two storage layers.
2Productivity
If high-performance storage devices are used, then IO throughput is improved, but storage cost increases
Solution Approach 1:
Different storage devices with different performance characteristics and cost structures are used in different locations within the storage architecture. The front-end uses high-performance storage devices locally to ensure fast IO throughput, while the back-end uses cost-effective scale-out network attached storage for bulk capacity, optimizing the cost-performance ratio across the entire system.
Solution Approach 2:
The storage architecture transitions from a single-dimensional homogeneous storage system to a multi-dimensional heterogeneous system with distinct front-end and back-end layers. Each layer operates in its own performance and cost dimension, with the data mover module coordinating between dimensions to achieve overall system optimization.
3Quantity of substance
If data is stored directly in back-end storage arrays, then storage capacity is improved, but system performance deteriorates
Solution Approach 1:
The data mover module performs preliminary actions by pre-processing data before it reaches the back-end storage arrays. This includes data segmentation, formatting, and optimization operations that prepare data for efficient storage while maintaining performance requirements. The front-end file system also performs preliminary caching and buffering operations to ensure performance is not compromised by back-end storage limitations.
Data Source
AI summary
A cluster file system includes a front-end file system comprising a hierarchical storage management application programming interface (API), a back-end file system, and a data mover module arranged between the front-end and back-end file systems and adapted for communication with the hierarchical storage management API of the front-end file system. The data mover module is configured to control movement of data between the front-end file system and the back-end file system responsive to information received via the hierarchical storage management API. For example, in some embodiments, the data mover module splits a given file received from the front-end file system into multiple smaller files for storage in the back-end file system and associates with each of the smaller files a corresponding file extension from a designated namespace to permit reconstruction of the given file upon retrieval of the smaller files from the back-end file system.