Object Storage Gateway for Network File System Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Lustre file systems face challenges in balancing storage capacity and IO throughput, leading to suboptimal performance or excessive costs, particularly in supercomputers and complex computing environments.
Innovation Solution
Implementing an object storage server-based gateway to a network file system cluster, utilizing different types of storage devices for dynamic balancing of storage capacity and IO throughput, with a controller managing data movement based on policy information and application needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If scale-out network attached storage is used for cost-effective storage, then storage capacity increases at lower cost, but IO throughput performance becomes inadequate for supercomputers and complex computing environments
Solution Approach 1:
The system segments storage functionality into two distinct layers: a cost-effective scale-out NAS layer for bulk storage capacity and a high-performance object storage gateway layer for IO operations. This segmentation allows each layer to be optimized independently - the NAS layer provides economical capacity while the gateway layer delivers high-speed access for supercomputing workloads.
Solution Approach 2:
The object storage gateway acts as an intermediary between the scale-out NAS storage and the Lustre file system. It translates and mediates IO operations, converting standard file system requests into optimized object storage operations that can leverage the performance characteristics of both the NAS backend and the high-speed network infrastructure.
2Productivity
If conventional Lustre implementations use back-end storage arrays directly, then IO operations can be performed, but storage devices may not be well matched to current system needs leading to suboptimal performance or excessive costs
Solution Approach 1:
The system implements dynamic storage device matching through the object storage gateway, which can adaptively route different types of IO operations to appropriate storage resources based on current system needs, data access patterns, and performance requirements. This dynamic approach replaces static direct-attached storage configurations with flexible, policy-driven resource allocation.
Solution Approach 2:
The object storage gateway changes the operational parameters of storage access by implementing different data movement strategies, caching policies, and protocol translations. These parameter changes enable the system to optimize performance for different workloads without requiring physical reconfiguration of storage devices.
Data Source
AI summary
A gateway of a cluster file system comprises a plurality of object storage servers each having one or more object storage targets implemented using storage devices of a first type. The gateway is configured to communicate with a network file system cluster implemented using storage devices of a second type different than the first type. The network file system cluster comprises storage directories corresponding to respective ones of the object storage targets. A controller that may be implemented at least in part within the gateway is configured to control movement of data between the object storage targets and the storage directories of the network file system cluster. The movement of data between the object storage targets and the storage directories may be controlled at least in part based on one or more of policy information provided by a policy engine and information provided by an application running on a client.