Virtual File System for Independent Data and Metadata Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Lustre cluster file systems face challenges in balancing storage capacity and IO throughput, leading to suboptimal performance and excessive costs due to mismatched storage devices for IO operations.
Innovation Solution
Implementing separate data and metadata object stores using a virtual file system that interfaces with an underlying physical file system, allowing for independent storage of data and metadata objects while presenting a file-based interface to clients, thereby accelerating IO operations and optimizing system performance at lower costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional Lustre implementations use object storage servers with back-end storage arrays for direct IO operations, then storage capacity is achieved, but IO throughput performance becomes suboptimal and system costs become excessive
Solution Approach 1:
The patent segments the storage system into distinct data object stores and metadata object stores, each optimized for their specific functions. Data object stores handle bulk data storage while metadata object stores manage file system metadata, allowing independent optimization of each component for its specific IO patterns and performance requirements.
Solution Approach 2:
The virtual file system acts as an intermediary layer between clients and the physical file system's object stores. It translates file-based client requests into object-based storage operations, enabling optimized IO paths while maintaining compatibility with conventional file system interfaces and applications.
2Productivity
If storage devices are matched to current system needs in conventional Lustre implementations, then cost is reduced, but IO operation performance becomes suboptimal
Solution Approach 1:
The patent applies local quality by providing different storage optimizations in different parts of the system. Data object stores can use high-capacity, lower-cost storage media optimized for bulk data, while metadata object stores can use faster, lower-capacity storage optimized for rapid access to file system metadata, matching each component's storage characteristics to its specific performance needs.
3Productivity
If separate data and metadata object stores are implemented using a virtual file system, then IO operations are accelerated and system performance is enhanced, but device complexity increases
Solution Approach 1:
The virtual file system provides multi-functionality by simultaneously supporting both file-based client interfaces and object-based storage operations. It can translate file operations into object operations while also providing direct object access when needed, enabling a single system to serve multiple access patterns and optimization strategies.
Data Source
AI summary
A cluster file system includes a physical file system and a virtual file system arranged between the physical file system and multiple clients. The virtual file system comprises a data object processing module configured to receive multiple data files from each of one or more of the clients and to transform the data files into data objects for object-based storage in the physical file system. The virtual file system further comprises a metadata object processing module configured to receive metadata files associated with the data files and to transform the metadata files into metadata objects for object-based storage in the physical file system. The virtual file system thereby implements multiple object stores utilizing the physical file system for independent storage of data objects and metadata objects but presents a file-based interface to the clients. In some embodiments, the virtual file system illustratively comprises a parallel log-structured file system (PLFS).