Object-Storage File System With Ephemeral Metadata Cache
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage solutions do not effectively utilize object storage as a primary storage tier due to high latency in accessing file system metadata, which is a limitation for meeting the real-time needs of commercial workloads, and there are no cost-effective methods to reduce this latency.
Innovation Solution
Implementing an external cache using local ephemeral storage to store file system metadata, with policies for adding, updating, and re-warming the cache to accelerate access, thereby reducing latency and leveraging object storage as the primary storage tier.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If object storage is used as primary storage tier, then cost is reduced, but read latency increases
Solution Approach 1:
The storage system is segmented into two distinct tiers: a high-performance ephemeral storage tier for caching frequently accessed metadata and a cost-effective object storage tier for persistent data storage. This segmentation allows the system to leverage the low cost of object storage while maintaining fast access speeds through the ephemeral cache layer, resolving the contradiction between cost reduction and latency reduction.
Solution Approach 2:
The system performs preliminary action by proactively caching metadata in ephemeral storage before it is needed for read operations. By maintaining a cache of frequently accessed metadata blocks in the fast ephemeral storage tier, the system prepares data in advance, reducing read latency when metadata is subsequently accessed while still utilizing the cost-effective object storage tier for actual data storage.
2Productivity
If external cache is implemented, then read performance is improved, but device complexity increases
Solution Approach 1:
The ephemeral storage acts as an intermediary layer between the client and the object storage tier. It mediates read operations by providing fast access to cached metadata while the actual data remains in the object storage tier. This intermediary cache layer improves read performance without requiring complex changes to the underlying object storage infrastructure, thus managing complexity effectively.
Solution Approach 2:
The system creates a copy of frequently accessed metadata blocks in the ephemeral storage tier. Instead of reading directly from the original object storage, the system reads from this cached copy, improving read performance. The copying approach is simple and direct, avoiding complex data replication or synchronization mechanisms while still achieving significant performance improvements.
Data Source
AI summary
Systems and methods for using object storage as a primary storage tier for a file system of a virtual storage system are provided. According to one embodiment, ephemeral storage of a host on which a virtual storage system is running in the cloud is used as an external cache in which all or some subset of file system metadata may be stored, for example, to boost read IOPS and reduce latency of the read path. In various examples, external cache policies may be defined to trigger adding to, updating, and/or re-warming the external cache to accelerate access to file system metadata, thereby making a virtual storage solution that maintains all file system data and file system metadata within object storage a suitable and cost-effective platform for certain types of workloads. In some example, the working set of a workload may also be kept in the external cache to further accelerate read performance of the virtual storage system.