POSIX File Hierarchies on Cloud Object Storage With Low-Latency Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cloud-based object storage systems have limitations such as restricted data access semantics, high latency, and incompatibility with local file system applications, making it challenging to integrate legacy applications without significant adaptation and increased costs.
Innovation Solution
Layering file system functionality on cloud object interfaces, specifically using POSIX interfaces and semantics, to provide cloud-based storage while maintaining file-based access and organization, and enabling memory mapping to ensure consistency between memory changes and writes, thereby transforming disk-based storage systems like ZFS into cloud-based storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud object storage is used, then elasticity and scale are improved, but latency increases and interface compatibility with legacy applications deteriorates
Solution Approach 1:
The system segments cloud object storage into two operational layers: a high-latency cloud object store for bulk storage and a low-latency local cache layer for frequent access. This segmentation allows the system to maintain elasticity and scale from cloud storage while reducing latency for common operations through local caching.
Solution Approach 2:
The system performs preliminary actions by pre-caching frequently accessed objects locally before they are actually needed. The predictive caching mechanism proactively loads data into local storage based on access patterns, thereby reducing latency when applications need to access this pre-cached data.
2Adaptability or versatility
If cloud object storage is used, then elasticity and scale are improved, but compatibility with local file system applications deteriorates
Solution Approach 1:
The system introduces an intermediary component that translates between cloud object storage interfaces and traditional file system interfaces. This intermediary layer allows legacy applications to interact with cloud storage using familiar file system calls while the system handles the conversion to object storage operations, thereby maintaining compatibility without sacrificing cloud storage benefits.
Solution Approach 2:
The system implements a universal access layer that supports multiple interface types (file system interfaces, object storage interfaces, and block interfaces) simultaneously. This multi-functional approach allows the same underlying cloud storage infrastructure to serve both legacy applications requiring file system semantics and modern applications using object storage APIs.
3Quantity of substance
If cloud object storage is used, then storage capacity is improved, but data access speed deteriorates
Solution Approach 1:
The system applies local quality by providing different access speeds for different data based on their access patterns. Frequently accessed data is cached locally with high-speed access, while less frequently accessed data remains in cloud storage with lower access speed. This differentiated approach optimizes overall system performance by ensuring hot data is accessed quickly while maintaining large storage capacity.
Data Source
AI summary
Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.