Cloud File System Hierarchies for Low-Latency Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud object storage systems have limitations such as high latency, limited search capabilities, and non-compliance with local file system interfaces, making it difficult to integrate legacy applications directly with cloud object storage without significant expense and complexity.
Innovation Solution
Layering file system functionality on cloud object interfaces, specifically using POSIX interfaces and semantics, to provide cloud-based storage while maintaining file-based access and organization, and enabling memory mapping to ensure consistency between memory map changes and writes, thereby transforming disk-based storage systems like ZFS into cloud-based storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud object storage is used directly, then elasticity and scale are improved, but latency increases and file system interface compatibility deteriorates
Solution Approach 1:
The patent introduces a cloud interface appliance as an intermediary layer between the ZFS file system and cloud object storage. This appliance translates file system operations into object storage operations, enabling legacy applications to access cloud storage with file system semantics while maintaining low latency through intelligent caching and operation batching strategies
2Quantity of substance
If cloud object storage is used directly, then storage scale is improved, but interface compatibility with legacy applications deteriorates
Solution Approach 1:
The cloud interface appliance serves as a translation layer that converts ZFS file system calls into cloud object storage operations. This allows legacy applications to continue using familiar file system interfaces while accessing cloud storage, eliminating the need for expensive application conversion
Solution Approach 2:
The system provides multiple interface types (POSIX file system interface and cloud object interface) through a single unified storage backend, allowing different applications to access the same cloud storage infrastructure using their preferred interface paradigm
3Adaptability or versatility
If cloud object storage is used, then storage elasticity is improved, but data security deteriorates due to encryption key management
Solution Approach 1:
The patent extracts encryption key management from the cloud provider's control and maintains it locally within the ZFS file system and cloud interface appliance. This extraction of the security-critical key management function ensures that even though data is stored in the cloud, the encryption keys remain under the customer's control, preventing cloud providers from accessing encrypted data
4Ease of operation
If POSIX interfaces are layered on cloud storage, then legacy application compatibility is improved, but system complexity increases
Solution Approach 1:
The cloud interface appliance consolidates the complexity of translating between POSIX and object interfaces into a single dedicated component. This centralization of translation logic simplifies the overall system architecture compared to implementing translation logic in every application or throughout the entire storage stack
Data Source
Figure 1
Figure 2
Figure 3A~3D
AI summary
Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.