Cloud File System Layering for POSIX Access and Low-Latency Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud object storage systems have limited search capabilities and high latency, making it difficult to align with the needs of local file system applications, and converting legacy applications to use an object interface is costly and impractical.
Innovation Solution
Layering file system functionality on cloud object interfaces, allowing for POSIX interfaces and semantics, while providing memory mapping to ensure consistency between memory map changes and writes, effectively transforming disk-based storage into cloud-based storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud object storage is used directly, then elasticity and scalability are improved, but search capability and latency are worsened
Solution Approach 1:
The patent introduces a file system layer as an intermediary between cloud object storage and applications. This file system layer provides file-based interfaces, search capabilities, and caching mechanisms while the underlying cloud storage provides elasticity and scalability. The intermediary translates file operations into object operations, resolving the contradiction between cloud storage benefits and application needs.
2Adaptability or versatility
If cloud object storage is used directly, then elasticity and scalability are improved, but latency is worsened
Solution Approach 1:
The file system layer performs preliminary actions by caching frequently accessed data locally before cloud storage operations are executed. This pre-caching and local buffering reduces the latency of cloud storage operations while maintaining the elasticity and scalability benefits of cloud storage for less frequently accessed data.
3Adaptability or versatility
If legacy applications are converted to use object interface, then compatibility with cloud storage is improved, but cost and practicality are worsened
Solution Approach 1:
Instead of converting legacy applications to use object interfaces (the conventional approach), the patent inverts the approach by bringing file system interfaces to the cloud storage layer. This allows legacy applications to continue using familiar file-based interfaces while the system internally handles cloud object storage operations, significantly reducing conversion costs and practicality issues.
4Adaptability or versatility
If file system functionality is layered on cloud object interfaces, then POSIX interface compatibility is improved, but system complexity is worsened
Solution Approach 1:
The file system layer is designed to provide multiple functions including POSIX interface compatibility, search capabilities, caching, compression, encryption, and snapshots. By consolidating these diverse functions into a single universal layer, the system achieves POSIX compatibility without proportionally increasing complexity, as the file system layer handles multiple responsibilities efficiently.
Data Source
AI summary
Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.