Cloud Gateway Layering ZFS Snapshots on Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cloud-based storage systems, particularly object-based storage, face limitations such as restricted data access semantics, high latency, and incompatibility with local file system applications, making it costly and impractical to convert legacy applications to use object interfaces, and compromising data security due to encryption key management.
Innovation Solution
Layering file system functionality on cloud object interfaces, specifically using POSIX interfaces and semantics, to provide cloud-based storage that maintains file-based access and organization, while enabling memory mapping and data services like compression, encryption, deduplication, snapshots, and clones, thereby bridging the gap between traditional file systems and cloud object stores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud object storage is used, then storage elasticity and scalability are improved, but data access latency increases and file system compatibility deteriorates
Solution Approach 1:
The patent segments cloud storage access into two paths: frequent/small objects accessed via cached pointers in the file system (low latency), and infrequent/large objects accessed directly from cloud storage (high elasticity). This segmentation resolves the contradiction by optimizing access patterns for different data types.
Solution Approach 2:
The patent introduces a cloud gateway as an intermediary layer between the file system and cloud object storage. The gateway manages caching, pointer updates, and data synchronization, enabling the file system to access cloud storage with latency comparable to local storage while maintaining cloud elasticity.
2Adaptability or versatility
If cloud object storage is used, then storage scalability is improved, but compatibility with legacy file system applications deteriorates
Solution Approach 1:
The cloud gateway acts as a transparent intermediary that translates file system operations into cloud object storage operations. Legacy applications continue to use standard file system calls without modification, while the gateway handles the complexity of cloud storage abstraction, maintaining full application compatibility.
Solution Approach 2:
The patent implements a copy-on-write caching mechanism where frequently accessed objects are copied to local cache with pointers in the file system. This copying strategy maintains file system semantics while enabling cloud storage backend, allowing legacy applications to operate unchanged.
3Reliability
If cloud object storage encryption is used, then data security is improved, but encryption key management vulnerability increases
Solution Approach 1:
The patent extracts encryption key management from the cloud storage provider and implements client-side encryption. Encryption keys remain exclusively on the client device, removing the key management vulnerability associated with cloud-based key storage while maintaining strong data encryption.
Solution Approach 2:
The patent applies encryption to data before it leaves the client device (pre-encryption). This preliminary encryption action ensures that data is protected end-to-end, with keys never exposed to the cloud provider, eliminating the key management vulnerability.
4Ease of operation
If file system functionality is layered on cloud object interfaces, then application compatibility is improved, but system complexity increases
Solution Approach 1:
The cloud gateway consolidates file system layering complexity into a single intermediary component. It implements the full file system interface (POSIX compliance, memory mapping, snapshots) while hiding the complexity from both applications and cloud storage, making the system easier to operate despite architectural complexity.
Data Source
Figure 1
Figure 2
Figure 3A~3D
AI summary
Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.