ZFS Cloud Deduplication via POSIX Layering and Local Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud object storage systems face limitations such as restricted data access semantics, high latency, and incompatibility with local file system applications, making it difficult to integrate legacy applications and ensure data security and availability.
Innovation Solution
Layering file system functionality on cloud object interfaces, specifically using POSIX interfaces and semantics, to provide cloud-based storage that mimics traditional file system access while ensuring data integrity and security through encryption and deduplication, and enabling seamless migration of data between local and cloud storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If cloud object storage is used, then storage capacity and scalability are improved, but access latency and compatibility with legacy applications worsen
Solution Approach 1:
The patent segments data into fixed-size blocks (e.g., 128KB) that can be independently managed, cached, and transferred between cloud and local storage. This segmentation enables selective caching of frequently accessed blocks locally, reducing access latency while maintaining overall cloud storage capacity.
Solution Approach 2:
The patent introduces a cloud storage appliance as an intermediary layer between cloud object storage and legacy applications. This appliance provides a file system interface that translates object storage operations into file system operations, masking the latency and incompatibility of direct cloud object storage access.
2Quantity of substance
If data is stored in cloud object storage, then storage scalability is improved, but data security and access control worsen
Solution Approach 1:
The patent implements local quality by maintaining encryption keys exclusively on the client side rather than in the cloud. This ensures that even though data is stored in the cloud, only the client can decrypt and access it, thereby maintaining strong security controls while enjoying cloud storage scalability.
Solution Approach 2:
The patent performs preliminary encryption of data blocks before transferring them to cloud storage. This pre-encryption ensures that data is secured before leaving the client environment, maintaining security requirements while enabling scalable cloud storage.
3Adaptability or versatility
If cloud object storage interfaces are used, then storage elasticity is improved, but compatibility with legacy file system applications worsens
Solution Approach 1:
The patent introduces a cloud storage appliance as an intermediary that presents a traditional file system interface (NFS, SMB, or iSCSI) to legacy applications while backing data to cloud object storage. This intermediary layer translates file system operations into object storage operations, maintaining application compatibility while providing cloud elasticity.
Solution Approach 2:
The patent makes the cloud storage appliance universally compatible with multiple legacy file system protocols (NFS, SMB, iSCSI) and cloud object storage APIs simultaneously. This multi-functionality allows the same appliance to serve both legacy applications requiring file system semantics and cloud storage requiring object interfaces.
4Quantity of substance
If data blocks are transferred to cloud storage, then storage capacity is improved, but network bandwidth consumption worsens
Solution Approach 1:
The patent segments data into blocks and implements intelligent caching of frequently accessed blocks in local storage. This reduces the need to repeatedly transfer the same data blocks over the network, thereby reducing network bandwidth consumption while maintaining access to cloud storage capacity.
Solution Approach 2:
The patent implements periodic synchronization between local cache and cloud storage, transferring data blocks only when necessary (e.g., when local cache is full or when data needs to be updated in the cloud). This periodic action reduces unnecessary network traffic compared to continuous synchronization.
Data Source
AI summary
Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.