ZFS Cloud Deduplication Layer for POSIX Access and Low Latency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cloud object storage systems have limitations such as high latency, limited search capabilities, and non-compliance with local file system interfaces, making it difficult and costly to convert legacy applications to use object interfaces, and they also pose security risks due to encryption key management.

Innovation Solution

Layering file system functionality on cloud object interfaces, specifically using POSIX interfaces and semantics, to provide cloud-based storage while maintaining file-based access and security, by transforming ZFS file system disk-based storage into cloud-based storage, enabling memory mapping and data services like compression, encryption, deduplication, snapshots, and clones.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If cloud object storage is used, then storage elasticity and scale are improved, but latency increases and search capabilities are limited

Engineering Contradiction:

Improvestorage elasticityVSAvoidaccess latency

Core Design Contradiction:

Adaptability or versatilityVSSpeed

Solution Approach 1:

The patent segments cloud storage access into two paths: frequent/small objects accessed via cached copies in local memory (fast path) and infrequent/large objects accessed via cloud object storage (slow path). This segmentation resolves the latency contradiction by providing different access speeds for different usage scenarios while maintaining cloud storage elasticity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary actions by pre-caching frequently accessed cloud objects in local memory before they are needed. This preliminary caching action reduces latency for subsequent accesses to the same objects, resolving the speed-latency contradiction while preserving cloud storage scale.

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If cloud object storage is used, then storage scale is improved, but interface compatibility with legacy applications deteriorates

Engineering Contradiction:

Improvestorage scaleVSAvoidinterface compatibility

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent introduces an intermediary layer (the storage system with cache manager and object storage interface) that translates between cloud object storage interfaces and traditional file system interfaces. This intermediary enables legacy applications to access cloud storage at file system level without modification, resolving the interface compatibility contradiction while maintaining cloud storage scale.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The storage system provides universal access by implementing multiple interface types (object storage interface, file system interface, memory mapping interface) on the same underlying cloud storage infrastructure. This multi-functionality allows different applications with different interface requirements to access the same cloud storage resources, resolving the compatibility contradiction.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Reliability

If cloud object storage encryption is used, then data security is improved, but key management vulnerability increases

Engineering Contradiction:

Improvedata securityVSAvoidkey management vulnerability

Core Design Contradiction:

ReliabilityVSObject-affected harmful factors

Solution Approach 1:

The patent extracts encryption key management from the cloud service provider's control and places it locally in the customer's environment. By taking out key management from the cloud, the system maintains data security through encryption while eliminating the vulnerability associated with cloud-managed keys, as customers now control their own encryption keys locally.

Inventive Principle:

Principle #2Taking out (Extraction)

4Ease of operation

If file system functionality is layered on cloud object interfaces, then application compatibility is improved, but system complexity increases

Engineering Contradiction:

Improveapplication compatibilityVSAvoidsystem architecture complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The storage system provides self-service by automatically managing the complexity of interfacing with cloud object storage. The cache manager automatically determines whether to serve requests from cache or cloud, automatically caches objects based on access patterns, and automatically manages encryption keys. This self-service automation hides the system architecture complexity from applications while maintaining compatibility.

Inventive Principle:

Principle #25Self-service

Data Source

PatentEP3566126B1ZFS block-level deduplication at cloud scale

Publication Date: 2023.11.15 ORACLE INT CORP

Figure 1
Figure 2
Figure 3A~3D

AI summary

Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.