Virtual Storage Appliance Metadata Caching for Cold Data Analytics

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing storage solutions do not effectively utilize object storage as a primary storage tier due to high latency in accessing file system metadata, making it impractical for real-time needs of commercial workloads, and lack efficient methods for performing analytics on tiered cold data.

Innovation Solution

Implement an external cache using local ephemeral storage to store file system metadata and utilize object storage as the primary tier, with policies for adding and re-warming metadata to reduce latency and enable efficient access, and create a virtual storage system that maintains all file system data and metadata within object storage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of energy

If object storage is used as primary storage tier, then storage costs are reduced, but read latency increases due to high latency in accessing file system metadata

Engineering Contradiction:

Improvestorage costsVSAvoidread latency

Core Design Contradiction:

Loss of energyVSLoss of time

Solution Approach 1:

The patent segments the storage system into two distinct components: object storage for bulk data retention and a virtual storage appliance with local cache for metadata and hot data. This segmentation allows each component to operate in its optimal performance zone while mitigating the latency issue of object storage through the local cache layer.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The virtual storage appliance acts as an intermediary between the high-latency object storage and the workload requirements. It provides a local cache layer that mediates access patterns, serving metadata and frequently accessed data locally while asynchronously syncing with object storage, thus hiding the latency of object storage access.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Volume of stationary object

If cold data is tiered to object storage, then storage capacity is extended, but analytics performance degrades due to data being distributed across on-premises and cloud locations

Engineering Contradiction:

Improvestorage capacityVSAvoidanalytics performance

Core Design Contradiction:

Volume of stationary objectVSProductivity

Solution Approach 1:

The virtual storage appliance creates a local copy (cache) of metadata and frequently accessed data blocks from object storage. This copying mechanism enables analytics workloads to access data locally at high speed while the original data remains in object storage, eliminating the performance degradation caused by distributed data access.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system performs preliminary actions by pre-loading metadata and anticipating data access patterns. The virtual storage appliance proactively caches metadata and data blocks before they are needed for analytics operations, reducing wait times and improving overall analytics performance when working with tiered cold data.

Inventive Principle:

Principle #10Preliminary action

3Loss of time

If external cache is implemented with local storage, then read latency is reduced, but device complexity increases

Engineering Contradiction:

Improveread latencyVSAvoidsystem complexity

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The virtual storage appliance serves multiple functions simultaneously: it acts as a cache layer, a metadata server, and a data synchronization point between object storage and workloads. This multi-functionality consolidates what could be multiple separate components into a single unified system, managing complexity while delivering cache performance benefits.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The virtual storage appliance implements self-service mechanisms through automated cache management, including intelligent data placement policies, automatic cache eviction, and dynamic synchronization with object storage. These self-managing features reduce the operational complexity and manual intervention required compared to manually managed caching systems.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS12386789B1Facilitating analytics involving cold data tiered to an object store by an on-premises storage solution by creating a virtual storage appliance on the object store

Publication Date: 2025.08.12 NETAPP INC

AI summary

Systems and methods for creating a virtual storage appliance on an object store to which an on-premises storage solution has tiered cold data are provided. According to one embodiment, a physical storage system deployed in an on-premises environment replicates hot data of a file system of the physical storage system stored on a set of disks associated with the physical storage system to an object storage service of a hyperscaler to which cold data of the file system has previously been tiered out by the file system. An instance of a virtual storage system may then be created within the hyperscaler that utilizes the object storage service as a primary storage tier. Finally, analytics involving a dataset, including both the data and the cold data, may be performed by accessing the dataset via the virtual storage system.