Software-Defined Storage Read Path Using Local Read-Only Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software-defined storage systems face challenges in providing flexibility to meet diverse performance and feature requirements of different workloads, with issues exacerbated by the lack of custom hardware for performance boosting.
Innovation Solution
Implementing a storage system architecture that utilizes a write cache and persistent storage, where data is cached in ephemeral local storage of virtual computing instances and synchronized with backend persistent storage, optimizing read requests by prioritizing local storage for read-only data and leveraging different storage tiers for performance and cost efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in backend persistent storage of software-defined storage system, then data capacity and persistence are improved, but read access speed and latency are worsened
Solution Approach 1:
The storage system is segmented into multiple layers: local ephemeral storage on virtual computing instances, backend persistent storage, and write cache. This segmentation allows different storage media to serve different purposes - local storage for fast read access and backend storage for persistent data retention, thereby resolving the contradiction between speed and reliability.
Solution Approach 2:
A write cache and synchronization mechanism act as intermediaries between the local ephemeral storage and backend persistent storage. The write cache stores data locally on virtual computing instances, providing fast read access while the synchronization mechanism ensures data is eventually persisted to backend storage, thus mediating between speed and reliability requirements.
2Speed
If data is cached in local ephemeral storage of virtual computing instances, then read performance is improved, but data loss risk on instance failure is increased
Solution Approach 1:
Data is pre-synchronized from backend persistent storage to local ephemeral storage before read operations occur. This preliminary action ensures that frequently accessed data is already available in fast local storage, improving read performance while the underlying persistent storage maintains data safety as a backup.
Solution Approach 2:
The write cache serves as an intermediary layer that decouples the volatility of ephemeral storage from the persistence requirements of backend storage. It allows the system to utilize fast ephemeral storage for caching while maintaining data safety through the persistent backend storage and synchronization mechanisms.
3Productivity
If storage system uses multiple storage tiers (local ephemeral, write cache, persistent storage), then performance and cost efficiency are improved, but system complexity is increased
Solution Approach 1:
The local storage of virtual computing instances serves multiple functions: it acts as ephemeral storage for the virtual machine, simultaneously functions as a read cache for the storage system, and provides fast storage for frequently accessed data. This multi-functionality reduces the need for dedicated cache hardware, simplifying the overall system architecture while maintaining performance benefits.
Solution Approach 2:
The storage system leverages existing resources (local storage on virtual computing instances) to provide caching functionality without requiring additional dedicated hardware. The virtual computing instances' local storage serves itself as cache storage, reducing system complexity by utilizing already-present resources for multiple purposes.
Data Source
AI summary
An apparatus comprises a processing device configured to receive, from one of two or more storage nodes of a software-defined storage system comprising a write cache and persistent storage, a request to read a given portion of data. One or more portions of data in the persistent storage are also stored as read-only data in local storage of virtual computing instances implementing the storage nodes. The processing device is configured to read the given portion of data from the local storage of the virtual computing instances responsive to determining that the given portion of data is part of the read-only data in the local storage of the virtual computing instances, and to read the given portion of data from the write cache or persistent storage responsive to determining that the given portion of data is not part of the read-only data in the local storage of the virtual computing instances.