Cloud Storage Cache Classes Across Availability Zones
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed database storage systems across multiple availability zones in cloud environments face issues with latency and cost due to inefficient data replication across these zones, which is not adequately addressed by existing technologies.
Innovation Solution
Implementing a cloud-based storage service with ephemeral storage caches and shared object storage, utilizing a cluster manager to instantiate storage caches across zones, ensuring low-latency data retrieval and durable persistence by replicating data across availability zones using NVMe-based storage and Amazon S3.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple availability zones to ensure data availability and prevent data loss, then reliability is improved, but latency and cost increase due to cross-zone data movement
Solution Approach 1:
The system segments data storage into two distinct layers: local ephemeral storage caches within each availability zone for low-latency access, and shared object storage for durable persistence. This segmentation allows data to be stored locally in hot caches while maintaining redundancy through cross-zone replication only when necessary, thereby reducing latency while preserving reliability.
Solution Approach 2:
The patent introduces shared object storage as an intermediary layer between ephemeral storage caches and the external environment. This intermediary enables durable data persistence and cross-zone replication without requiring direct cross-zone data movement for every operation, thus reducing latency and cost while maintaining data availability across zones.
2Reliability
If data is replicated across multiple availability zones to ensure data availability, then reliability is improved, but cost increases due to cross-zone data movement
Solution Approach 1:
The system applies local quality by providing high-performance local storage caches within each availability zone for frequently accessed data, while using shared object storage for less frequently accessed data and cross-zone replication. This differentiated approach ensures that data availability is maintained through selective replication only where necessary, reducing the cost of cross-zone data movement while preserving reliability.
Solution Approach 2:
The patent employs ephemeral storage caches that are inexpensive and location-specific, accepting that data may be lost locally but can be recovered from shared object storage or other zones. This approach reduces the cost of maintaining local replicas while ensuring data availability through the shared storage layer, effectively using cheap short-living local storage supplemented by more durable shared storage.
3Speed
If local storage caches are used to reduce latency, then speed is improved, but data loss risk increases in failure scenarios
Solution Approach 1:
The system implements beforehand cushioning by maintaining durable copies of data in shared object storage and potentially in other availability zones before failures occur. This preparatory replication ensures that when local ephemeral storage fails, data can be recovered from the shared storage layer or other zones, cushioning against data loss while maintaining fast local access during normal operation.
Solution Approach 2:
The patent merges ephemeral local storage caches with durable shared object storage into a unified storage system. This combination allows the system to leverage the speed of local storage for low-latency data retrieval while simultaneously using shared storage for data persistence and failure recovery, thus achieving both speed and reliability simultaneously.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Techniques are disclosed relating to managing distributed storage of data for various entities according to classifications for these entities. A database node of a distributed storage system may receive, from a first entity of a plurality of entities, a request to store a set of data. The database node may further obtain metadata associated with the first entity, wherein the metadata specifies one of a plurality of classifications for the entities. The database node may provide the set of data to one or more of a plurality of caches for storage. The caches may be located in two or more availability zones and are configured to store the set of data based on the classification for the first entity identified in the metadata associated with the first entity. The database node may also store the set of data in a shared object storage coupled to the database node.