Tiered Cloud Storage with Hot-Cold Data Separation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based database services with shared-nothing architectures face inefficiencies due to inelastic resource use, leading to non-optimal utilization of system resources, as all data is stored for high availability despite varying access needs.
Innovation Solution
A tiered storage system with a 'shared-nothing, shared-disc' architecture that separates frequently accessed 'hot' data for high availability on local storage and less frequently accessed 'cold' data on a disaggregated medium, allowing independent scaling of storage and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data is stored for high availability in shared-nothing architecture, then reliability is improved, but resource utilization deteriorates
Solution Approach 1:
The patent segments data into hot data (frequently accessed) and cold data (infrequently accessed), storing hot data on local storage media for high availability and cold data on shared storage media. This segmentation allows different availability levels for different data portions, improving overall resource utilization while maintaining reliability for critical data.
Solution Approach 2:
The patent applies local quality by providing high availability (local storage) only for hot data that requires it, while cold data uses shared storage with lower availability. This differentiated approach ensures reliability where needed while optimizing resource utilization across the system.
2Device complexity
If storage and computational resources are scaled together in shared-nothing architecture, then system simplicity is maintained, but adaptability deteriorates
Solution Approach 1:
The patent separates storage resources into local storage media (attached to specific compute instances) and shared storage media (accessible by multiple instances). This segmentation allows independent scaling of storage capacity without requiring proportional scaling of computational resources, enhancing adaptability while maintaining architectural simplicity.
Solution Approach 2:
The shared storage media serves multiple functions: storing cold data, providing backup capacity, and enabling resource sharing across multiple compute instances. This multi-functionality allows the system to adapt to varying workloads and scaling requirements without increasing architectural complexity.
3Reliability
If unused data is stored in high-availability data structures, then data accessibility is improved, but energy consumption increases
Solution Approach 1:
The patent segments data based on access patterns, storing frequently accessed hot data in high-availability local storage structures and infrequently accessed cold data in shared storage. This prevents unused data from occupying expensive local storage resources, reducing energy consumption while maintaining accessibility for data that actually needs it.
Solution Approach 2:
The patent applies local quality by providing high-availability storage (with associated energy costs) only for hot data requiring frequent access, while cold data resides in shared storage with lower energy consumption. This optimized allocation reduces overall energy usage while maintaining data accessibility where required.
Data Source
AI summary
According to examples, a cloud platform implementing a tiered storage that decouples Request Units from storage is disclosed. Data received from partitions that are characterized by corresponding partition keys is stored to the local storage medium of a virtual machine (VM) as hot data. The received data is formatted as immutable segments and stored to a disaggregated storage medium shared by a plurality of VMs. The immutable segments are grouped into sorted runs within the disaggregated storage medium and are further assigned to different levels. The hot data is deleted from a local storage medium when time-to-cold (TTC) criteria are met. The cold data can be accessed for document retrieval via a search key that includes a partition key and a documentID. Additional Request Units can be allocated or de-allocated for a portion of the cold data without moving the cold data.