Database Data-Page Storage in Cloud Object Storage Using LSM Trees
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storing data pages of a database in cloud object storage is challenging due to the differences in latency and throughput between traditional block storage and cloud object storage, leading to poor performance and significant amplification factors such as read and write amplification.
Innovation Solution
Implementing a storage solution that utilizes a log structured merge (LSM) tree organization in cloud object storage, with multiple levels of sorted string tables and a clustering key mechanism to optimize data organization and access, reducing write amplification and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If cloud object storage is used for database storage, then storage scalability and cost efficiency are improved, but latency increases significantly (10x difference: 100-300ms vs 10-30ms)
Solution Approach 1:
The patent segments the storage system into multiple layers: hot data layer (frequently accessed data pages), warm data layer (less frequently accessed), and cold data layer (rarely accessed). This segmentation allows frequently accessed data to be stored in low-latency block storage while less frequently accessed data resides in cost-effective object storage, thus reducing overall latency while maintaining storage scalability.
Solution Approach 2:
The patent introduces an intermediary storage layer (block storage) between the database and cloud object storage. This intermediary buffers and caches data pages, absorbing the latency difference between fast database operations and slower object storage access. The block storage layer acts as a mediator that presents low-latency access to the database while eventually persisting data to the object storage layer.
2Loss of time
If larger block sizes are used in cloud object storage, then latency cost per operation is amortized, but random-access patterns at block level become less efficient
Solution Approach 1:
The patent applies different block size strategies to different data access patterns. For sequential access workloads, larger block sizes (order of megabytes) are used to amortize latency costs. For random access workloads, smaller block sizes (order of kilobytes) are used to maintain access efficiency. This local quality adaptation allows the system to optimize for the specific access pattern of each data set.
Solution Approach 2:
The patent dynamically adjusts block sizes based on access patterns and data characteristics. The system monitors access patterns and automatically selects appropriate block sizes, transitioning between small and large blocks as needed. This dynamic adaptation resolves the contradiction by allowing the system to optimize for latency amortization when using large blocks while maintaining random-access efficiency when using small blocks.
3Productivity
If traditional block storage is used, then random-access patterns are efficient, but storage scalability and cost efficiency are limited compared to cloud object storage
Solution Approach 1:
The patent merges the advantages of both block storage and object storage by combining them in a unified architecture. Block storage provides efficient random access for active data, while object storage provides scalable, cost-effective storage for less frequently accessed data. The system combines these two storage types into a single virtualized storage layer that presents a unified interface to the database, thereby achieving both random-access efficiency and storage scalability.
Data Source
AI summary
Examples described herein provide a computer-implemented method for storing data pages of a database in a remote storage. The method includes receiving a write request associated with one or more data pages of the database. The method further includes storing, in a first storage layer having a write ahead log, the write request associated with the one or more data pages. The method further includes storing, in a first write buffer, the received write request associated with the one or more data pages that each include a legacy data page identifier. The method further includes storing, in a second storage layer having a second write buffer, a sorted string table (SST) of the received write request. The method further includes storing, in a top level of a log structured merge (LSM) tree disposed in the remote storage, the SST from the second write buffer.