LSM Tree Cache Staging with Segmented Dual Caches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current object storage platforms face inefficiencies such as high write amplification and latency due to unsupported overwriting and handling of small files, leading to costly and inefficient data operations, as well as inconsistencies during data staging, which can result in corrupted data and user experience degradation.
Innovation Solution
A computerized method and system utilizing a log-structured merge (LSM) tree file system with a dual caching mechanism, where a live data cache is converted to a stable cache, and a cloned live data cache is generated to enable parallel data staging and writing, ensuring consistent data transfer and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is staged from cache to object storage platform, then data consistency is improved, but system latency increases and cache availability decreases
Solution Approach 1:
The cache is segmented into multiple independent cache partitions, allowing data staging operations to be performed on one partition while other partitions remain available for new data writes. This segmentation enables parallel operations and prevents the entire cache from being blocked during staging.
Solution Approach 2:
The system performs preliminary actions by pre-configuring multiple cache partitions and establishing the rotation mechanism in advance. When staging is needed, the system has already prepared alternative cache partitions that can immediately accept new data, eliminating the need to wait for staging completion before accepting new writes.
2Productivity
If data staging is performed on the cache, then data can be transferred to storage, but the cache becomes occupied and unresponsive to new data
Solution Approach 1:
The cache is divided into multiple independent partitions that can operate independently. While one partition is occupied with data staging, other partitions remain fully available to receive and process new data writes, maintaining cache availability and system responsiveness throughout the staging operation.
Solution Approach 2:
The rotation mechanism ensures continuous operation by seamlessly switching between cache partitions. New data writes continue uninterrupted during staging operations by being directed to available partitions, ensuring the cache remains operational and responsive without idle time.
3Productivity
If the cache accepts new data during staging, then cache utilization is maintained, but data inconsistencies and corruption may occur
Solution Approach 1:
By segmenting the cache into isolated partitions, the system allows new data to be written to one partition while another partition undergoes staging operations. This physical and logical separation ensures that new writes cannot interfere with or corrupt data being staged, maintaining data consistency while keeping the cache fully utilized.
Solution Approach 2:
The cache rotation mechanism acts as an intermediary that manages data flow between different cache partitions and the storage system. It coordinates write operations and staging operations across partitions, ensuring that data consistency is maintained while maximizing cache utilization through parallel operations.
4Quantity of substance
If object storage platform is used directly for data operations, then storage capacity is achieved, but write amplification and cost increase
Solution Approach 1:
The system performs preliminary actions by batching multiple data writes in the cache before transferring them to object storage as a single operation. This preliminary aggregation of data reduces the number of separate write operations to the storage platform, minimizing write amplification and associated costs while maintaining full storage capacity utilization.
Data Source
AI summary
The disclosure herein describes writing data to a log-structured merge (LSM) tree file system on an object storage platform. Write data instructions indicating data for writing to the LSM tree file system are received. Based on the received instructions, the data is written to the first data cache. Based on an instruction to transfer data in the live data cache to the LSM tree file system, the first data cache is converted to a stable cache. A second data cache configured as a live data cache is then generated based on cloning the first data cache. The data in the first data cache is then written to the LSM tree file system. Use of a stable cache and a cloned live data cache enables parallel writing data to the file system by the stable cache and handling write data instructions by the live data cache.