Amortizing Metadata Updates in Log Structured Storage via LSM Tree
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face inefficiencies in metadata updates due to data delete operations, particularly when dealing with large amounts of metadata, as deleted data chunks can become randomly spread across multiple segments, increasing the number of metadata update operations and overall costs.
Innovation Solution
Implementing a segment database and a deleted chunk database configured as a log structured merge (LSM) tree, allowing for efficient memory usage and reduced metadata updates through ordered traversals and segment cleaning operations to recover storage space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deleted data chunks are tracked using conventional metadata structures, then data location changes can be recorded, but the number of metadata update operations increases significantly due to random distribution across segments
Solution Approach 1:
The patent segments the metadata management by creating a separate deleted chunk database distinct from the main segment database. This segmentation allows deleted chunk metadata to be managed independently, preventing it from increasing the update operations on the main metadata structure. The deleted chunk database is organized into multiple levels (L0, L1, L2) that segment the storage and processing of deleted chunk information.
Solution Approach 2:
The deleted chunk database acts as an intermediary structure between the segment database and the physical storage segments. Instead of directly updating metadata on every segment when chunks are deleted, the system uses this intermediary database to track deleted chunks, which then enables batch processing and optimization of the actual metadata updates through ordered traversals.
2Quantity of substance
If deleted chunk metadata is stored in a traditional database structure, then data can be tracked, but memory usage efficiency decreases and metadata update costs increase
Solution Approach 1:
The patent transitions from a traditional flat database structure to a multi-dimensional hierarchical structure with multiple levels (L0, L1, L2). This dimensional change allows the system to organize deleted chunk metadata by recency and frequency of access, with L0 containing recently deleted chunks in memory and L1/L2 containing older data on storage. This hierarchical dimensionality optimizes memory usage by keeping only essential data in expensive memory resources.
Solution Approach 2:
The system changes the organizational parameters of the metadata database by implementing a log-structured merge tree with specific parameters: level hierarchy (L0, L1, L2), size thresholds for each level, and ordered traversal sequences. These parameter changes enable the database to adapt its structure based on data characteristics, optimizing memory usage and update efficiency dynamically.
3Productivity
If segments are cleaned to recover storage space, then storage utilization improves, but the complexity of tracking and recovering deleted chunks increases
Solution Approach 1:
The patent performs preliminary organization of deleted chunk metadata into the multi-level hierarchical structure before the actual segment cleaning operation. By pre-sorting and organizing the deleted chunk information in the deleted chunk database with ordered traversals defined, the system eliminates the need for complex real-time tracking during the cleaning process itself, reducing the operational complexity while maintaining high recovery efficiency.
Data Source
AI summary
Techniques for amortizing metadata updates due to data delete operations in data storage systems that implement log structured storage of data from virtual volumes. The techniques employ a segment database (DB) and a deleted chunk DB. The segment DB is implemented as a key-value store. The deleted chunk DB is likewise implemented as a key-value store, but configured as a log structured merge (LSM) tree. By configuring the deleted chunk DB as an LSM-tree, more efficient use of memory and improved reduction of metadata updates can be achieved. Stored segments of log structured data can also be effectively “cleaned” in a background process that involves ordered traversals of the segment DB and the deleted chunk DB, allowing for more efficient recovery of storage space consumed by the deleted data chunks.