LSM Tree Compaction via Internal Metadata Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional relational database management systems (RDBMS) are inefficient for storing large-scale data and handling high-concurrent access in web applications, as they require transaction consistency and complex SQL queries, which are not necessary for many web applications, leading to the need for a more efficient storage architecture like key-value (KV) stores with Log Structured Merge (LSM) trees.
Innovation Solution
A key-value storage system with a storage device that includes a plurality of memory blocks, where user data pieces are stored and meta data pieces are generated, triggering a compaction process when the highest level exceeds a storage limit, generating compacted meta data pieces by excluding overlapping elements and performing garbage collection to delete corresponding user data pieces, thereby optimizing storage and reducing write amplification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional RDBMS is used for storing large-scale data and handling high-concurrent access, then transaction consistency guarantee and support of complicated SQL queries are provided, but storage efficiency and system performance deteriorate
Solution Approach 1:
The patent extracts and removes unnecessary complex SQL query processing and transaction consistency mechanisms from the storage system, retaining only the essential key-value storage functionality. This extraction allows the system to achieve high storage efficiency and concurrent access performance while maintaining basic data integrity through simplified LSM tree structure and garbage collection mechanisms.
2Productivity
If compaction process is performed by the host, then data merging and sorting are completed, but write amplification increases and storage device lifespan decreases
Solution Approach 1:
The patent implements self-service by enabling the storage device to autonomously perform compaction operations on its own without requiring continuous intervention from the host system. The storage device automatically triggers compaction when storage limits are reached, manages meta data pieces internally, and performs garbage collection to delete victim user data elements, thereby reducing write amplification and extending device lifespan while maintaining data merging efficiency.
3Speed
If meta data pieces are stored in the highest level of LSM tree, then data access is optimized, but storage limit is exceeded leading to frequent compaction
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting storage limits and compaction thresholds based on current storage conditions and data access patterns. The system monitors storage utilization in the highest level and automatically triggers compaction only when necessary, optimizing the balance between data access speed and storage capacity utilization.
4Quantity of substance
If garbage collection is performed to delete victim user data elements, then storage space is reclaimed, but additional I/O operations are required
Solution Approach 1:
The patent implements preliminary action by pre-identifying and marking victim user data elements during normal operation before they are fully deleted. The storage device maintains metadata about which data elements can be safely removed, allowing garbage collection to proceed more efficiently by processing only identified candidates rather than scanning all data, thus reducing the time and I/O operations required for space reclamation.
Data Source
AI summary
In a compaction scheme for a log structured merge (LSM) tree, a storage device is configured to: receive a first user data piece from a host; generate a first meta data piece for a highest level of the LSM tree, corresponding to the first user data piece; when the highest level exceeds a set storage limit, trigger a compaction process on the first and second meta data pieces to generate compacted meta data pieces excluding overlapping meta data elements of the second meta data pieces overlapped with the first meta data piece. Through a garbage collection, victim user data elements corresponding to the overlapping meta data elements are deleted.