File System–Integrated Key-Value Store for Lower Write Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional key-value stores suffer from write amplification, inefficiencies in logical separation of keys, and lack of integration with file system functionalities, leading to reduced performance and resiliency.
Innovation Solution
Integrate a key-value store with a file system to leverage data management and resiliency features, store keys separately from value data items, and implement optimizations such as random distribution search, key splitting, and keyspaces to reduce write amplification and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If key-value pairs are stored together in an append log of a log structured merge tree, then the key-value store can be implemented externally to the file system, but write amplification occurs because both key and value data item are rewritten during merge operations
Solution Approach 1:
The patent segments the key-value store into two separate components: keys are stored in the file system while value data items are stored separately. This segmentation allows the key-value store to leverage file system functionalities without storing everything together in a single append log, thereby reducing write amplification during merge operations.
Solution Approach 2:
The patent extracts the value data items from the traditional append log structure and stores them separately in the file system. Only keys are maintained in the log structured merge tree, which reduces the amount of data rewritten during merge operations and eliminates write amplification for value data items.
2Adaptability or versatility
If conventional key-value stores are implemented externally to the file system, then they can operate independently, but they cannot leverage data management functionality, resiliency, and storage management features provided by the file system
Solution Approach 1:
The patent merges the key-value store with the file system by storing keys in the file system's log structured merge tree and value data items separately in the file system. This integration allows the key-value store to leverage file system functionalities such as data management, resiliency features, and storage management while maintaining operational independence through the separate value storage mechanism.
Solution Approach 2:
The patent makes the file system serve multiple functions: it acts as both the storage medium for value data items and the management system for the key-value store. By integrating these functions, the system gains access to universal file system features like snapshots, backups, and data integrity checking while maintaining key-value store operations.
3Device complexity
If keys and value data items are stored together, then storage structure is simple, but logical separation of keys is inefficient and performance is reduced
Solution Approach 1:
The patent segments the storage structure into distinct components: keys are stored in the log structured merge tree with efficient indexing, while value data items are stored separately in the file system. This segmentation enables efficient key lookup operations while maintaining a relatively simple overall storage structure that leverages the file system's native capabilities.
Data Source
AI summary
Techniques are provided for key-value store and file system integration to optimize key value store operations. A key-value store is integrated within a file system of a node. A log structured merge tree of the key-value store may be populated with a key corresponding to a content hash of a value data item stored separate from the key. A random distribution search may be performed upon a sorted log of the log structured merge tree to identify the key for accessing the value data item. A starting location for the random distribution search is derived from key information, a log size of the sorted log, and/or a keyspace size of a keyspace associated with the key.