Key-Value Storage IO Stack Simplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional key-value storage systems implemented on traditional file storage systems suffer from suboptimal performance due to a long IO path with multiple layers, leading to increased write amplification and resource utilization, which deteriorates system efficiency and reduces the lifespan of NAND flash memory.
Innovation Solution
A storage system that simplifies the IO stack by decoupling data movement from NAND flash memory garbage collection, removing the intermediate chunk layer, and using a volatile buffer, persistent journal, and periodic snapshots to maintain data consistency and reduce access latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If key-value store is implemented on conventional file storage to take advantage of existing IO stack, then compatibility and ease of operation are improved, but IO path length increases and write amplification worsens
Solution Approach 1:
The patent extracts the key-value store implementation from the conventional file storage layer and places it directly on the NAND flash memory, eliminating the intermediate file system layers. This removes the long IO path while maintaining compatibility through the preserved IO stack interface, directly resolving the contradiction between ease of operation and device complexity.
Solution Approach 2:
The patent segments the storage system into distinct functional layers: the key-value store layer directly interfacing with NAND flash memory, and the IO stack layer preserved for compatibility. This segmentation allows each layer to operate optimally without the burden of intermediate processing layers, reducing IO path length while maintaining external compatibility.
2Adaptability or versatility
If multiple layers provide separate data processing operations in the IO path, then functional capability is improved, but write amplification increases and resource utilization worsens
Solution Approach 1:
The patent merges the data processing operations that were previously distributed across multiple intermediate layers into a single optimized path. The key-value store directly manages data on NAND flash memory, combining what were separate processing functions into one efficient operation, thereby reducing write amplification while maintaining data processing capability.
Solution Approach 2:
The key-value store implementation performs its own data management and processing operations directly on the NAND flash memory without requiring intermediate processing layers. This self-service approach eliminates redundant data processing across multiple layers, reducing write amplification and improving resource utilization while preserving full data processing functionality.
3Stability of the object's composition
If periodic sorting of keys is performed to maintain order, then data organization is improved, but system burden and resource utilization increase
Solution Approach 1:
The patent applies preliminary action by pre-sorting keys before they are stored in the key-value store. This initial sorting ensures that keys are already in the correct order, eliminating the need for periodic re-sorting operations. The system maintains key order stability without the recurring productivity loss from repeated sorting, as the ordering is established once during data ingestion.
Data Source
AI summary
During operation, a key-value storage system can receive a request to write data to a data region in a first non-volatile memory. The system can determine a key associated with the data and the key can correspond to an entry in a data structure maintained by a volatile memory the storage system. In response to determining the key, the system can write the data to the data region in the first non-volatile memory. The system can update in the volatile memory the data structure entry corresponding to the key with a physical location in the data region of the first non-volatile memory to which the data is written. The system can write the data structure update to a second non-volatile memory. The system can then store a snapshot of the data structure in a metadata region associated with the first non-volatile memory, thereby allowing persistent storage of the data structure.