Network Key Value Indexing for SSD Metadata Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing key-value solid state drives (KV-SSDs) face performance issues due to the lack of efficient indexing systems, which result in high metadata overhead when implementing persistent indexing designs, impacting input/output (IO) performance and making them non-crash tolerant.
Innovation Solution
A network key value (NKV) indexing design is implemented using a hierarchical data structure in volatile memory, where keys are parsed into prefixes, buckets, and keys based on delimiters, allowing for the generation of a hash table that is crash tolerant and rebuildable without additional metadata, thereby maintaining IO performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a persistent NKV indexing design is implemented in persistent memory, then indexing capability is improved, but metadata overhead increases impacting IO performance
Solution Approach 1:
The patent extracts the indexing structure from persistent memory and places it entirely in volatile memory. The hierarchical data structure (prefix trees, hash tables) is built and maintained only in volatile memory during runtime, while persistent storage contains only the essential key-value data without indexing metadata. This extraction eliminates the metadata overhead problem while maintaining indexing capability during operation.
Solution Approach 2:
The indexing structures are treated as temporary, disposable objects that exist only in volatile memory during system operation. Upon system crash or shutdown, these indexing structures are discarded and automatically rebuilt from the key-value data on persistent storage using the O(N) listing algorithm. This approach accepts the temporary loss of indexing during recovery but eliminates the permanent metadata overhead burden.
2Ease of operation
If a persistent NKV indexing design is implemented, then indexing functionality is improved, but system crash tolerance deteriorates
Solution Approach 1:
The system performs preliminary actions by maintaining the hierarchical indexing structure in volatile memory during normal operation, enabling fast lookups. Upon crash, the system relies on the fact that the underlying key-value data remains intact on persistent storage and can be used to rebuild the indexing structure. The O(N) listing algorithm provides a preliminary mechanism to efficiently reconstruct indexes without requiring complex checkpointing or journaling mechanisms.
3Productivity
If indexing is implemented in volatile memory, then IO performance is maintained, but crash tolerance must be rebuilt upon startup
Solution Approach 1:
The patent changes the operational parameters of the indexing system by accepting O(N) rebuild complexity as a temporary startup cost in exchange for maintaining optimal IO performance during normal operation. The system parameters are configured to build the hierarchical structure in volatile memory with fast access characteristics, and the rebuild process leverages the O(N) listing algorithm to efficiently reconstruct indexes after crashes. This parameter change accepts intermittent rebuild time loss but maintains consistently high IO performance during operational periods.
Data Source
AI summary
Provided is a method of indexing in a network key value indexing system. The method includes retrieving a first key name from a storage device of the network key value indexing system, the first key name identifying a first prefix, a first bucket, and a first key, the first prefix indicating the first bucket, parsing the first key name into the first prefix, the first bucket, and the first key, determining the first prefix, the first bucket, and the first key based on a first delimiter, and generating a hash table in a memory cache of the network key value indexing system to associate the first prefix with the first key.