Native Key-Value Storage Distributed System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems that present a key-value interface face inefficiencies due to the need for additional software layers to map key-value data structures onto block interfaces, impacting performance and scalability.
Innovation Solution
A hybrid block-based native key-value distributed storage system that utilizes KV-SSDs, allowing for native key-value support throughout the system, eliminating the need for block-based mapping and enabling direct key-value operations, with a namenode and datanodes storing key-value tuples on KV-SSDs, and a client device with a hybrid storage and networking stack.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If key-value data structures are mapped onto block interface using additional software layers, then compatibility with block storage devices is achieved, but system complexity and overhead increase
Solution Approach 1:
The patent extracts the key-value interface functionality directly into the storage device (KV-SSD), removing the need for software layers to perform mapping operations. The storage device itself provides native key-value support, eliminating the intermediary mapping software and reducing system complexity while maintaining adaptability.
Solution Approach 2:
The patent introduces an intermediary translation layer within the storage device that converts block interface operations into native key-value operations. This intermediary mechanism enables compatibility with block storage devices while maintaining native key-value performance, resolving the contradiction between adaptability and complexity.
2Quantity of substance
If key-value tuples are stored on block-based storage devices, then storage capacity is achieved, but performance and efficiency deteriorate due to mapping overhead
Solution Approach 1:
The patent extracts the mapping overhead from the software stack and relocates it to dedicated hardware structures within the KV-SSD. The key-value data structures are stored natively in the storage device without requiring software-based mapping, thereby maintaining storage capacity while dramatically improving operation performance and efficiency.
3Adaptability or versatility
If additional software layers are introduced for key-value to block mapping, then interface compatibility is achieved, but latency increases
Solution Approach 1:
The patent performs preliminary action by pre-translating block interface operations into native key-value operations within the storage device before actual data access occurs. This preliminary translation eliminates the need for runtime mapping software layers, thereby maintaining interface compatibility while significantly reducing operation latency.
Data Source
AI summary
Inventive aspects include a method, apparatus, and system for supporting a native key-value distributed storage system. The system includes a namenode having a KV-SSD and one or more datanodes each including one or more KV-SSDs. The system includes a client device that is communicatively coupled to the namenode and the one or more datanodes. The client device includes a native key-value storage and networking stack. Some embodiments include a hybrid block-based native key-value distributed storage system that supports both block-based files and native key-value tuples.