Hybrid Key-Value Store Splitting Records for Latency and Capacity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face challenges in efficiently storing and retrieving key-value pairs, particularly due to limitations in in-memory storage for large values and the need for fault tolerance and low latency, which existing technologies have not adequately addressed.
Innovation Solution
A hybrid key-value store is implemented, dividing key-value pairs into smaller records for in-memory storage and larger records for secondary storage, with a version counter and index structure to manage versions and support multi-version concurrency control, while a garbage collector optimizes memory usage by removing obsolete data records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If key-value pairs are stored entirely in in-memory storage, then retrieval latency is reduced, but storage capacity is limited and cost increases
Solution Approach 1:
The patent segments the key-value store into two distinct components: a hot data tier in in-memory storage for frequently accessed small key-value pairs, and a cold data tier in persistent storage for less frequently accessed or larger data. This segmentation allows the system to optimize for both speed and capacity by placing different types of data in appropriate storage media.
Solution Approach 2:
The patent introduces a hierarchical dimension to the storage architecture by adding persistent storage below in-memory storage. This creates a multi-layered storage hierarchy that extends the effective storage capacity beyond what single in-memory systems can provide, while maintaining fast access paths for critical data.
2Speed
If large key-value pairs are stored in in-memory storage, then retrieval speed is improved, but memory usage increases and cost increases
Solution Approach 1:
The patent applies local quality by storing only the portions of key-value pairs that are frequently accessed (hot data) in in-memory storage, while storing less frequently accessed portions (cold data) in persistent storage. This selective placement optimizes memory usage by keeping only essential data in expensive in-memory resources.
Solution Approach 2:
The patent implements partial action by storing only the necessary portion of large key-value pairs in in-memory storage - specifically, the metadata and frequently accessed segments - while storing the complete data in persistent storage. This partial in-memory storage provides sufficient performance benefit without the full cost of keeping entire large objects in memory.
3Quantity of substance
If data is divided into multiple records, then storage efficiency is improved, but query complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing metadata about divided key-value pairs, including information about which segments are stored where and how to reconstruct the complete value. This metadata is prepared in advance, so that during query execution, the system can efficiently retrieve and reassemble segments without complex real-time processing.
Data Source
AI summary
A method for storing a key-value pair can include dividing the key-value pair into a first data record and a second data record. The first data record can include a key associated with the key-value pair. The second data record can include a portion of a value associated with the key-value pair. The second data record can be stored in a secondary data store based on a size of the second data record exceeding a threshold value. The first data record can be stored in an in-memory key-value store based on a size of the first data record not exceeding the threshold value. The first data record can include a reference to the second data record in the secondary data store. A query requiring the key-value pair can be executed by retrieving the first data record from the in-memory key-value store. Related systems and articles of manufacture are also provided.