Key-Value Store Memory Optimization via Disk Block Addressing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed-computing systems face performance issues due to excessive memory consumption by key-value stores, which hinder client performance as they require significant memory to store keys and values, leading to resource bottlenecks.
Innovation Solution
Implementing a key-value store that stores keys and values on disk storage instead of memory, using addresses of blocks of keys to locate values, thereby reducing memory usage and optimizing resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If keys and values are stored in memory, then access speed is improved, but memory consumption increases excessively
Solution Approach 1:
The patent extracts the bulk data storage function from memory to disk storage, keeping only essential metadata (addresses of blocks) in memory. This separation allows fast access to location information while storing large volumes of data on disk, resolving the contradiction between access speed and memory consumption.
Solution Approach 2:
The patent transitions from a single-dimensional memory storage model to a hierarchical model with two dimensions: memory for metadata (block addresses) and disk storage for actual data. This dimensional change enables the system to maintain fast access characteristics while dramatically reducing memory requirements.
2Ease of operation
If individual key addresses are stored in memory, then key lookup is simplified, but memory usage increases
Solution Approach 1:
The patent merges multiple individual key addresses into a single block address that represents a collection of keys stored contiguously on disk. This consolidation reduces the number of address entries needed in memory from potentially millions of individual key addresses to a manageable number of block addresses, simplifying the addressing structure while maintaining lookup efficiency.
Solution Approach 2:
The patent segments the key-value store into discrete blocks on disk, where each block contains multiple keys and their corresponding values. This segmentation allows the system to manage data in fixed-size units, making it feasible to store block addresses in memory without requiring all individual key addresses, thus reducing memory usage while preserving lookup capability.
3Quantity of substance
If more memory is allocated to key-value store, then storage capacity is improved, but network performance deteriorates due to resource bottlenecks
Solution Approach 1:
The patent extracts the bulk storage function from the memory subsystem to the disk storage subsystem, preventing memory from becoming a bottleneck for network operations. By offloading large data storage to disk, the memory resource remains available for network processing and other performance-critical operations, thus improving overall network performance while maintaining storage capacity.
Data Source
AI summary
A computer system performs a technique for reducing memory usage when a key-value store is being implemented. A first key associated with data is received. A block address of a block of keys is obtained from memory. The block of keys is stored on disk storage, and the keys in the block of keys correspond to respective values stored on the disk storage. The block of keys is obtained from the disk storage using the block address. A second key in the block of keys is located. Locating the second key includes determining that the second key matches the first key. A value of the respective values is obtained using the second key.