Key-Value Store Table Stack for Parallelism and I/O Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Key-value stores face inefficiencies in parallelism, input/output efficiency, and data integrity due to dynamic space allocation and internal bottlenecks, particularly in solid-state drives, which affect performance under varying workloads and require optimized storage and retrieval strategies.
Innovation Solution
A stack of tables with monotonically decreasing page sizes is implemented, where each page addresses a larger key space range in subsequent tables, using Bloom filters for efficient page determination and a RAM table for caching recently used entries to manage key-value entries across multiple storage tiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If dynamic space allocation is used in key-value stores, then storage flexibility is improved, but parallelism and input/output efficiency deteriorate due to internal bottlenecks
Solution Approach 1:
The key-value store is segmented into a stack of tables with monotonically decreasing page sizes. Each table handles a specific range of the key space, with table 0 having the most pages and table N having the fewest pages. This segmentation eliminates internal bottlenecks by distributing operations across multiple independent tables, thereby improving parallelism while maintaining storage flexibility through the hierarchical structure.
Solution Approach 2:
The invention introduces a hierarchical dimension to the key-value store by organizing data across multiple tables with different page sizes. Instead of a single flat structure, data is distributed across N+1 tables where each table operates independently. This dimensional transformation allows concurrent access to different tables, improving parallelism while the varying page sizes maintain adaptability to different storage requirements.
2Device complexity
If a single table structure is used for key-value storage, then device complexity is reduced, but performance under varying workloads deteriorates due to internal bottlenecks
Solution Approach 1:
The system dynamically selects which table to access based on the key space range and workload characteristics. Each table is configured with a specific number of pages that monotonically decreases with table index, allowing the system to adapt to varying workload patterns. This dynamic selection mechanism improves workload performance while maintaining relatively simple individual table structures.
3Quantity of substance
If larger page sizes are used in all tables, then storage capacity is improved, but retrieval efficiency deteriorates due to scanning overhead
Solution Approach 1:
Different tables in the stack have different page sizes optimized for their specific function. Table 0 has the most pages with smaller individual page sizes for fine-grained access, while subsequent tables have progressively larger pages for broader coverage. This local optimization of page sizes allows efficient retrieval by directing queries to the appropriate table level, avoiding unnecessary scanning overhead while maintaining adequate storage capacity.
Data Source
AI summary
Organization and management of key-value stores is described. An example method includes providing a stack of tables. Each of the tables includes a set of pages of a pre-determined size for storing a set of key-value entries. The method includes monotonically decreasing a number of pages in the tables according to a position of the table in the stack. The method includes configuring each of the pages in the tables to address a particular range of a key space in such a way that each of the pages in a subsequent table in the stack is configured to address a range addressed by at least two pages in a preceding table in the stack. An action with a key-value entry is carried out by starting with a table in the top of the stack and moving to the next table if the action cannot be carried out.