Persistent KV Iterator with Unified Keys for Concurrent Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Key-value databases (KVDBs) face challenges in efficient data indexing and retrieval due to the need for maintaining multiple data structures, leading to increased latency, reduced throughput, and complex cross-sharding issues, particularly with secondary indexes, which introduce contention and serialization among concurrent operations.
Innovation Solution
A persistent key-value (KV) iterator is introduced, merging primary and secondary index information into a single key structure, allowing for hierarchical partitioning and concurrent access without the need for locking mechanisms, enabling efficient data traversal and updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple data structures are maintained for indexing, then data retrieval completeness is improved, but system complexity and maintenance overhead increase
Solution Approach 1:
The patent merges primary key information and secondary index information into a single unified key structure. This key includes both the primary key identifier and secondary index fields (such as time range, object type, state) that were previously maintained in separate data structures. By combining these elements, the system achieves complete data retrieval capability while reducing the complexity of maintaining multiple separate indexing structures.
2Reliability
If locking mechanisms are used during concurrent operations, then data consistency is improved, but throughput and latency are worsened
Solution Approach 1:
The unified key structure enables segmentation of data access by different index fields (primary key, secondary index, time range, object type, state). This segmentation allows different processes to access different partitions of the key space concurrently without requiring global locking mechanisms, thereby maintaining data consistency while improving throughput and reducing latency.
3Reliability
If cross-sharding issues are addressed with complex mechanisms, then data distribution reliability is improved, but system complexity increases
Solution Approach 1:
The unified key structure serves multiple functions simultaneously: it acts as the primary key identifier, contains secondary index information for filtering, and provides basis for data sharding and distribution. This multi-functionality eliminates the need for separate cross-sharding mechanisms, achieving reliable data distribution while reducing system complexity.
4Productivity
If hierarchical partitioning is implemented, then data traversal efficiency is improved, but key structure complexity increases
Solution Approach 1:
The patent adds multiple dimensional fields to the key structure (time range, object type, state) beyond the traditional single-dimensional primary key. This multi-dimensional key structure enables hierarchical partitioning and efficient data traversal across different dimensions, improving productivity while the structured organization of fields keeps the complexity manageable.
Data Source
AI summary
Various techniques are provided for enabling an iterator (called persistent key-value (KV) iterator) that uses a special key structure embedded with secondary index information for efficient search (or query) of a key-value database (KVDB). In some embodiments, the key structure comprises a plurality of fields categorized into two groups, infrastructure hierarchy fields and object characteristic fields. Keys for the KVDB may be partitioned into one or more levels of hierarchy to generate many smaller key lists for easy search/query of the KVDB. These smaller key lists can be used for concurrent searches/queries.