3D Probabilistic Data Structure for Parallel Key Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current probabilistic data structures in database systems are inefficient for parallel key checks, lack memory efficiency, and are not optimized for open-range or closed-range key lookups, leading to wasted network bandwidth and false positives.
Innovation Solution
Implementing a three-dimensional probabilistic data structure with multiple bitmaps that encode database keys as transitions between characters, allowing parallel checks and maintaining lineage information to reduce false positives, along with compression to optimize memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional probabilistic data structures are used for key lookups, then memory usage is reduced, but parallel key checks become inefficient and false positives increase
Solution Approach 1:
The patent segments the probabilistic data structure into multiple independent bitmaps, each responsible for checking specific portions of database keys. This segmentation enables parallel processing of key checks across different bitmaps while maintaining accurate lineage tracking for each segment, thereby improving both parallel efficiency and reducing false positives through precise origin identification.
Solution Approach 2:
The patent introduces a new dimension to traditional probabilistic structures by organizing bitmaps in a multi-dimensional array configuration. This dimensional transformation allows simultaneous access to multiple bitmaps for parallel key checks while preserving the ability to trace lineage information, resolving the contradiction between parallel efficiency and false positive reduction.
2Loss of energy
If probabilistic data structures are implemented in database nodes, then network bandwidth is saved, but memory efficiency is poor and transaction processing rates decrease
Solution Approach 1:
By dividing the probabilistic data structure into segmented bitmaps that can be independently accessed and processed, the patent enables efficient parallel transaction processing while maintaining compact memory representation. This segmentation allows database nodes to quickly evaluate multiple keys simultaneously without proportionally increasing memory usage, thus improving transaction processing rates while preserving network bandwidth savings.
Solution Approach 2:
The patent optimizes memory efficiency by dynamically adjusting parameters such as bitmap size, number of bitmaps, and lineage information storage based on specific workload requirements. This parameter tuning allows the system to achieve better memory efficiency while maintaining high transaction processing rates, resolving the contradiction between network bandwidth savings and processing performance.
3Reliability
If lineage information is maintained for each key transition, then false positives are reduced, but memory footprint increases
Solution Approach 1:
The patent segments lineage information storage across multiple bitmaps rather than maintaining complete lineage for each key in a single structure. Each bitmap stores condensed lineage data for its specific segment, reducing overall memory footprint while preserving the ability to trace key origins and eliminate false positives through coordinated verification across segments.
Solution Approach 2:
The patent extracts only the essential lineage information needed for false positive detection from complete key paths. By storing minimal sufficient lineage data in each bitmap (such as hashed prefixes or selective markers) rather than full key histories, the system maintains false positive reduction capability while significantly reducing memory footprint.
Data Source
AI summary
Techniques are disclosed relating to probabilistic data structures. A database node may maintaining a probabilistic data structure capable of encoding database keys. The probabilistic data structure may include a plurality of levels that are each capable of storing an indication of a transition between successive characters in a database key. The database node may insert a particular database key into the probabilistic data structure and the particular database key may comprise a series of characters. The inserting may include setting, for each transition between successive characters of the series of characters, an indication in a corresponding level of the plurality of levels that is indicative of that transition. The database node may further maintain lineage information specifying one or more lineages that correspond to the transition.