Bitwise Trie Confluence Index for Database Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database indexing methods, such as B-Trees, face inefficiencies in memory usage and query performance, especially with sparse data and low cardinality queries, and are limited in handling multidimensional queries and flexible key storage.
Innovation Solution
The implementation of a bitwise trie data structure that uses bitmaps to store non-empty pointers, allowing for efficient memory allocation and query processing, and enables flexible key storage and querying through the use of a 'confluence index' that combines B-Trees and inverted indexes into a single universal solution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If B-Tree indexes are used for database indexing, then data can be stored and retrieved with logarithmic time complexity, but access time increases significantly with large data sizes and query performance deteriorates for low cardinality criteria
Solution Approach 1:
The patent segments the index into multiple tries organized in a hierarchical structure (forest of tries). Each try handles a specific range of keys, allowing parallel processing and reducing the time complexity from O(log n) to O(m) where m is the maximum key length. This segmentation enables simultaneous access to multiple index partitions, dramatically improving query speed for large datasets.
Solution Approach 2:
The patent transitions from the traditional B-Tree's height-based organization to a trie structure that operates in the dimension of key character positions. By organizing data along the dimension of key prefix matching rather than hierarchical levels, the system achieves constant time complexity relative to data size, only dependent on key length.
2Speed
If traditional trie data structures are used with arrays to store child pointers, then constant time access is achieved, but memory usage becomes extremely inefficient for nodes with few children
Solution Approach 1:
The patent implements dynamic child pointer storage by using sorted arrays instead of fixed-size arrays. The number of child pointers is adjusted dynamically based on the actual number of children, and binary search is used to maintain O(log k) access time where k is the number of children. This dynamic approach allows the structure to adapt to sparse nodes while maintaining efficient access performance.
Solution Approach 2:
The patent changes the parameter of child pointer storage from fixed array size to variable size based on actual children count. By storing only the necessary number of pointers and using binary search for lookup, the memory consumption is reduced from O(alphabet size) to O(actual children), while access time remains efficient at O(log k).
3Productivity
If B-Tree indexes are used for multidimensional queries, then single-dimensional queries can be handled efficiently, but multidimensional queries involving multiple criteria cannot be joined efficiently
Solution Approach 1:
The patent creates a universal index structure (confluence index) that can handle both single-dimensional and multidimensional queries efficiently. By organizing data in a trie structure that preserves prefix properties, the system can simultaneously support point queries, range queries, and multidimensional queries with the same index, eliminating the need for separate indexes for different query types.
Solution Approach 2:
The patent merges multiple indexing capabilities into a single trie-based confluence index. Instead of maintaining separate B-Trees for different dimensions or query types, the system combines prefix-based indexing with suffix array techniques to create a unified structure that handles various query dimensions efficiently through the same index mechanism.
4Productivity
If indexes are created on attributes with low cardinality values, then query filtering should be improved, but the index actually consumes more memory without providing significant performance benefit
Solution Approach 1:
The patent applies partial indexing by creating tries only for attributes that meet certain criteria (such as minimum cardinality threshold). For low cardinality attributes, the system chooses not to create an index at all, accepting full table scans when necessary. This selective approach avoids the overhead of maintaining indexes that would provide minimal performance benefit while consuming significant memory resources.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
The invention provides a time-efficient way of performing a query in a database or information retrieval system comprising operations such as intersection, union, difference and exclusive disjunction on two or more sets of keys stored in a database or information retrieval system. In a novel execution model, all data sources and operators are tries, and in input trie for a higher-order set operation can be the output of a lower-order set operation which is evaluated on demand. Two or more input tries are combined in accordance with the respective set operation, to obtain the set of keys associated with the nodes of a respective resulting trie. The physical algebra of the implementation of tries based on bitmaps corresponds directly to the logical algebra for the set operations and allows for efficient implementation by means of bitwise Boolean operations.