Succinct Trie Partitioning for Memory Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems using succinct trie data structures face inefficiencies in memory usage and query performance due to the large size of index data structures, particularly in LSM Tree Structures, where traditional node-and-pointer implementations are not memory-efficient, and existing partitioning methods do not effectively reduce the depth of trie structures.
Innovation Solution
A method for partitioning a succinct trie by separating sub-tries from the main trie using specific sequences of values, updating entries in the 'values' sequence with memory addresses of sub-tries, and reducing the depth of the trie structure to improve query performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a traditional node-and-pointer implementation of a trie is used, then the trie structure is simple to implement, but the memory footprint becomes excessively large
Solution Approach 1:
The patent transforms the trie from a traditional node-and-pointer structure into a succinct representation using four bit-packed sequences. Each sequence stores specific trie information in a compressed format: keys store key prefixes, labels store node labels, first_child stores parent-child relationships, and next_sibling stores sibling relationships. This parameter transformation reduces memory usage by packing multiple bits of information into single memory locations rather than using separate pointer objects.
2Loss of time
If the trie depth is reduced to improve query performance, then query time decreases, but the trie structure becomes less comprehensive in representing all data
Solution Approach 1:
The patent partitions the large trie into multiple smaller sub-tries, each stored in separate memory blocks. The succinct trie structure maintains references to these sub-tries through the four sequences, allowing the system to benefit from reduced query depth in each sub-trie while maintaining complete data representation across all partitions. This segmentation enables parallel processing and reduces the maximum query path length.
Solution Approach 2:
The patent introduces a new dimension of organization by storing the trie horizontally across four parallel sequences rather than vertically in a traditional tree structure. This dimensional transformation allows efficient random access to any node's properties (key, label, children, siblings) without traversing the entire path from root, effectively reducing query time while maintaining complete data representation.
3Speed
If filters and indexes are kept in high cost storage media for fast querying, then query speed improves, but storage cost increases
Solution Approach 1:
The patent applies parameter changes by transforming the storage format of the index from traditional pointer-based nodes to succinct bit-packed sequences. This transformation reduces the memory footprint of the index structure, allowing it to fit in smaller, faster storage media with lower cost, while maintaining random access capabilities for fast querying through direct computation of node positions from the sequences.
Data Source
AI summary
The present disclosure provides a method comprising receiving an instruction to partition a trie in a database, wherein the trie comprises a first sequence of values comprising information that indicates labels of keys in the trie, a second sequence of values comprising information that indicates whether the labels in the first sequence have one or more child nodes, a third sequence of values comprising information that indicates whether the labels in the first sequence are first child nodes under a parent node, and a fourth sequence of values comprising information that indicates values that corresponds to the labels in the first sequence; separating a sub-trie from the trie by removing entries corresponding to the sub-trie from the first, the second, the third, and the fourth sequences; and updating an entry in the fourth sequence corresponding to a parent node of the sub-trie using a memory address of the sub-trie.