Sorted Radix Tries for Append-Only Database Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies for storing and accessing massive datasets, such as B-trees, face challenges including high memory requirements, potential unbalancing due to changing data distribution, and inefficiency in range queries.
Innovation Solution
A system that stores data in sorted radix tries within an append-only, disk-optimized database, optimizing storage and access efficiency even in low-memory environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If B-trees are used to store massive datasets, then efficient insertion, deletion, and search operations are achieved, but large amounts of memory are required which is challenging when working with massive datasets
Solution Approach 1:
The patent replaces the traditional B-tree data structure with a radix trie-based system that stores data directly on disk rather than in memory. This substitution of the mechanical memory-based system with a disk-based system allows the database to handle massive datasets without requiring large amounts of RAM, while maintaining efficient search operations through the radix trie structure that organizes data by prefixes.
Solution Approach 2:
The patent transitions from a two-dimensional in-memory B-tree structure to a three-dimensional disk-based radix trie structure with hierarchical organization. The radix trie creates multiple levels of indexing based on prefix lengths, adding a dimensional aspect to data organization that enables efficient searching without consuming excessive memory resources.
2Reliability
If B-trees are used to store massive datasets, then balanced data structure with efficient operations is achieved, but the data structure can become unbalanced if data distribution changes significantly over time leading to degraded performance
Solution Approach 1:
The patent implements a dynamic radix trie structure that automatically adapts to changing data distributions. The radix trie allows nodes to be dynamically created, deleted, or merged based on the actual data inserted, enabling the structure to evolve with changing data characteristics without requiring rebalancing operations or risking unbalance, thus maintaining reliable performance under various data distribution scenarios.
3Productivity
If B-trees are used to store massive datasets, then efficient insertion and search operations are achieved, but range queries require traversing multiple nodes which reduces efficiency
Solution Approach 1:
The patent segments the data into hierarchical levels within the radix trie based on prefix lengths. This segmentation allows range queries to be answered by traversing only the relevant segments of the trie rather than multiple B-tree nodes. The hierarchical structure groups data by common prefixes, enabling efficient range queries through targeted traversal of specific trie levels and branches.
Data Source
AI summary
A system stores data in sorted radix tries in an append-only, disk-optimized database. Sorted radix tries are one of the most space-efficient ways to store hierarchical data and have excellent storage and access characteristics for enormous sets of keys. The append-only nature of the database makes it ideal for storing huge volumes of generated data, such as IP addresses, DNS data, log messages, activity logs, data measurements, and transaction histories, that are inserted in batches or aggregated over time. The organization of data in the database enables it to be queried particularly efficiently, even in low-memory and low-computing resource environments.