Tree-Based Dictionary Memory Management for Large Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for creating dictionaries are inefficient in handling large volumes of data, particularly multi-millions of entries, as they either rely on hash tables that restrict search to whole words or phrases or database indexes that exceed main memory limits, leading to slow lookup rates and inability to perform prefix or suffix searches effectively.
Innovation Solution
A method for creating a compact tree-based dictionary using a continuous array in memory, allowing for efficient storage and retrieval of data elements, with support for prefix and suffix searches, by distributing the array across primary and secondary memory and employing a PATRICIA tree architecture with UTF-8 coding for efficient memory management and search operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If hash tables are used for dictionary storage, then lookup speed is improved, but search flexibility is restricted to whole words only
Solution Approach 1:
The patent segments the dictionary storage into a trie data structure where words are divided into character sequences. Each node represents a character or substring, allowing the structure to be divided into hierarchical levels that support both complete word lookup and partial word (prefix/suffix) search operations efficiently.
Solution Approach 2:
The patent transitions from the traditional hash table's flat one-dimensional storage to a multi-dimensional trie structure where words are organized in hierarchical levels based on character positions. This dimensional transformation enables simultaneous support for exact matching and partial matching operations.
2Quantity of substance
If database indexes are used for dictionary storage, then storage capacity is improved, but lookup speed deteriorates due to main memory limitations
Solution Approach 1:
The patent segments the large dictionary database into a trie structure that can be loaded in hierarchical portions into main memory. The hierarchical nature allows frequently accessed nodes to remain in memory while less frequently accessed parts can be stored on secondary storage, optimizing the balance between storage capacity and access speed.
Solution Approach 2:
The patent performs preliminary organization of dictionary data into a trie structure during the indexing phase, creating a memory-efficient layout that minimizes memory access requirements during lookup operations. This preliminary structuring enables faster retrieval by reducing the need for random access to secondary storage.
3Weight of stationary object
If traditional dictionary structures are used, then memory efficiency is improved, but ability to handle large datasets deteriorates
Solution Approach 1:
The patent implements a nested trie structure where subtrees can be recursively organized and potentially stored in compressed or external formats. This nesting allows the dictionary to handle large datasets by organizing them in a space-efficient hierarchical structure that can be selectively loaded into memory.
Solution Approach 2:
The patent creates a dynamic trie structure that can adapt its memory usage based on the size of the dataset. The structure allows for dynamic allocation of nodes and can expand or contract its in-memory footprint depending on the working set size, enabling it to handle datasets of varying magnitudes efficiently.
Data Source
AI summary
The invention refers to a method creating a data structure including many data elements comprising the steps of receiving multiple data elements to be stored from an input data structure, providing a continuous array in memory, allocating memory space within the continuous array for the data elements and storing the data elements in the allocated memory space.In particular, the invention refers to a method for creating a tree based dictionary from data elements received from an input structure. Moreover, it refers to a method for searching data structures including tokens by means of a tree based dictionary and a method for annotating tokens in a data structure. The invention does also refer to corresponding computer systems, software and storage media.