Binary Tree Substring Search Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Performing substring searches on very large datasets is inefficient in terms of memory and processing resources, particularly when dealing with millions of records, as existing techniques require extensive comparisons and resource utilization.
Innovation Solution
Implementing a filtering process using binary trees, where the data set is partitioned based on key lengths and contents, and excluding entries that cannot contain the input string, along with parallel processing and optimized algorithms like the Knuth-Morris-Pratt algorithm, to reduce the number of substring searches and maximize system resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional substring search methods are used on large datasets, then search completeness is maintained, but memory usage and processing time increase significantly
Solution Approach 1:
The patent segments the large dataset into multiple partitions based on key length ranges. Each partition contains a subset of records grouped by their key length, allowing the search algorithm to process only relevant partitions rather than scanning the entire dataset. This segmentation reduces memory usage and improves search speed by eliminating unnecessary comparisons.
Solution Approach 2:
The patent performs preliminary actions by pre-grouping records into partitions based on key length during data ingestion or indexing. This pre-organization allows the search algorithm to quickly identify and access only the partitions that could potentially contain the search substring, avoiding the need to load or scan unrelated data into memory.
2Measurement precision
If all entries in the binary tree are searched, then search accuracy is maintained, but the number of comparisons increases
Solution Approach 1:
The patent extracts and excludes entries from the binary tree search based on the substring length criterion. Specifically, any tree entry with a key length less than the search substring length is immediately excluded from further comparison. This extraction of irrelevant entries maintains search accuracy for valid matches while dramatically reducing the number of comparisons required.
3Productivity
If filtering processes are applied to reduce comparisons, then processing speed improves, but system complexity increases
Solution Approach 1:
The patent applies local quality by implementing filtering logic specifically at the partition level and entry level rather than uniformly across the entire dataset. The filter criteria are locally optimized based on substring length characteristics, applying simpler filters to certain partitions and more sophisticated filters to others, thereby improving performance without uniformly increasing complexity throughout the system.
Data Source
Figure 1
Figure 2
AI summary
Accelerated sub-string searches on large data sets can be performed using filtering processes that can improve or optimize run time performance. A first filtering process can include partitioning a binary tree into sections to enable an exact search to replace a substring search for part of the binary tree and for part of the binary tree to be potentially excluded from substring searching. A second filtering process can include comparing count representations of entries in the binary tree and of a received input string to potentially further exclude entries from substring searching.