Multi-Pattern Search Using Segmented Hash Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-pattern searching algorithms, such as the Wu-Manber method, are impractical for handling hundreds of millions to billions of search patterns due to their reliance on primary storage, leading to poor performance when data structures are moved to secondary storage.
Innovation Solution
The system employs a preprocessing engine that populates SUFFIX, PREFIX, and PATTERN tables, using linear residual pointers and sorted hash values to facilitate efficient searches, allowing these tables to be stored in secondary storage and enabling the handling of a large number of patterns, with a pattern analysis engine comparing patterns against source files to determine similarity metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the Wu-Manber algorithm is used for multi-pattern searching, then the search speed is improved, but the number of search patterns is limited to what can fit in primary storage
Solution Approach 1:
The patent divides the search pattern data into multiple segments that can be stored in secondary storage. Each segment is processed separately, allowing the system to handle billions of patterns by loading only relevant segments into primary storage during search operations, thus resolving the contradiction between search speed and the number of patterns.
Solution Approach 2:
The patent introduces a new dimension of storage hierarchy by moving from purely primary storage to a combination of primary and secondary storage. This dimensional change allows the system to accommodate billions of patterns in secondary storage while maintaining fast access through intelligent loading strategies into primary storage.
2Quantity of substance
If data structures are moved from primary storage to secondary storage to handle more patterns, then the number of search patterns increases, but the performance deteriorates
Solution Approach 1:
The patent performs preliminary actions by pre-processing the pattern data into segmented structures with optimized indexing before storage. This pre-organization allows for efficient retrieval during search operations, mitigating the performance penalty typically associated with secondary storage access.
Solution Approach 2:
The patent replaces the mechanical limitation of primary storage capacity with an intelligent data structure and access strategy. By using sorted segments with efficient search algorithms, the system substitutes the physical constraint with a computational solution that maintains performance while enabling secondary storage usage.
3Ease of manufacture
If conventional PAT_POINT tables are used, then the implementation is simple, but the system cannot handle billions of patterns
Solution Approach 1:
The patent replaces the conventional single-table PAT_POINT structure with segmented tables that divide patterns into manageable chunks. This segmentation maintains implementation simplicity by using familiar table structures while enabling the system to handle billions of patterns through systematic division and organized storage.
Data Source
AI summary
To perform multi-pattern searching, a preprocessing engine populates a SUFFIX table, a PREFIX table and a PATTERN table. The SUFFIX table combines data conventionally stored in SHIFT and HASH tables. Pointers in the SUFFIX table refer to corresponding segments in the PREFIX table. Each PREFIX table segment is sorted by a prefix hash. A PATTERN table includes a hash of each full pattern sorted and grouped into segments, with each segment corresponding to a suffix hash and prefix hash combination. Pointers in the PREFIX table refer to corresponding segments in the PATTERN table. The PREFIX and PATTERN can be kept in secondary storage, allowing potentially billions of patterns to be used. After preprocessing, patterns are evaluated against a source file. A document metric is determine to qualitatively describe the similarity between the source file and each pattern file.