Multi-Pattern Search Using Segmented Suffix Prefix Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-pattern searching algorithms, such as the Wu-Manber method, are impractical for handling hundreds of millions to billions of search patterns due to their reliance on primary storage and inefficient data structures, leading to poor performance when moved to secondary storage.
Innovation Solution
The system employs a preprocessing engine that populates SUFFIX, PREFIX, and PATTERN tables, using linear residual pointers and sorted hash values to facilitate efficient searches, allowing these tables to be stored in secondary storage and enabling the handling of potentially billions of patterns, with a pattern analysis engine comparing patterns against source files to determine similarity metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional multi-pattern searching algorithms (e.g., Wu-Manber) are used, then search speed is improved through optimized data structures, but the system cannot handle hundreds of millions to billions of patterns due to primary storage limitations
Solution Approach 1:
The patent segments the large-scale pattern storage system into multiple components: PREFIX tables stored in primary storage for quick access, and SUFFIX tables stored in secondary storage for capacity. This segmentation allows the system to handle billions of patterns by distributing data across different storage layers, resolving the contradiction between quantity handling and system complexity.
Solution Approach 2:
The patent introduces a new dimensional approach by utilizing secondary storage (disk) in addition to primary storage (memory). This dimensional expansion from single-layer to multi-layer storage architecture enables the system to accommodate hundreds of millions to billions of patterns while maintaining search performance through strategic data placement and access patterns.
2Quantity of substance
If data structures are moved to secondary storage to handle massive patterns, then pattern quantity capacity is improved, but search performance deteriorates due to slower access speeds
Solution Approach 1:
The patent extracts the most frequently accessed pattern data (PREFIX tables) and keeps them in primary storage, while moving less frequently accessed data (SUFFIX tables) to secondary storage. This extraction strategy ensures that critical search operations remain fast while still allowing the system to handle massive numbers of patterns through the secondary storage layer.
Solution Approach 2:
The patent performs preliminary organization and indexing of pattern data before storage, creating optimized data structures with pointers and indexes that enable efficient retrieval. This preliminary action on data organization allows the system to maintain fast search performance even when patterns are stored across secondary storage, by pre-computing and storing access paths.
3Speed
If more patterns are stored in primary storage to improve search speed, then search performance is improved, but the number of patterns is limited by available memory
Solution Approach 1:
The patent creates a universal storage architecture where the same search engine can operate on patterns stored in either primary or secondary storage. The system provides multi-functionality by handling both fast-access scenarios (patterns in primary storage) and large-scale scenarios (patterns in secondary storage) through a unified interface, allowing the system to adapt to different performance and capacity requirements.
Data Source
AI summary
To perform multi-pattern searching, a preprocessing engine populates a SUFFIX table, a PREFIX table and a PATTERN table. The SUFFIX table combines data conventionally stored in SHIFT and HASH tables. Pointers in the SUFFIX table refer to corresponding segments in the PREFIX table. Each PREFIX table segment is sorted by a prefix hash. A PATTERN table includes a hash of each full pattern sorted and grouped into segments, with each segment corresponding to a suffix hash and prefix hash combination. Pointers in the PREFIX table refer to corresponding segments in the PATTERN table. The PREFIX and PATTERN can be kept in secondary storage, allowing potentially billions of patterns to be used. After preprocessing, patterns are evaluated against a source file. A document metric is determine to qualitatively describe the similarity between the source file and each pattern file.