Streaming String Matching With Rolling Hash and Single-Pass Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional pattern matching techniques require significant memory resources and multiple passes over large datasets, making them inefficient for real-time and large-scale data analytics, especially in asymmetric streaming settings where data exceeds available memory and time-efficient computation is crucial.
Innovation Solution
A pattern matching system that uses a rolling hash function to compute mapping values for candidate substrings in a single pass, comparing them to a query string, allowing for sub-linear memory usage and efficient detection of matches in streaming data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional pattern matching techniques are used to search large datasets, then matching accuracy is achieved, but memory resource consumption increases significantly
Solution Approach 1:
The patent segments the large text dataset into smaller substrings of fixed length (equal to query string length). Each substring is independently hashed and compared with the query string hash, eliminating the need to load the entire dataset into memory. This segmentation approach maintains matching accuracy while reducing memory consumption from O(N) to O(1) where N is the dataset size.
Solution Approach 2:
The patent introduces a hash function as an intermediary between the substring comparison and exact matching. By hashing both the query string and each substring, the system can quickly filter potential matches without performing expensive string comparisons. The hash function acts as a mediator that reduces the search space while preserving the ability to identify exact matches, thereby maintaining accuracy with reduced memory usage.
2Reliability
If multiple passes over large datasets are performed to ensure accurate pattern matching, then matching completeness is improved, but processing time increases
Solution Approach 1:
The patent implements a single-pass streaming algorithm that continuously processes substrings as they become available in the data stream. The rolling hash technique allows the hash value to be updated incrementally as the window moves through the text, eliminating the need for multiple passes. This continuous processing maintains matching completeness while reducing processing time from O(N×M) to O(N) where N is the text length and M is the query length.
Solution Approach 2:
The patent pre-computes the hash value of the query string before processing the dataset. This preliminary action allows for immediate comparison with hashed substrings during the single pass through the data, eliminating the need for repeated query processing in multiple passes. The pre-computed hash serves as a reference that enables efficient real-time matching without sacrificing completeness.
3Measurement precision
If the entire stream of data is stored to update mapping values, then pattern matching accuracy is maintained, but memory usage increases
Solution Approach 1:
The patent extracts only the necessary information (fixed-length substrings and their hash values) from the complete data stream for processing. Instead of storing the entire stream, the system extracts substrings of length equal to the query string, computes their hashes, and immediately compares them with the query hash. This extraction approach maintains pattern matching accuracy by preserving all potential matches while reducing memory usage from storing the entire stream to storing only the current window and its hash.
4Measurement precision
If conventional string matching algorithms are used, then exact match detection is achieved, but computation complexity increases
Solution Approach 1:
The patent replaces the mechanical string comparison process with a mathematical hash function. Instead of character-by-character comparison which has O(M) complexity where M is the query length, the system uses hash functions that compute mapping values in O(1) time for each substring. This substitution maintains exact match detection capability while reducing computation complexity from O(N×M) to O(N) for the entire search process.
Data Source
AI summary
Embodiments provide systems, methods, and computer storage media for determining string similarity and pattern matching in strings that arrive in a stream. A stream representing string of characters is received and used to compute mapping values that are compared to a mapping value of a query string to identify a match between strings in the stream of characters and the query string. The stream of characters is searched in a single sequential pass to detect a match or the longest matching substring with a query string. An identified match or absence of a match is provided.