N-Hamming Distance Search Index for Fast String Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search technologies are inefficient for finding quasi-random strings with errors in large databases, particularly for biological sequences like proteins and DNA, as they rely on sequential comparisons or hash indices that are not error-stable, leading to slow search times in databases with billions of records.
Innovation Solution
The method involves partitioning query and record strings into disjoint partitions, creating hash values for each partition, and using an n-Hamming search index to quickly identify records with a Hamming distance of n or less, allowing for fast and exact similarity searches by reducing the search space through indexed storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sequential comparison of each character is used to search for quasi-random strings with errors, then exact results can be obtained, but the search time becomes extremely long in databases with billions of records
Solution Approach 1:
The query string and database records are divided into multiple partitions. Hash values are computed for each partition independently, allowing parallel processing and reducing the overall search time while maintaining accurate error detection across the entire string.
Solution Approach 2:
Hash values for all database records are pre-computed and stored in an index structure before the actual search query is executed. This preliminary preparation enables fast retrieval during search operations without requiring full string comparisons at query time.
2Speed
If simple hash index is used to pick quasi-random strings from database, then search can be performed quickly, but the hash index is not error-stable and cannot identify strings with errors
Solution Approach 1:
Instead of computing a single hash value for the entire string, the method computes multiple hash values for different partitions of the string. This excessive computation of partial hashes provides error stability because if some partitions have errors, other partitions can still provide correct hash matches for filtering candidate records.
3Measurement precision
If Levenshtein distance is used to measure string similarity allowing insertions, deletions or substitutions, then comprehensive error detection is achieved, but the computation time scales at least with the length of the string
Solution Approach 1:
The string comparison problem is segmented into multiple independent partition comparisons. By computing hash values for each partition separately and combining results, the method achieves comprehensive error detection across the entire string while reducing overall computation time through parallelizable operations.
Data Source
AI summary
A method for performing an n-Hamming distance search is provided. The database comprises a plurality of records and an indexed storage. Each record comprises a record string associated with n+1 record hash values stored in an index. The method comprises partitioning a query string into n+1 query partitions. The n+1 query partitions are pairwise disjoint. A hash value is created for each query partition resulting in n+1 query hash values. Records having at least one record hash value equal to one of the n+1 query hash values are identified. The identified records are searched for records fulfilling a search condition. The search condition is that the record strings have a Hamming distance smaller than or equal to n with respect to the query string or a more limited search condition.