Low-Memory String Similarity Index via Tightly-Packed Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing string similarity search methods incur latency and are inefficient for real-time search-as-you-type functionality due to their reliance on general-purpose search engines or database technologies, which introduce prefix filtering and round-trip database server interactions.
Innovation Solution
A computer-based string similarity analysis system that transforms candidate strings into features, generates a low-memory index using tightly-packed lists and maps, and employs a guaranteed rank threshold to optimize memory usage and search efficiency, allowing for near-instant results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If general-purpose search engine technology is used for string similarity search, then indexing and retrieving documents is improved, but search results become sub-optimal and latency increases due to prefix filtering reliance
Solution Approach 1:
The patent segments the search process into two distinct phases: a preprocessing phase that builds an inverted index mapping n-gram features to document positions, and a query phase that efficiently retrieves candidates using this pre-built index. This segmentation allows the system to optimize for both indexing efficiency and search result quality by using the inverted index to eliminate irrelevant documents before applying similarity metrics.
Solution Approach 2:
The patent performs preliminary action by pre-processing the document collection to build an inverted index that maps n-gram features to their occurring positions in documents. This pre-built index structure enables fast retrieval during query time without requiring complex real-time processing, thus reducing latency while maintaining high search result quality through accurate feature-based filtering.
2Productivity
If database technology is used for string similarity search, then searching values in a column is improved, but latency increases due to round-trip to database server
Solution Approach 1:
The patent extracts the search functionality from the database server by implementing a local inverted index structure that can be built and queried independently. The inverted index maps n-gram features to document positions, allowing the system to perform search operations locally without requiring round-trips to a database server, thus eliminating the latency penalty while maintaining comprehensive search capability.
Solution Approach 2:
The patent introduces an inverted index as an intermediary structure between the document storage and the search query processing. This intermediary enables efficient search by pre-organizing document information in a format that allows fast retrieval based on n-gram features, eliminating the need for direct database round-trips and significantly reducing search latency.
3Ease of operation
If existing string similarity search methods are used, then search functionality is provided, but memory consumption is high and real-time search-as-you-type is challenging
Solution Approach 1:
The patent applies local quality by focusing the index structure on the most important features for similarity search - specifically n-gram features that capture local character patterns within strings. The inverted index is built using only these locally relevant features rather than attempting to index all possible string properties, which significantly reduces memory consumption while maintaining effective search functionality for real-time applications.
Data Source
AI summary
Embodiments perform string similarity analysis by receiving candidate strings of a collection to be searched and transforming each candidate string into one or more features. Embodiments generate a feature index that maps each of the features to one or more candidate strings which include the features, and transform the feature index into a low-memory index by byte encoding a sorted integer array into an encoded byte array. The transforming into a low-memory index further includes hashing each feature to an integer value and storing using a tightly-packed map a hashed value to feature identification (“ID”) mapping, using a first tightly-packed list to map each of the feature IDs to one or more candidate strings which include the features, and using a second tightly-packed list to store the feature IDs.