N-gram Tokenization for Structured Search Query Suggestions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search query suggestion technologies do not effectively utilize the structure of the local search index, leading to a large search space and inefficient results, as they do not take advantage of metadata fields, making it difficult for users to find specific information.
Innovation Solution
A method and apparatus that suggests tokenized query completions by using the last N grams of the query prefix to match specific fields in a structured database, ranking suggestions based on the number of tokens to provide more focused search results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional query completion matching words in the query prefix is used, then the search suggestions can be generated, but the search space remains large and the results are inefficient
Solution Approach 1:
The patent segments the query prefix into N-gram tokens (sequences of N words) and matches them against structured metadata fields in the search index. This segmentation transforms the traditional word-by-word matching into structured field-based matching, thereby reducing the search space and improving search efficiency.
Solution Approach 2:
The patent applies local quality by matching different parts of the query prefix (specific N-grams) to specific metadata fields in the search index. This allows the system to focus the search on relevant portions of the index rather than performing a broad search across all content, thus reducing the search space while maintaining result quality.
2Measurement precision
If keyword-based metadata matching is used, then specific words can match specific metadata, but the format is difficult to use
Solution Approach 1:
The patent introduces an intermediary layer that automatically tokenizes the query prefix into N-grams and maps them to structured metadata fields. This intermediary process handles the complex matching logic behind the scenes, allowing users to input simple natural language queries without needing to understand the underlying metadata structure, thus maintaining precision while improving ease of use.
Solution Approach 2:
The system performs automatic tokenization and field mapping without requiring user intervention. The query completion suggestion system self-services by taking the raw query prefix, tokenizing it into N-grams, matching against the structured index, and generating suggestions automatically, thereby maintaining high precision while requiring minimal user effort.
Data Source
AI summary
A method and apparatus of a device that suggests a tokenized query completion for an input query prefix is described. In an exemplary embodiment, the device receives a query prefix from a client, wherein the query prefix includes a plurality of words. The device further generates a results set by searching a structured database using the query prefix for matches to the plurality of words in the query prefix. The device additionally determines a subset of query prefix that match specific fields of the results set by using the last N grams in the query prefix. In addition, the device ranks a tokenized query completion as a search suggestion using the query prefix, where the tokenized query completion includes a token that is a match between a matching word in the subset of query prefix and the corresponding specific field for the matching word.