Predictive Stemming for Web Search Using Statistical Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Search engines face challenges in efficiently transforming search query terms into relevant variants to improve search results, leading to increased computational overhead and reduced precision due to the generation of numerous less relevant files, which slows down search results and diminishes user experience.
Innovation Solution
A method is introduced to generate a dictionary of frequently used search queries offline, using word stems and transformations to create candidate queries, and applying algorithms like N-best and language modeling to select the most probable queries for execution, thereby minimizing overhead and enhancing relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If search engines transform every query term into all possible word variants, then more relevant search results are found, but computational overhead increases significantly and search speed decreases
Solution Approach 1:
The patent pre-computes and stores word variants, their stems, and relevance relationships in dictionaries and indexes before actual search queries are processed. This preliminary preparation allows the search engine to quickly retrieve pre-analyzed variant information without performing computationally expensive transformations during real-time search operations, thus maintaining high relevance while improving processing speed
Solution Approach 2:
Instead of transforming every query term into all possible variants, the patent selectively applies transformation only to terms that are likely to have relevant variants based on statistical analysis and language models. This partial action approach generates fewer candidate queries than exhaustive transformation, reducing computational overhead while still finding most relevant results
2Adaptability or versatility
If search engines generate all possible query variants, then comprehensive search coverage is achieved, but the number of less relevant files increases and precision decreases
Solution Approach 1:
The patent uses language models and statistical analysis to evaluate and rank candidate query variants based on their likelihood of returning relevant results. This feedback mechanism filters out low-probability variants that would generate irrelevant files, maintaining comprehensive coverage of promising variants while improving precision by eliminating unpromising ones
Solution Approach 2:
The patent changes the parameter of query transformation from deterministic (transforming all terms) to probabilistic (transforming terms based on calculated probability scores). By adjusting transformation parameters based on language model confidence scores, the system achieves comprehensive coverage of high-probability variants while filtering out low-probability ones that would reduce precision
3Reliability
If search engines perform exhaustive word transformation, then all possible relevant results are retrieved, but computational resources and time are excessively consumed
Solution Approach 1:
The patent pre-computes word variants, stems, and their relationships offline and stores them in optimized data structures. This preliminary action eliminates the need for expensive real-time computation during search, allowing the system to retrieve pre-analyzed variant information quickly while maintaining completeness of relevant results
Solution Approach 2:
The patent performs partial transformation by selecting only the top N most probable query variants for execution, rather than exhaustively processing all possible variants. This partial action approach retrieves most relevant results (completeness) while limiting the number of candidate queries to a manageable set, thereby reducing processing time
Data Source
AI summary
Techniques for determining when and how to transform words in a query to return the most relevant search results while minimizing computational overhead are provided. A dictionary is generated based upon words used in a specified number of previous most frequent search queries and comprises lists of transformations that may include variants based upon the stems of words, synonyms, and abbreviation expansions. When a query is received from a user, candidate queries are generated based upon replacing particular words in the query with a transformation of the particular words. Candidate queries are selected that have a high probability of returning relevant results by computing values of the query using language model scoring and translation scoring. The selected candidate queries and the original query are executed to return search results. The search results are displayed to the user with the words in the original query and the transformed words in bold.