Statistical Machine Translation for Query Rewriting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search engines face challenges in addressing the query-document vocabulary mismatch, where user search queries do not yield desired results due to inherent biases in human-created and human-annotated lists used for query reformulation.
Innovation Solution
The system analyzes user click log data to identify related strings, which are then input into a statistical machine translation model to calculate the probability of relatedness between terms, characterizing replacement terms that exceed a threshold for incorporation into a query rewriting candidate database, thereby reducing human bias and improving search results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If human-created and human-annotated lists are used for query reformulation, then query rewriting can be performed, but inherent biases in these lists cause query-document vocabulary mismatch and reduce search accuracy
Solution Approach 1:
The patent replaces the manual, human-created query rewriting system with an automated statistical machine translation system. Instead of relying on human experts to create replacement term lists, the system uses SMT models trained on bilingual corpora to automatically generate and score query document pairs, eliminating human bias while maintaining query rewriting functionality
Solution Approach 2:
The patent introduces statistical machine translation as an intermediary between the user's query and the document retrieval process. The SMT system acts as a mediator that translates queries into multiple candidate documents with probability scores, rather than directly returning results or relying on pre-defined human lists
2Productivity
If conventional query reformulation methods are used, then search queries can be processed, but the query-document vocabulary mismatch prevents identification of desired results
Solution Approach 1:
The patent makes the query rewriting process dynamic by using probability scores from the statistical machine translation model to rank and select candidate documents. Instead of static, pre-defined replacement lists, the system dynamically generates and scores multiple query document pairs based on the specific input query, allowing flexible adaptation to different search contexts
Solution Approach 2:
The patent changes the parameter of query representation by translating queries through statistical machine translation which generates probability scores for different document matches. This transforms the rigid binary match/no-match approach into a continuous probability-based ranking system, improving both relevance and efficiency
Data Source
AI summary
Systems, methods, and computer media for identifying query rewriting replacement terms are provided. A list of related string pairs each comprising a first string and second string is received. The first string of each related string pair is a user search query extracted from user click log data. For one or more of the related string pairs, the string pair is provided as inputs to a statistical machine translation model. The model identifies one or more pairs of corresponding terms, each pair of corresponding terms including a first term from the first string and a second term from the second string. The model also calculates a probability of relatedness for each of the one or more pairs of corresponding terms. Term pairs whose calculated probability of relatedness exceeds a threshold are characterized as query term replacements and incorporated, along with the probability of relatedness, into a query rewriting candidate database.