Query Term Weighting via Surprising Word Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional Question Answering (QA) systems often fail to return relevant passages with correct answers due to inadequate query formulation, where common terms are dropped or noise is introduced through excessive expansion, leading to less useful search results.
Innovation Solution
A method that calculates weights for search terms using a statistical language model, based on log likelihood statistics from n-gram models, to selectively boost and include terms, ensuring key terms are not dropped and noise is minimized, thereby improving query formulation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If query expansion is performed by including synonyms and related terms, then the comprehensiveness of search results is improved, but noise is introduced and relevant passages are ranked lower
Solution Approach 1:
The patent applies local quality by selectively expanding only certain query terms based on their information content and contribution to the question, rather than uniformly expanding all terms. The system identifies which terms should be expanded and which should remain unchanged, applying different treatment to different parts of the query to avoid noise while maintaining comprehensiveness.
Solution Approach 2:
The patent changes parameters by adjusting the expansion ratio and selection criteria for query terms based on their statistical properties and importance. The system dynamically modifies which terms are expanded and to what extent, using parameters such as term frequency, information content, and question structure to control the expansion process and minimize noise introduction.
2Device complexity
If common stop words are dropped from the query, then the query is simplified, but key terms are accidentally removed and search accuracy decreases
Solution Approach 1:
The patent applies local quality by treating different words in the query differently based on their specific characteristics. Instead of applying a uniform stop word removal rule, the system evaluates each term individually to determine whether it should be retained or removed, considering its role in the question and its information content. This selective approach preserves key terms while maintaining query simplicity.
Solution Approach 2:
The patent introduces dynamics by making the stop word filtering process adaptive rather than static. The system dynamically determines which words to retain based on the specific question context, term importance, and statistical analysis, allowing the query formulation to adapt to different questions and preserve necessary terms that would traditionally be filtered out.
3Device complexity
If traditional weighting methods are used to boost certain terms, then the query formulation is simplified, but relevant passages are not ranked highly enough
Solution Approach 1:
The patent changes parameters by using sophisticated weighting schemes that go beyond traditional methods. The system adjusts term weights based on multiple factors including information content, term frequency, position in the question, and statistical language model probabilities. This parameter-rich approach improves ranking accuracy while maintaining systematic query formulation.
Solution Approach 2:
The patent introduces feedback mechanisms by using statistical language models and probability distributions to evaluate and adjust term weights. The system incorporates feedback from the language model about term likelihood and information content to refine the weighting of query terms, creating a more accurate ranking system that leverages statistical evidence from the corpus.
Data Source
AI summary
An approach is provided that receives a question at a question answering (QA) system. The question includes a number of words. The approach operates by calculating weights that correspond to search terms included in the plurality of words. The search terms include the plurality of words and may include terms that are one or more sequences of adjacent words included in the question. Based on the calculated weights and the words in the question, the approach generates a query that is used to search a corpus that is managed by the QA system with the search resulting in one or more search results.