Document Passage Selection for Relevance and Diversity Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search systems fail to extract passages from documents that are both relevant and diverse, unable to select multiple passages from a document that are quantifiably diverse from one another but similarly relevant to a search query.
Innovation Solution
A document analysis system that selects multiple, diverse passages of a document based on a relevance model, evaluating features and weights to determine passages that are both relevant to the search query and diverse from one another.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional search engines select search results based on query terms, then relevancy to the search query is improved, but diversity of the search results deteriorates
Solution Approach 1:
The patent segments the search result selection process into multiple independent components: a relevancy model that evaluates query-term matching, a diversity model that measures passage similarity using cosine similarity, and a combined scoring mechanism. This segmentation allows each component to optimize for its specific function while the integration mechanism balances both relevancy and diversity requirements.
Solution Approach 2:
The patent changes the evaluation parameters from单一的 relevancy scoring to a multi-parameter system that includes both relevancy scores and diversity metrics. The diversity is quantified through cosine similarity calculations between passage vectors, and the final selection uses a composite scoring mechanism that adjusts parameters to achieve both high relevancy and high diversity in the selected passages.
2Adaptability or versatility
If multiple passages are selected from a document, then coverage of topics expressed in the search query is improved, but redundancy of information deteriorates
Solution Approach 1:
The patent implements a feedback mechanism where the diversity model continuously evaluates the similarity between selected passages and previously selected passages. The cosine similarity calculation provides feedback on redundancy, and this feedback is used to adjust the selection process, ensuring that newly selected passages add new topic coverage rather than repeating information already covered by previously selected passages.
3Measurement precision
If passages are selected based on relevance model features and weights, then accuracy of passage selection is improved, but complexity of the selection process deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing passage vectors that capture the semantic meaning of each passage. These pre-computed vectors are stored and can be quickly retrieved and compared during the selection process. This preliminary processing reduces the complexity of real-time similarity calculations, as the heavy computational work of vector representation is done in advance rather than during passage selection.
Data Source
AI summary
Systems and methods for selecting multiple, diverse passages of a document based on such a relevancy model are disclosed.