Passage-Level Search Ranking With Document Context Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI-based information retrieval models face limitations in processing large documents, leading to suboptimal search results when documents are divided into passages, as seen in systems like BERT, where relevant documents may be misranked due to unrelated content and uneven distribution of query terms across passages.
Innovation Solution
A method integrating document-level and passage-level search results by using multiple search models, including a first model for fast processing and high recall, and a second AI-based model for re-ranking, followed by rank fusion and aggregation techniques to determine final document rankings based on passage relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If documents are divided into multiple passages for AI-based search, then processing efficiency and search precision are improved, but document context is lost and relevant documents may be misranked
Solution Approach 1:
The system divides documents into multiple passages for AI-based processing, enabling efficient and precise search of individual passages while maintaining the ability to reconstruct full document context through passage aggregation and ranking
Solution Approach 2:
The system combines passage-based search results with document-based search results through rank fusion techniques, integrating the precision of AI passage matching with the contextual understanding of full document search to produce comprehensive search results
2Measurement precision
If AI-based passage models are used, then search precision is improved, but processing time and computational resources increase
Solution Approach 1:
The system applies AI-based passage models selectively to the most relevant passages identified through initial filtering, rather than processing all passages equally, thus achieving high precision while reducing overall computational time
Solution Approach 2:
The system performs preliminary passage indexing and embedding generation before search queries are submitted, preparing passage representations in advance to enable faster retrieval and matching during actual search operations
3Loss of information
If entire documents are searched, then document context is preserved, but processing efficiency decreases and AI model limitations are exceeded
Solution Approach 1:
The system segments large documents into smaller passages that fit within AI model processing limits, enabling efficient processing while maintaining the ability to reconstruct and evaluate full document context through passage-level analysis and aggregation
Data Source
AI summary
A method is presented for enhancing search results by segmenting documents into smaller passages and utilizing those passages as the search unit. This method integrates the passage rankings from two search models to produce a new document ranking and arranges the documents accordingly. The ranking of document is also rearranged based on the proportion of passages taken from the same document versus the total number of passages in that document. The final search ranking system combines document-level and passage-level search rankings to rank documents. This method ensures that when conducting a passage search, the search results consider the general relevance of the entire document, which leads to better performance compared to searching only by document-level or passage-level searches.