Federated Search Semantic Corpus Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing federated search systems require users to manually select relevant corpora, leading to potential omission of important documents or inclusion of irrelevant ones, and lack the ability to perform searches over subsets of documents within a corpus.
Innovation Solution
A system that performs semantic analysis of search queries to identify relevant concepts, selects appropriate corpora or subsets, and submits formatted queries to multiple search endpoints, combining results with global relevancy ranking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If users manually select corpora for federated search, then they can control which databases are searched, but they may omit important corpora or include irrelevant ones, reducing search accuracy
Solution Approach 1:
The system performs automatic corpus selection based on semantic analysis of the search query, allowing the system to serve itself rather than requiring user intervention. The semantic parser automatically identifies relevant corpora by analyzing query semantics and matching them with appropriate corpus metadata, eliminating the need for users to manually select corpora while maintaining high search accuracy
Solution Approach 2:
The system changes the parameter of corpus selection from user-defined to system-defined based on semantic relevance. By introducing semantic analysis as a new parameter for corpus selection, the system dynamically determines which corpora to search based on the semantic content of the query rather than static user preferences or manual configuration
2Quantity of substance
If users include multiple corpora in federated search, then more documents may be found, but irrelevant results increase and search time increases
Solution Approach 1:
The system performs preliminary semantic analysis of the search query before executing the federated search. By pre-identifying the most relevant corpora through semantic parsing and concept matching, the system prepares an optimized subset of corpora to search, avoiding the time waste of searching irrelevant corpora while ensuring all potentially relevant documents are included
Solution Approach 2:
The system applies different selection criteria to different corpora based on their specific characteristics and relevance to the query. Rather than uniformly searching all corpora or using a single selection rule, the system evaluates each corpus's semantic relevance and selects only those with sufficient local quality match to the query, optimizing the balance between document coverage and search efficiency
3Reliability
If federated search searches over entire corpora, then completeness is improved, but ability to search specific subsets is lost
Solution Approach 1:
The system segments the corpus selection process into two levels: corpus-level segmentation where entire corpora are selected based on semantic relevance, and document-level segmentation where specific subsets within corpora are identified through semantic concept matching. This allows the system to search complete relevant corpora while also enabling searches over specific subsets by filtering results based on semantic concepts, thereby maintaining both completeness and flexibility
Data Source
AI summary
A method comprises receiving at a computing device, a search query, performing, by the computing device, a semantic analysis of the search query to identify one or more semantic concepts contained within the query, selecting, by the computing device, one or more corpora, or portions thereof, based on the identified semantic concepts, and performing, by the computing device, a search of the one or more corpora based on the search query.