Query Processing via Term Normalization for Search Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search engines fail to effectively retrieve relevant information from a corpus due to variations in search term expressions, requiring multiple queries to capture different variants of a single concept, and lack deep understanding of language and knowledge.
Innovation Solution
A method that normalizes search terms using language models to map various forms of a concept to a single preferred term, allowing a single query to return matches even if the search terms and corpus text do not share common words, by generating and applying language models to both user queries and the corpus during preprocessing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional search engines use exact term matching, then search precision is maintained, but search recall deteriorates due to inability to retrieve variants of the same concept
Solution Approach 1:
The patent transforms search terms from their original form into normalized representations by changing linguistic parameters (synonyms, grammatical forms, variations) while maintaining the core concept. This allows the search system to match different expressions of the same concept, improving recall without sacrificing precision through the use of normalized search terms that represent conceptual equivalence.
Solution Approach 2:
The patent introduces normalized search terms as an intermediary layer between the user's query and the corpus. This mediator translates various concept variants into a standardized form, enabling the search engine to retrieve relevant information regardless of the specific wording used, thus resolving the contradiction between exact matching and variant retrieval.
2Loss of information
If multiple queries are submitted to capture different concept variants, then search recall improves, but productivity deteriorates due to increased query effort
Solution Approach 1:
The patent performs normalization of search terms as a preliminary action before executing the search query. By pre-processing the query to include normalized terms that represent various concept variants, the system enables a single query to capture multiple concept expressions, eliminating the need for users to submit multiple queries and thus improving productivity while maintaining high recall.
3Adaptability or versatility
If language models are applied to normalize search terms, then adaptability improves by understanding concept variants, but device complexity increases due to model integration
Solution Approach 1:
The patent extracts and applies pre-existing language models (such as Word2Vec, GloVe, or BERT) that were trained independently for language understanding tasks. By integrating these external models into the search system, the patent gains advanced language understanding capabilities without having to build complex models from scratch, thus improving adaptability while managing device complexity through leveraging established resources.
Data Source
AI summary
System and computer program product to perform an operation for query processing based on normalized search terms. The operation begins by, responsive to receiving a query, generating a normalized search term for a concept in the query based on a first language model, of a plurality of language models each having a predefined association with a respective concept. The operation then modifies the query to include the normalized search term, and executes the modified query against an indexed corpus of evidence including a first item of evidence. The operation then, upon determining that the first item of evidence includes the normalized search term, returns the first item of evidence as responsive to the query.