Semantic Vector Search Recall Using Pre-trained Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search engine recalling methods are limited by their inability to generalize semantics, affecting accuracy due to reliance on literal matches using indicators like TFIDF and BM25.
Innovation Solution
A method utilizing a pre-trained neural network model to calculate semantic vectors of search terms, establishing an index based on these vectors, and recalling target documents from candidate documents, thereby improving recall accuracy by leveraging semantic characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If literal match-based indicators (TFIDF, BM25) are used for recalling, then the matching process is simple and fast, but the recalling accuracy deteriorates due to inability to generalize semantics
Solution Approach 1:
The patent replaces the traditional mechanical/literal matching system (inverted index with TFIDF/BM25) with a neural network-based semantic vector system. The neural network model processes search terms and documents to generate semantic vectors that capture meaning beyond literal word matches, thereby improving recalling accuracy while maintaining system manageability through automated vector computation
Solution Approach 2:
The patent changes the fundamental parameter of document representation from literal word frequency counts (TFIDF, BM25) to semantic vectors derived from neural network processing. This parameter transformation enables the system to capture semantic similarities and generalize meanings, directly addressing the accuracy limitation of literal match-based approaches
2Adaptability or versatility
If neural network models are used to calculate semantic vectors, then the semantic generalization capability is improved, but the computational complexity and training requirements increase
Solution Approach 1:
The patent applies preliminary action by pre-training the neural network model offline using extensive training data and computational resources before actual search operations. The model is trained to map words and documents to semantic vectors in advance, so that during real-time searching, only lightweight vector similarity computations are needed, thereby managing the complexity burden outside the main search process
Solution Approach 2:
The patent uses the pre-trained neural network model as a reusable computational resource that can be applied to multiple search queries without retraining. The model's semantic vector generation capability is copied and applied consistently across different documents and search terms, enabling semantic generalization while avoiding the repeated computational burden of training
Data Source
AI summary
Embodiments of the present disclosure disclose a method, an apparatus, a server, and a storage medium for recalling for a search. The method for recalling for a search includes: acquiring a search term inputted by a user; calculating a semantic vector of the search term using a pre-trained neural network model; and recalling, according to a pre-established index, target documents related to the semantic vector of the search term from candidate documents, the index being established based on the semantic vectors of the candidate documents, and the semantic vectors of the candidate documents being calculated using the pre-trained neural network model. The embodiments of the present disclosure may solve a problem in the existing method for recalling that the recalling accuracy is affected by failing to generalize semantics, to improve the accuracy of recalling for a search.