Search Ranking Model Using Vector Embeddings for Semantic and Lexical Relevance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current transformer-based machine-learning models for search results ranking struggle to effectively consider both semantic and lexical relations between search queries and digital documents, leading to missed relevant results and poor user experience.
Innovation Solution
The proposed solution involves receiving a search query and identifying semantically and lexically related digital documents. By feeding the search query and these related documents into a transformer-based ML model, along with vector embeddings generated from specific phrases within the documents, the system can rank documents based on both semantic and lexical relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If transformer-based ML models use attention mechanism to assign greater significance to some portions of input, then semantic understanding is improved, but lexical relations between search query and digital documents are disregarded
Solution Approach 1:
The patent segments the input processing into two distinct pathways: one for semantic understanding via transformer attention mechanism, and another for lexical matching via exact word/phrase matching. This segmentation allows each pathway to specialize in its strength without compromising the other, resolving the contradiction between semantic precision and lexical reliability
Solution Approach 2:
The patent merges the outputs of the semantic analysis pathway and lexical matching pathway into a unified ranking system. By combining these two complementary approaches, the system achieves both semantic understanding and lexical relation detection, eliminating the trade-off between them
2Adaptability or versatility
If transformer-based ML models are trained using large datasets, then model generalization is improved, but ability to catch lexical relations is reduced
Solution Approach 1:
The patent segments the relevance determination into two independent components: semantic relevance (handled by the trained transformer model) and lexical relevance (handled by exact matching). This segmentation allows the model to be trained on large datasets for semantic understanding without sacrificing lexical relation detection capabilities
Solution Approach 2:
The patent introduces an intermediary lexical matching layer that operates independently of the trained model's semantic analysis. This intermediary component ensures lexical relations are captured through direct word/phrase matching, compensating for the trained model's tendency to overlook exact lexical matches
Data Source
AI summary
A method and a server for ranking digital documents at a digital platform are provided. The method comprises: receiving a search query submitted to the digital platform; generating a first vector embedding representative of the search query; identifying a plurality of digital document candidates responsive to the search query; retrieving, for each one of the plurality of digital document candidates, a second vector embedding representative; identifying, in a given one of the plurality of digital document candidates, at least one phrase candidate that is lexically related to the search query; generating a third vector embedding representative of the at least one phrase candidate; based on the first, second, and third vector embeddings, determining, for the given one of the plurality of digital document candidates, a respective value of a ranking parameter; and ranking the plurality of digital document candidates according to respective values of the ranking parameter.