Distributed Keyword-Embedding Search for Real-Time Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy Information Retrieval (IR) systems rely solely on keyword-based searching, which is ineffective in capturing semantics and context, especially in large search spaces, and deep learning-based methods struggle with real-time performance and generalization to new words or queries.
Innovation Solution
A hybrid IR system combining keyword similarity search with embedding similarity search, utilizing a hybrid scoring function that integrates keyword match scores and cosine similarity scores, deployed in a distributed database with sharding and replication for real-time performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If keyword-based searching is used, then real-time performance is achieved, but semantic understanding and accuracy deteriorate
Solution Approach 1:
The patent combines keyword-based search (for speed) with embedding-based semantic search (for accuracy) into a hybrid search system. The keyword search component provides real-time performance by quickly filtering documents, while the embedding-based component enhances semantic understanding by comparing vector representations of queries and documents, thus resolving the contradiction between speed and accuracy.
Solution Approach 2:
The search system uses a composite scoring mechanism that integrates both keyword match scores and embedding similarity scores. This composite approach allows the system to leverage the strengths of both methods: keyword matching for fast initial filtering and embedding similarity for accurate semantic ranking, achieving both real-time performance and high accuracy simultaneously.
2Measurement precision
If embedding-based search is used, then semantic understanding improves, but real-time performance deteriorates
Solution Approach 1:
The search process is segmented into two stages: first, keyword-based filtering quickly narrows down the document set to relevant candidates, and second, embedding-based semantic comparison is applied only to this reduced subset. This segmentation allows the computationally intensive embedding operations to be performed on fewer documents, maintaining real-time performance while achieving high semantic accuracy.
3Measurement precision
If deep learning methods are used, then semantic accuracy improves, but generalization to new words and queries deteriorates
Solution Approach 1:
The system employs a universal search mechanism that works across both seen and unseen vocabulary. Keyword matching provides exact and partial matches for new words without requiring prior training, while embedding-based search captures semantic relationships that generalize to new concepts. This multi-functional approach ensures the system adapts to new words and queries effectively.
4Measurement precision
If hybrid scoring function is used, then search accuracy improves, but computational complexity increases
Solution Approach 1:
The hybrid scoring process is segmented into distinct computational stages: keyword extraction and matching, embedding generation, cosine similarity calculation, and final score aggregation. This segmentation allows for optimized computation at each stage and makes the complex process more manageable and efficient, reducing overall computational burden while maintaining high accuracy.
Data Source
AI summary
A computer-implemented method for performing searches in a document database is disclosed. The method comprises automatically detecting a line of business associated with a user, receiving a text query from the user, and generating a query embedding from the text query. The method further comprises scoring entries in a reverse index using a hybrid scoring function. The reverse index comprises titles, title embeddings, sentences, sentence embeddings, and entity tags corresponding to documents in the document database. The hybrid scoring function is used to generate a score based both on a keyword match score between the text query and the reverse index and on a cosine similarity score calculated from embeddings in the text query and in the reverse index. The method also comprises ranking scores for sentences in the document database, and displaying a sentence associated with a top score to the user.