Database Search Using Word Embeddings for Semantic Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional search engine implementations rely heavily on keyword searches and term frequency inverse document frequency weighting, which are inefficient in identifying documents based on semantic similarity, particularly in applications requiring regulatory compliance and risk assessment.
Innovation Solution
A system and method utilizing word embeddings to generate vector representations of input and stored documents, determining similarity metrics through models like cosine similarity, and training these models based on user feedback to improve relevance ranking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional keyword search and term frequency inverse document frequency weighting methods are used, then the search system is simple to implement, but the accuracy of document retrieval based on semantic similarity is poor
Solution Approach 1:
The patent transforms documents from traditional keyword-based representation to vector embeddings in a continuous semantic space. This parameter change enables semantic similarity measurement through geometric operations (cosine similarity, Euclidean distance) rather than keyword matching, directly improving retrieval accuracy while maintaining computational efficiency through established vector operations.
Solution Approach 2:
The patent replaces the mechanical keyword-matching system with a semantic vector space model. Instead of mechanically comparing keywords and computing TF-IDF scores, the system uses vector embeddings where semantic meaning is captured through distributed representations, allowing for more nuanced similarity detection based on vector geometry rather than discrete keyword overlap.
2Measurement precision
If word embeddings and similarity models are implemented, then the semantic similarity measurement is improved, but the computational complexity increases
Solution Approach 1:
The patent pre-computes and stores vector embeddings for all documents in the database before actual search operations. This preliminary action transforms the computationally intensive embedding generation task into a one-time setup process, while subsequent searches only require efficient vector similarity computations, significantly reducing the computational power needed during runtime operations.
Solution Approach 2:
The patent creates vector representations (copies) of documents in a compressed semantic space. Instead of storing and processing full text documents during search, the system works with compact vector embeddings that capture essential semantic information, reducing computational requirements while preserving retrieval accuracy.
3Reliability
If user feedback is used to train the model, then the relevance ranking is improved, but the system requires continuous training and maintenance
Solution Approach 1:
The patent implements a feedback loop where user interactions (clicks, selections, rejections) are collected and used to retrain the similarity model. This continuous feedback mechanism allows the system to adapt to changing user preferences and improve relevance ranking over time, transforming manual training effort into an automated iterative improvement process.
Solution Approach 2:
The system enables itself to improve through automated retraining using collected user feedback. Rather than requiring continuous manual intervention, the system automatically processes user interactions, updates the similarity model, and deploys improved rankings, reducing the extent of manual automation needed while maintaining high reliability.
Data Source
AI summary
Systems and methods of the present disclosure enable database search. The systems and/or methods may include receiving a search query that includes an input document having text. Word embeddings are generated within the input document, where the word embeddings include vector representations of words in the text of the input document. An average input document word embedding vector is determined for the word embeddings of the input document. A set of stored documents is accessed, where each stored document includes a stored text has a particular average stored document word embedding vector. A similarity model is used to determine a similarity metric measuring the similarity between the input document and each stored document based on the average input document word embedding vector and the particular average stored document word embedding vector of each stored document.