Targeted Document Set Generation with Lexical and Contextual Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative machine learning models, such as large language models (LLMs), often generate irrelevant or inaccurate content due to the inclusion of data from unrelated sources, leading to 'hallucinations' and reduced relevancy in model-generated content.
Innovation Solution
A method involving a query platform that receives a natural language query and a document identifier, determines a document-type parameter, generates an optimized query, performs a lexical search on a document pool, and a contextual search on a vector database to generate a targeted document set, which is then input to a generative model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a model uses a collection of pre-generated content for content generation, then the model can generate content based on available data, but the model may generate irrelevant content or hallucinations due to inclusion of data from irrelevant sources
Solution Approach 1:
The patent segments the search process into two distinct phases: lexical search (matching exact terms and metadata tags) and contextual search (comparing vector embeddings for semantic similarity). This segmentation allows each search type to focus on specific aspects of relevance, improving overall accuracy while maintaining manageable system complexity through specialized, modular search components
Solution Approach 2:
The patent introduces vector embeddings as an intermediary representation that bridges the gap between lexical matching and semantic understanding. By converting both queries and document chunks into vector embeddings, the system creates a mediating layer that enables contextual similarity measurement without requiring complex natural language processing logic, thus improving reliability while controlling complexity
2Loss of information
If the model includes data from multiple sources to improve content generation, then more information is available, but the relevancy of input data with respect to the query decreases
Solution Approach 1:
The patent changes the parameter of data representation from raw text to vector embeddings, enabling mathematical similarity measurements. This parameter transformation allows the system to quantitatively assess and filter relevancy across multiple data sources, maintaining information completeness while improving measurement precision through vector similarity scoring and threshold-based filtering
Solution Approach 2:
The patent performs preliminary filtering of the document pool by executing lexical searches and contextual searches before presenting data to the generative model. This preliminary action pre-screens multiple data sources to retain only those with high lexical and contextual relevancy, ensuring both information completeness and precision by eliminating irrelevant sources in advance
3Measurement precision
If the system performs both lexical search and contextual search to improve data relevancy, then the quality of input data increases, but the processing time and computational resources increase
Solution Approach 1:
The patent segments the retrieval process into two parallel search streams: lexical search for exact term matching and contextual search for semantic similarity. By segmenting the workload, the system can process different types of queries through the most appropriate search method, improving relevancy while optimizing time usage by avoiding unnecessary computational overhead for each query type
Solution Approach 2:
The patent implements partial action by retrieving a limited number of top-matching document chunks from each search type (lexical and contextual) rather than processing all available data. This partial retrieval approach achieves sufficient relevancy precision by focusing on the most promising matches while significantly reducing processing time and computational resources compared to exhaustive search
Data Source
AI summary
In some aspects, the techniques described herein relate to a method including: receiving, at a query platform, a query and a document identifier; determining, based on the document identifier, a document-type parameter; generating, based on metadata related to the document-type parameter, an optimized query; generating a query vector embedding from the optimized query; executing a lexical search on a document pool, wherein the lexical search is based on the optimized query, and wherein the lexical search returns a first document set; executing a contextual search on a vector database, wherein the contextual search is based on the query vector embedding and returns a second document set; and generating a targeted document set from the first document set and the second document set.