Multi-Modal LLM Enhancement with Vector Retrieval Without Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) suffer from hallucinations, particularly in domains like medical decision-making, where unreliable answers can cause harm, and current re-architecting and re-training methods are resource-intensive and provide limited success.
Innovation Solution
A system that improves LLM alignment by integrating multi-modal information without altering the model's architecture or extensive re-training, using Retrieval Augmented Generation (RAG) with vector search and augmented proximity metrics to ground generated text in trusted data, incorporating context vectors to enhance relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If re-architecting and re-training the LLM model is performed to improve alignment, then the model's reliability is improved, but the resource consumption increases and the process becomes slow
Solution Approach 1:
The patent introduces an intermediary retrieval system that acts as a mediator between the user query and the LLM. This retrieval system fetches relevant information from external knowledge bases before the LLM generates responses, allowing the model to access accurate information without requiring architectural changes or extensive retraining. The intermediary layer enables the LLM to produce more reliable outputs while avoiding the high resource costs of model re-architecting.
2Reliability
If re-architecting and re-training the LLM model is performed to improve alignment, then the model's reliability is improved, but the development time increases
Solution Approach 1:
The patent implements preliminary action by pre-processing and organizing knowledge bases into structured formats before the LLM needs to generate responses. The retrieval system is pre-configured with relevant information and indexing structures, allowing it to quickly fetch accurate data during inference. This eliminates the need for time-consuming model retraining while still improving alignment, as the relevant information is already prepared and accessible when needed.
3Reliability
If the LLM is restricted to a specific domain of knowledge through RAG, then the reliability is improved, but the versatility decreases
Solution Approach 1:
The patent implements a universal retrieval system that can adapt to multiple domains through a single unified architecture. The system uses domain-agnostic techniques such as semantic search and vector embeddings that work across different knowledge domains. By making the retrieval mechanism itself universal rather than domain-specific, the system maintains versatility while improving reliability in each specific domain through appropriate knowledge base selection and query routing.
Data Source
AI summary
A system and method for enhancing query responses from large language models without retraining by converting a query into a query vector; using a proximity metric to measure a proximity from the query vector to a plurality of vector embeddings stored in a vector database; ranking the plurality of vector embeddings based on proximity to the query vector; mapping the query to a homogenized context vector from a plurality of homogenized context vectors; using an augmented proximity metric to convert the proximity to an augmented proximity for each vector embeddings; performing an augmented ranking to refine the vector embeddings to those most relevant to the query; creating a prompt for a large language model comprising the query and the text data corresponding to refined vector embeddings as context; and feeding the prompt to the large language model to generate a response to the query.