LLM Data Curation With Embedded Article Retrieval for Accurate Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI and ML technologies face challenges in accurately understanding user queries, retrieving relevant documents, and providing contextually relevant search results during chat sessions, often leading to inefficient resource utilization and less accurate outputs.
Innovation Solution
A computer-implemented method that utilizes a large language model (LLM) to generate outputs by embedding articles from knowledge and curated databases, selecting relevant articles based on similarity scores, and iteratively prompting the LLM with critiques to enhance accuracy and relevance, while optimizing processing and memory resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If articles from knowledge and curated databases are embedded and selected based on similarity scores, then output accuracy and relevance are improved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary embedding of articles from knowledge and curated databases before actual queries are processed. This pre-computation of embeddings allows for rapid similarity scoring and article selection during chat sessions, improving output accuracy without adding processing delays during user interactions.
Solution Approach 2:
The system extracts only the most relevant articles from the embedded database by comparing query vectors against stored embeddings and selecting those exceeding a threshold similarity score. This extraction approach focuses computational resources on retrieving only pertinent information rather than processing entire databases, thereby maintaining accuracy while reducing processing time.
2Loss of information
If multiple databases (knowledge and curated) are integrated and queried, then information completeness and relevance are improved, but memory resource requirements increase
Solution Approach 1:
The system merges articles from both knowledge and curated databases into a unified embedding space, allowing seamless integration and comparison of information from multiple sources. This unified approach ensures information completeness by drawing from both databases while managing memory resources through efficient vector storage and selective retrieval based on query relevance.
3Productivity
If LLM is applied to both external and internal queries in parallel, then productivity and response time are improved, but computational power consumption increases
Solution Approach 1:
The system dynamically routes queries to appropriate LLM processing paths based on their nature - external queries requiring broad knowledge access versus internal queries leveraging curated information. This dynamic approach allows parallel processing of multiple query types to improve productivity while optimizing computational power consumption by matching query characteristics with appropriate processing resources.
Data Source
AI summary
Systems and methods for improved data curation and integration are disclosed. An example system includes one or more processors; and a memory storing instructions that cause the system to: receive an initial query, query an external database to obtain a set of keywords, topics, and other queries pertinent to the initial query, and embed articles into a database. The system may also append the keywords, topics, and other queries to the initial query to obtain an external data query and embedded articles to the initial query to obtain an internal data query; and generate (i) an external output by applying a large language model (LLM) to the external data query and (ii) an internal output by applying the LLM to the internal data query. The system may also combine the internal output with the external output to obtain a combined output and cause the combined output to be displayed.