Adaptive Data Scoring for Search Result and LLM Relevance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) lack the ability to assess and leverage the quality and relevance of data sources, leading to suboptimal search result retrieval and response generation, as they do not consider heuristic metadata associated with data sources such as frequency of retrieval, update frequency, and relevance.
Innovation Solution
A multi-metric analytic function is applied to score data sources based on heuristic metadata, generating composite scores to select a subset of search results and corresponding data sources, which are then used to generate a prompt for the LLM, enhancing the relevance and quality of responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLMs generate responses based solely on search result content without assessing data sources, then the generation process is simple and fast, but the quality and relevance of responses deteriorate
Solution Approach 1:
A data source scoring module is introduced as an intermediary component between the search engine and LLM. This module assesses data sources using multiple metrics (relevance, freshness, authority, coverage) and generates scores that are fed to the LLM as metadata, enabling the LLM to make informed decisions about which data sources to rely on without directly analyzing the data source structures themselves.
Solution Approach 2:
The system performs preliminary assessment of data sources before the LLM generates responses. By pre-scoring data sources based on their metadata and characteristics, the system prepares quality information in advance that the LLM can utilize during response generation, separating the assessment task from the generation task.
2Productivity
If heuristic metadata of data sources is not leveraged, then the system is simpler to operate, but the speed of retrieval, relevancy of search results, and response generation are suboptimal
Solution Approach 1:
The system automatically extracts and leverages heuristic metadata from data sources without requiring manual intervention or complex configuration. The data source scoring module self-adjusts by analyzing metadata fields (relevance, freshness, authority, coverage) and automatically generates appropriate scores, making the system easy to operate while maintaining high productivity.
Solution Approach 2:
The system transforms raw metadata parameters into normalized score parameters through a scoring function. By converting multiple metadata fields into a unified score representation that the LLM can process, the system optimizes retrieval speed and relevancy while keeping the operation simple through automated parameter transformation.
3Measurement precision
If all search results are provided to LLM without filtering, then the LLM has more information to work with, but the response generation becomes less accurate and contextually appropriate
Solution Approach 1:
Instead of treating all search results uniformly, the system applies different quality weights based on local characteristics of each data source. By scoring each data source individually based on its specific metadata (relevance, freshness, authority, coverage) and incorporating these scores into the LLM prompt, the system enables the LLM to prioritize high-quality results while maintaining accuracy.
Solution Approach 2:
The system performs preliminary filtering and scoring of search results before they are provided to the LLM. By pre-ranking and selecting the most relevant data sources based on their metadata scores, the system reduces the quantity of results the LLM needs to process while maintaining or improving accuracy through focused attention on high-quality information.
Data Source
AI summary
A method includes obtaining, as a response to a user query, a multitude of search results and corresponding similarity scores. The method further includes obtaining a multitude of data sources corresponding to the multitude of search results. The method further includes processing a multi-metric analytic function for the search results, and corresponding data sources to obtain a composite score for the search results. The method further includes selecting a subset of search results from the multitude of search results based on a composite score threshold to obtain a selected subset of search results. The method further includes selecting a corresponding subset of data sources from the multitude of data sources to obtain a selected subset of data sources. The method further includes generating a prompt to a large language model (LLM), including at least the user query, the selected subset of search results, and the selected subset of data sources.