LLM Domain Relevance Scoring Using Chunk Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for evaluating the relevance of Large Language Model (LLM) responses to user queries are time-consuming, biased, inconsistent, and lack domain-specific relevance measures, making them non-scalable and requiring domain experts.
Innovation Solution
A method and system that calculates domain relevance scores by splitting LLM responses into chunks, generating vector embeddings using sentence transformers, computing cosine distances with domain-specific training data embeddings, and calculating a score based on these distances and chunk counts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual evaluation of LLM-generated responses is performed, then relevance assessment can be conducted, but the process is time-consuming and non-scalable
Solution Approach 1:
The patent replaces manual mechanical evaluation with an automated computational system. The system converts LLM responses and domain-specific training data into vector embeddings using sentence transformers, then calculates cosine distances to automatically assess domain relevance, eliminating the need for manual human evaluation while maintaining assessment accuracy
Solution Approach 2:
The system enables self-service evaluation by automatically computing domain relevance scores without requiring human experts. The automated pipeline processes responses through embedding generation and cosine distance calculation, allowing the system to evaluate itself against domain standards without external human intervention
2Measurement precision
If manual evaluation is used, then domain expertise can be leveraged, but bias and inconsistency occur
Solution Approach 1:
The patent changes the evaluation parameter from subjective human judgment to objective computational metrics. By transforming text into vector embeddings and using cosine distance calculations, the system converts qualitative relevance assessment into quantitative, repeatable measurements that are consistent across different evaluators and time points
Solution Approach 2:
The system substitutes human judgment with automated computational algorithms that apply consistent mathematical operations (embedding generation, cosine distance calculation) to all responses, eliminating the variability and bias inherent in manual evaluation while maintaining domain relevance assessment
3Measurement precision
If existing relevance metrics are used, then factual relevance can be measured, but domain-specific relevance cannot be captured
Solution Approach 1:
The patent applies local quality by tailoring the evaluation to specific domains. The system uses domain-specific training data to generate embeddings that capture domain-specific concepts and terminology, allowing the relevance metrics to be adapted to the specific requirements of different domains (e.g., healthcare, legal, finance) rather than using a one-size-fits-all approach
Solution Approach 2:
The system segments the evaluation process into distinct components: response chunking, embedding generation, and cosine distance calculation. This segmentation allows the domain-specific training data to be integrated at the embedding generation stage, enabling the system to capture domain-specific relevance while maintaining the ability to evaluate factual accuracy through the same unified framework
Data Source
AI summary
A method for calculating domain relevance scores for responses generated by LLMs is disclosed. The method includes receiving a response generated by LLM corresponding to user query. The user query is associated with a domain. The method further includes splitting the response into a plurality of response chunks using a splitting technique. The method further includes generating a plurality of response vector embeddings based on the plurality of response chunks using at least one sentence transformer. The method further includes computing a plurality of cosine distances between the plurality of response vector embeddings and a corresponding plurality of training data vector embeddings, wherein the plurality of training data vector embeddings corresponds to domain-specific training data of the LLM. The method further includes calculating a domain relevance score corresponding to the response, based on a sum of the plurality of cosine distances and a number of the plurality of chunks.