LLM Confidence Scoring Using Readability Features

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large language models (LLMs) often generate inaccurate or hallucinated outputs, affecting their reliability and trustworthiness, making it difficult to determine the accuracy of their responses.

Innovation Solution

A method and system that generate a confidence score for LLM outputs by parsing input and output text to create a feature vector encoding readability metrics, using a classifier model to predict the accuracy of the output and provide insights into potential hallucinations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If LLMs generate text output based on input prompts, then the fluency and coherence of the text is improved, but the reliability and accuracy of the output deteriorates due to hallucinations

Engineering Contradiction:

Improvefluency and coherence of textVSAvoidaccuracy of output

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent introduces an intermediary system that includes a readability model and classifier model. These models act as mediators between the LLM output and the user, analyzing readability features and generating confidence scores to indicate the likelihood of hallucinations, thus providing a way to assess reliability without affecting the LLM's text generation capability

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements feedback by generating confidence scores that indicate the reliability of LLM outputs. This feedback mechanism allows users to understand the trustworthiness of generated text and helps identify areas where the LLM may be hallucinating, enabling continuous improvement and informed decision-making

Inventive Principle:

Principle #23Feedback

2Reliability

If confidence scores are generated to assess LLM output accuracy, then the reliability measurement is improved, but the system complexity increases due to additional models and processing

Engineering Contradiction:

Improvemeasurement of output accuracyVSAvoidsystem structure

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system is segmented into distinct functional components: a readability model that extracts readability features from text, and a classifier model that generates confidence scores based on these features. This segmentation allows each component to specialize in a specific task, making the overall system more manageable and interpretable despite its complexity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The readability model and classifier model serve as intermediary layers between the LLM and the user. These intermediaries process the LLM output through readability analysis and confidence score generation, adding reliability measurement capability without requiring direct modification of the LLM architecture itself

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12566928B2Readability based confidence score for large language models

Publication Date: 2026.03.03 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

The present disclosure relates to methods and systems that generate a confidence score for the generated large language model (LLM) output. The methods and systems use the text of the input provided to the LLM and the text from the generated LLM output to produce a feature vector that encodes a readability of the text from the input and the text of the LLM output. The feature vector is used to determine a corresponding confidence score for the generated LLM output. The confidence score is used to evaluate a quality of the generated LLM output.