Language Model Confidence Scoring via Word String Combination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional natural language processing using machine learning language models struggles with irregular situations, leading to incorrect outputs due to high confidence calculations, which are not effectively suppressed, resulting in suboptimal performance.
Innovation Solution
The approach involves acquiring and combining multiple word strings with a target sentence to input into a language model, calculating confidence based on the difference in output probability distributions, and suppressing outputs if confidence does not meet a predetermined threshold to optimize output quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If confidence calculation is used to suppress incorrect output, then output reliability is improved, but incorrect outputs are still produced when confidence is high
Solution Approach 1:
The patent segments the confidence assessment into multiple independent components: (1) base confidence from the language model output, (2) context dependency score from multiple word string combinations, and (3) final composite confidence. This segmentation allows each component to evaluate different aspects of output reliability independently, preventing any single flawed metric from determining the final result.
Solution Approach 2:
The patent introduces context dependency as an intermediary metric that mediates between the language model output and the final confidence determination. By calculating how much the output changes when different word strings are combined with the target sentence, the system creates an intermediate assessment layer that filters out high-confidence incorrect outputs before they are finalized.
2Manufacturing precision
If multiple word strings are combined with target sentence, then output accuracy is improved, but calculation complexity increases
Solution Approach 1:
The patent applies partial action by selecting a limited number of representative word strings to combine with the target sentence, rather than exhaustively testing all possible combinations. This selective approach achieves sufficient accuracy improvement while keeping the calculation complexity manageable through focused sampling of critical context variations.
Data Source
AI summary
A non-transitory computer readable recording medium stores therein a program that causes a computer to execute a process including: acquiring a plurality of word strings relating to a target sentence; inputting each of a plurality of combined sentences for which each of the acquired word strings is combined with the target sentence, and the target sentence into a language model, generated by using a machine learning; calculating, based on a difference between each distribution of an output result when each of the combined sentences is input into the language model, confidence in output when the target sentence is input into the language model; and outputting, based on the calculated confidence, an output result when the target sentence is input into the language model.