LLM Output Explanation Through Input Perturbation Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) produce varying outputs for the same input, lack transparency, and require internal model access for explanation, making it difficult for users to evaluate accuracy and consistency, and compliance with regulatory requirements.
Innovation Solution
A method to perturb input portions, generate perturbed outputs, calculate scalar differences, aggregate importance scores, and train an importance scoring model to explain LLM outputs without requiring internal model access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If LLM internal model access is required for explanation, then explanation accuracy is improved, but system complexity and ease of operation deteriorate
Solution Approach 1:
The patent introduces an intermediary explanation generation system that uses perturbation analysis and importance scoring to bridge the gap between LLM outputs and human-understandable explanations. This mediator analyzes input-output relationships without requiring direct access to internal model weights or architecture, thereby maintaining explanation accuracy while reducing system complexity.
Solution Approach 2:
The patent replaces the mechanical approach of directly accessing internal model structures with a computational methodology based on perturbation analysis and scalar importance scoring. This substitution enables explanation generation through observable input-output behaviors rather than direct inspection of internal mechanisms, simplifying the system while preserving explanatory power.
2Reliability
If LLM internal model access is required for explanation, then explanation faithfulness is improved, but ease of operation deteriorates
Solution Approach 1:
The patent enables the LLM to explain its own outputs through self-reflection mechanisms. By analyzing the relationship between inputs and its own outputs using perturbation methods, the system generates faithful explanations without requiring external interpreters or access to internal model states, thereby maintaining reliability while improving ease of operation.
3Loss of information
If perturbation analysis is performed on multiple portions, then explanation completeness is improved, but computational cost increases
Solution Approach 1:
The patent applies partial action by performing perturbation analysis selectively on portions of the input that are most likely to influence the output, rather than exhaustively analyzing all possible portions. This approach maintains explanation completeness by focusing computational resources on critical areas while reducing overall computational cost through targeted rather than comprehensive analysis.
Data Source
AI summary
An embodiment causes a target large language model (LLM) to generate, from a first input to the target LLM, a first output. The embodiment perturbs a portion of the first input. The embodiment causes the target LLM to generate a first perturbed output from the perturbed input. The embodiment scalarizes the first perturbed output. The embodiment aggregates, into an importance score corresponding to the portion, the scalar and a set of additional scalars representing differences between the first output and an additional perturbed output generated by the target LLM from an additional perturbation of the portion. The embodiment explains, responsive to determining that the importance score is the highest importance score in a set of importance scores, the first output using the portion. The embodiment trains, using the portion and the importance score, an importance scoring model.