Black-Box API Output Confidence via Paraphrase Robustness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Black-box Application Programming Interfaces (APIs) lack the ability to estimate output confidence, making it difficult to assess the reliability of their responses, especially when dealing with natural language inputs.
Innovation Solution
A computer-implemented method that generates paraphrases for an input text, calculates the distance between the input text and each paraphrase, sorts and selects the paraphrases based on distance, and then inputs both the original text and the selected paraphrases into the API to estimate output confidence by evaluating the robustness of the output scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If black-box API is used to process natural language inputs, then functionality and accessibility are improved, but the ability to estimate output confidence and assess reliability deteriorates
Solution Approach 1:
The patent introduces paraphrases as intermediary elements between the input text and the black-box API. By generating multiple paraphrased versions of the input and feeding them to the API, the system creates a mediator layer that enables confidence estimation without modifying the black-box API itself. The variability in API responses to different paraphrases of the same input serves as a proxy for estimating output confidence.
2Reliability
If multiple paraphrases are generated and processed through the API, then output confidence estimation is improved, but computational complexity and processing time increase
Solution Approach 1:
The patent applies partial action by generating a limited set of paraphrases (e.g., top N paraphrases based on distance sorting) rather than exhaustively processing all possible paraphrases. This selective approach allows the system to obtain sufficient confidence estimation data without the full computational burden of processing every conceivable paraphrased version, thus balancing reliability improvement with computational feasibility.
Solution Approach 2:
The system performs preliminary actions by pre-generating and sorting paraphrases based on their distance from the original input before submitting them to the API. This preliminary organization allows the selection of the most relevant paraphrases (those closest to the original input) to be processed first, optimizing the computational resources used for confidence estimation while maintaining accuracy.
Data Source
AI summary
A computer-implemented method is provided for estimating output confidence of a black box Application Programming Interface (API). The method includes generating paraphrases for an input text. The method further includes calculating a distance between the input text and each respective one of the paraphrases. The method also includes sorting the paraphrases in ascending order of the distance. The method additionally includes selecting a top predetermined number of the paraphrases. The method further includes inputting the input text and the selected paraphrases into the API to obtain an output confidence score for each of the input text and the selected paraphrases. The method also includes estimating, by a hardware processor, the output confidence of the input text from a robustness of output scores of the input text and the selected paraphrases.