Large Language Model Certainty Scoring for High-Certainty Actions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence engines face challenges in accurately and efficiently generating actions based on user inputs due to issues with certainty determination methods, such as white-box methods requiring inaccessible internal values and black-box methods being computationally costly.
Innovation Solution
A computing system employing a large language engine and an estimation engine to generate multiple actions with associated certainty values, using a trained estimation model to filter actions above a threshold certainty value for presentation, reducing computational cost and improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If white-box methods are used for certainty determination, then measurement precision is improved, but device complexity increases due to requiring inaccessible internal values
Solution Approach 1:
The patent introduces an intermediary component that bridges the gap between the large language model and the certainty determination process. This intermediary translates internal model states into accessible certainty values without requiring direct access to the model's internal workings, thus maintaining measurement precision while reducing system complexity.
Solution Approach 2:
The patent creates a simplified copy or representation of the complex internal model states. Instead of directly accessing and processing the full complexity of the large language model's internal values, the system generates a copied representation that captures the essential certainty information in a more manageable form.
2Device complexity
If black-box methods are used for certainty determination, then device complexity is reduced, but productivity decreases due to computational cost
Solution Approach 1:
The patent extracts only the essential certainty-related information from the complex black-box model operations. By taking out and isolating the specific certainty values needed for decision-making, the system avoids the full computational overhead of the black-box method while maintaining simplicity.
Solution Approach 2:
The patent applies partial action by performing certainty determination on only the necessary subset of model outputs rather than processing all possible outputs. This selective approach reduces computational cost while still providing adequate certainty assessment for the actions that matter most.
3Adaptability or versatility
If all generated actions are presented to the user, then adaptability is improved, but loss of information increases due to including low-certainty hallucinated actions
Solution Approach 1:
The patent implements a feedback mechanism where certainty values are used to evaluate and filter the generated actions before presentation. This feedback loop allows the system to maintain adaptability by providing multiple action options while simultaneously reducing information loss by filtering out low-certainty hallucinated actions based on their certainty assessments.
4Reliability
If multiple actions are generated with certainty values, then reliability is improved, but use of energy increases due to additional computational processing
Solution Approach 1:
The patent applies preliminary action by generating and assessing multiple actions with certainty values before final selection. This advance processing ensures reliability by identifying high-certainty actions early, while the energy consumption is managed through efficient algorithms that assess certainty without requiring excessive computational resources.
Data Source
AI summary
A computer-implemented method including receiving, by a computing system, a first user input associated with a requested action, executing, by the computing system, a large language engine to generate a plurality of actions to be performed by at least a subset of the accessory devices based at least in part on the received user input, executing, by the computing system, an estimation engine to generate a certainty value associated with each action of the plurality of actions, presenting, by the computing system, a subset of actions of the plurality of actions to a user based at least in part on the certainty values of the plurality of actions, receiving, by the computing system, a second user input selecting an action of the subset of actions, and instructing, by the computing system, one or more accessory devices of the at least a subset of the accessory devices to perform the action.