Decoder Plugin Constrained Token Vocabulary for Reliable AI Output
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges such as hallucinations, autoregressive drift, limited context window size, and difficulties in preventing harmful outputs, which affect accuracy and resource efficiency.
Innovation Solution
A computing system with a decoder plugin that executes guidance logic to compute a constrained output token vocabulary, using oversight models and search algorithms to select output tokens based on probability distributions and context, thereby controlling the output of machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the full output token vocabulary is used for generation, then the model has higher flexibility and creativity, but the risk of hallucinations and harmful outputs increases
Solution Approach 1:
The output token vocabulary is segmented into a constrained subset that is dynamically selected based on the input context. The decoder plugin divides the full vocabulary into context-relevant tokens, allowing the model to focus on appropriate output options while maintaining reliability.
Solution Approach 2:
The constrained output token vocabulary is pre-computed and prepared before the main generation process. By preliminarily filtering and organizing relevant tokens based on the input prompt, the system reduces the search space and prevents harmful outputs before generation begins.
2Productivity
If prompting is used to control outputs, then the model can generate relevant responses, but the context window size limit restricts the amount of input information that can be processed
Solution Approach 1:
The system extracts and isolates the most relevant tokens from the full vocabulary based on the input context. By taking out only the necessary tokens for the current task, the system reduces the effective vocabulary size and computational burden while maintaining generation quality.
Solution Approach 2:
The system dynamically changes the parameter of vocabulary size based on the input context. Instead of using a fixed large vocabulary, the vocabulary size is adjusted to match the specific requirements of each prompt, optimizing both efficiency and relevance.
3Adaptability or versatility
If the full output token vocabulary is used, then more output options are available, but computational resources and time are consumed
Solution Approach 1:
The system applies partial action by using only the necessary portion of the full vocabulary required for each specific task. Instead of processing the entire vocabulary, it selectively activates and processes only the relevant subset, reducing computational overhead while maintaining output diversity where needed.
Solution Approach 2:
The relevant tokens are pre-identified and prepared before the generation process. This preliminary filtering of the vocabulary based on input context reduces the computational burden during actual generation, as the model only needs to work with a smaller, pre-selected set of tokens.
Data Source
AI summary
A computing system including one or more processing devices configured to receive a prompt. At a machine learning model that has an output token vocabulary including candidate output tokens, the one or more processing devices are further configured to compute output token probabilities over the output token vocabulary based at least in part on the prompt. At a decoder plugin, the one or more processing devices are further configured to compute a constrained output token vocabulary as a proper subset of the output token vocabulary. The one or more processing devices are further configured to select output tokens based at least in part on the computed output token probabilities. The output tokens are selected from among the candidate output tokens included in the constrained output token vocabulary. The one or more processing devices are further configured to transmit an output including the output tokens to an additional computing process.