LLM Token Selection Using a Distance Model for Length-Constrained Text
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current large language models (LLMs) lack the ability to constrain generated text to a specific predetermined length in tokens while maintaining sentence coherence, often resulting in incoherent outputs due to abrupt termination.
Innovation Solution
A method involving a distance model trained offline to predict the probability of a token fitting within a token budget, combined with a scoring phase to select the next token based on both LLM and distance model probabilities, ensuring coherence and adherence to the budget.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a hard-stop maximum token limit is used to constrain text generation, then the token budget is strictly adhered to, but sentence coherence is compromised due to abrupt termination
Solution Approach 1:
The system performs preliminary action by training a distance model offline to predict the distance to the end of a sentence. This pre-computed knowledge is then used during text generation to anticipate when the sentence will end, allowing the system to plan token selection in advance to ensure both budget adherence and coherent sentence completion.
Solution Approach 2:
The distance model serves as an intermediary between the LLM and the token budget constraint. Instead of directly imposing a hard stop, the distance model provides probabilistic guidance about sentence completion status, enabling the scoring mechanism to balance coherence and budget adherence indirectly through predicted distances.
2Quantity of substance
If text generation is terminated mid-sentence to comply with a specified budget, then the token budget is respected, but the generated text becomes incoherent
Solution Approach 1:
The system implements feedback by using the distance model to continuously monitor the predicted distance to sentence end during generation. This feedback loop allows the scoring mechanism to adjust token selection dynamically, favoring tokens that maintain coherence while progressively approaching the budget limit, and knowing when to terminate with a complete sentence.
Solution Approach 2:
The system changes parameters by transforming the hard-stop token limit into a soft constraint through the scoring mechanism. The distance model's probabilistic predictions modify the scoring parameters, allowing flexible token selection that adapts to the remaining budget while maintaining sentence coherence, rather than rigidly enforcing a fixed token count.
3Ease of operation
If external libraries with maximum token parameters are used, then the generation process can be stopped at a defined limit, but no guarantee of coherent output is provided
Solution Approach 1:
The distance model acts as an intermediary that transforms the simple maximum token parameter into intelligent coherence-aware generation. It mediates between the user-specified budget and the LLM's generation process, providing probabilistic guidance that ensures coherent output while respecting the operational constraint.
Solution Approach 2:
The system implements self-service by enabling the LLM to autonomously monitor its own generation progress through the distance model. The model serves itself by predicting sentence completion status and adjusting token selection to ensure coherent output, eliminating the need for external coherence verification mechanisms.
Data Source
AI summary
One example method includes receiving a set of input tokens, generating, using a text generation LM, respective first probabilities, for each of the input tokens, that the input token will be a next token in a text string, selecting, based on the respective first probabilities, a set of candidate tokens from the set of input tokens, inputting the candidate tokens to a distance model, generating, by the distance model, respective second probabilities for each of the candidate tokens, where each of the second probabilities is a probability that a corresponding one of the candidate tokens can be added to the text string without exceeding a token budget for that text string, performing, using the first probabilities and the second probabilities, a scoring phase to compute a score for each of the candidate tokens, and selecting the token with a highest score to be added as the next token in the text string.