Self-Reward Autoregressive Sampling for Safer LLM Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) generate undesirable responses, such as harmful or toxic sentences, due to autoregressive sampling, which can be harmful or humiliating towards certain groups, necessitating improved governance and traceability to manage and monitor AI-based activities.
Innovation Solution
Implement a self-reward guided autoregressive sampling process using a self-reward model that generates scores based on sentence attributes, altering token sampling probabilities to ensure responses meet defined toxicity, harmfulness, or other attribute thresholds, employing Bayes optimal classifiers to model an optimal embedding space and interact with the LLM's decoding mechanism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If autoregressive sampling is used for text generation, then generation speed and simplicity are improved, but harmful or toxic content is generated
Solution Approach 1:
The patent implements a feedback mechanism where the reward model evaluates generated tokens and provides reward signals that are used to update sampling probabilities. This closed-loop feedback system allows the model to self-correct and avoid harmful content while maintaining generation speed through efficient probability reweighting.
Solution Approach 2:
The patent introduces a reward model as an intermediary component between the language model and the sampling process. This intermediary evaluates token quality based on harmfulness attributes and modulates sampling probabilities without directly controlling generation, thus filtering harmful content while preserving natural language flow.
2Object-affected harmful factors
If token sampling probabilities are updated to reduce harmful content, then content safety is improved, but computational overhead increases
Solution Approach 1:
The patent changes the probability parameters of token sampling dynamically based on reward signals. By adjusting sampling probabilities rather than regenerating entire sequences or using complex filtering mechanisms, the system achieves toxicity reduction with minimal computational overhead, leveraging efficient probability reweighting operations.
Solution Approach 2:
The patent applies partial action by updating sampling probabilities only for tokens that receive low reward signals from the harmfulness model. Instead of processing all tokens equally or using exhaustive filtering, the system selectively modifies probabilities for potentially harmful tokens, reducing unnecessary computational burden while maintaining safety.
3Reliability
If self-reward model is introduced to guide sampling, then response quality and safety are improved, but system complexity increases
Solution Approach 1:
The patent designs the self-reward model to serve multiple functions: evaluating token quality, generating reward signals, and guiding sampling probabilities. This multi-functional component improves response safety without requiring separate specialized modules for each function, thereby limiting the increase in system complexity while achieving reliable safe responses.
Data Source
AI summary
One or more systems, devices, computer program products and/or computer-implemented methods of use provided herein relate to self-reward guided autoregressive sampling for large language models (LLMs). The system can comprise a processor that can execute computer executable components stored in a memory, where the computer executable components can comprise at least one self-reward model. The at least one self-reward model can generate a score for a sentence generated by an LLM, where the score can be based on one or more tokens comprised in the sentence and an attribute associated with the at least one self-reward model. The at least one self-reward model can further alter a text generation process employed by the LLM to generate the sentence, such that respective sampling probabilities of respective tokens comprised in a vocabulary employed by the LLM to generate a new token can be updated by the LLM based on the score.