Token Generator Subsystem for Reinforcement Learning Text Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning-based controlled natural language generation systems face challenges with 'sparse rewards' that are only available at the output sentence level, leading to poor performance in tasks like text style transfer, as they require rewards at the output token level for precise feedback.
Innovation Solution
A system and method that incorporate a token generator subsystem, attention layer, scoring layer, reward generation layer, and feedback score generation layer to provide dense rewards and feedback scores for each output token, enabling precise token-level feedback and improving the quality of generated text.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If rollout strategy is used to overcome sparse rewards, then token-level rewards can be obtained, but precise reward assignment is not achieved leading to poor performance
Solution Approach 1:
The patent implements a feedback mechanism where the generated output is evaluated by a critic model that provides dense rewards at each token generation step. The critic model compares the generated token with the target style and provides immediate feedback, enabling precise reward assignment that guides the generator model to improve its output quality iteratively.
Solution Approach 2:
The patent introduces a critic model as an intermediary between the generator model and the reward signal. This critic model acts as a mediator that evaluates the generated tokens and translates the sparse sentence-level rewards into dense token-level rewards, enabling precise feedback without requiring direct access to parallel corpus data.
2Measurement precision
If dense rewards are provided at token level, then precise feedback is achieved, but system complexity increases
Solution Approach 1:
The patent segments the natural language processing task into distinct components: a generator model that produces tokens, a critic model that evaluates tokens, and a reinforcement learning module that updates policies. This segmentation allows each component to specialize in one function, achieving precise token-level feedback while managing system complexity through modular architecture.
Solution Approach 2:
The critic model serves as an intermediary that simplifies the complex task of evaluating generated tokens. Instead of implementing complex evaluation logic in the generator, the critic model handles the sophisticated analysis of token quality, style adherence, and fluency, thereby achieving precise feedback without overcomplicating the overall system architecture.
Data Source
AI summary
A system for reinforcement learning based controlled natural language generation is disclosed. The system includes a token generator subsystem to generate an initial output phrase including a sequence of output tokens. The system includes trained models associated with corresponding predefined tasks. Each trained model includes an attention layer to compute attention-based weights for each output token. The trained models include a scoring layer to generate a phrase sequence level score for the output phrase. The trained models include a reward generation layer to generate dense rewards for each output token based on the attention-based weights and the phrase sequence level score. The trained models include a feedback score generation layer to generate a feedback score based on the dense rewards and reward weights assigned to the dense rewards of the corresponding trained models. The feedback score generation layer provides the feedback score iteratively to the token generator subsystem.