Neural Network Training via Conditional Reward Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current reinforcement learning methods for training neural networks face instability and reduced exploration during training, leading to suboptimal generalization performance.
Innovation Solution
The system computes importance weights using a likelihood score-based factor that accounts for candidate network outputs, promoting exploration and stability by normalizing rewards conditionally and using a robust importance weighting scheme.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning is used to train neural networks, then the network can learn complex tasks, but training stability deteriorates and exploration is reduced
Solution Approach 1:
The patent applies parameter changes by modifying the reward normalization approach from global to conditional normalization based on input features. By computing separate normalization statistics for different input conditions, the system maintains appropriate reward scales across diverse scenarios, improving training stability while preserving the network's learning capability for complex tasks.
Solution Approach 2:
The patent implements dynamics by making the normalization parameters adaptive rather than static. The conditional reward normalization dynamically adjusts normalization factors based on the specific input condition being processed, allowing the training process to adapt to varying difficulty levels and maintain stability throughout training while enabling exploration of diverse solutions.
2Adaptability or versatility
If reinforcement learning is used to train neural networks, then the network can learn complex tasks, but generalization performance is reduced
Solution Approach 1:
The patent improves generalization performance by changing the normalization parameters from global averages to conditional statistics computed per input type. This allows the reward signals to be appropriately scaled for each specific task condition, enabling the network to learn more robust policies that generalize better to unseen inputs while maintaining the ability to learn complex task-specific behaviors.
3Stability of the object's composition
If exploration is reduced during training, then training becomes more stable, but the network fails to discover diverse solutions
Solution Approach 1:
The patent resolves this contradiction by implementing dynamic conditional normalization that adapts to the specific input condition. For difficult inputs requiring exploration, the normalization preserves larger reward variations that encourage exploration, while for easier inputs, it provides stable gradients. This dynamic adjustment maintains training stability overall while preserving exploration capability when needed.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network to perform a machine learning task through reinforcement learning. In one aspect, the training uses importance weights generated using standardized absolute deviations of quality scores generated by the neural network for candidate network outputs.