Stochastic Neural Network Gating for Stable Diverse Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in training stochastically activated neural networks lies in accurately and efficiently leveraging previous layers' activations while maintaining diversity and stability, particularly in models like Mixture of Experts (MoE), where activation saturation and partial observability create conflicts in training procedures.
Innovation Solution
The approach involves using fixed or adjustable information weights alongside control-weights to modulate the stochastic policy, allowing the network to be trained without back-propagation by directly applying a policy gradient step to each node, with control nodes determining which parts of the network to activate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the control network leverages previous layers' activation values to determine node activation, then the control actions become clear and decisive, but the node output diversity decreases making it difficult for subsequent layers to determine which nodes to activate
Solution Approach 1:
The patent segments the weight parameters into two distinct sets: control-weights that determine activation policy and information-weights that preserve output diversity. This segmentation allows the control network to simultaneously achieve clear activation decisions through control-weights while maintaining diverse node outputs through information-weights, resolving the contradiction between control clarity and output diversity.
2Device complexity
If the control network uses the same weights for both control and information flow, then the network structure is simplified, but training becomes unstable due to conflicting objectives of clear policy and expressive output
Solution Approach 1:
The patent divides the weight parameters into separate control-weights and information-weights, eliminating the conflict between control clarity and output diversity by giving each function its own dedicated parameters. This segmentation prevents training instability while maintaining structural organization through the dual-weight framework.
Solution Approach 2:
The patent applies different weight sets to different functional requirements: control-weights are optimized for clear activation decisions while information-weights are optimized for diverse and expressive node outputs. This local quality differentiation ensures that each part of the network serves its specific function effectively without interfering with other functions.
3Manufacturing precision
If the entire network is trained using back-propagation, then all parameters can be optimized, but computational overhead increases significantly
Solution Approach 1:
The patent extracts the control-weights from the standard back-propagation training process and trains them separately using policy gradient methods. This extraction allows the control policy to be optimized independently from the main network parameters, reducing computational overhead while still achieving effective parameter optimization for both control and information flow.
Data Source
AI summary
Embodiments of the present disclosure provide a stochastic neural network with training where the inputs to the gating function are not dependent on the control values of the previous layers' learnable weights but rather from a set of control-information weights, which may be fixed or adjustable. In various embodiments, a control node contains one or more control-parameters which control a set of activations and one or more information-parameters for the control node's output, to be consumed as inputs to the subsequent layers of control nodes. In various embodiments, one or more control-parameters and one or more information-parameters may be shared.