Balancing Expert Utilization in Sparse Mixture of Experts Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks with sparse mixture of experts (sMoE) layers face challenges in balancing expert utilization, leading to specialists in non-semantic distinctions rather than semantic ones, due to unstable gating and the need for auxiliary loss functions.
Innovation Solution
A method involving the calculation of an exponential moving average of the non-differentiable component of the auxiliary loss function over multiple minibatches and accelerator units, combined with a differentiable component and a scaling factor, to produce a balancing auxiliary loss that promotes balanced expert utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If an auxiliary loss function is used to balance expert utilization in sMoE layers, then expert balance is improved, but gating stability deteriorates and experts may erroneously specialize using non-semantic distinctions
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing the non-differentiable component (expert assignment indicators) from the auxiliary loss function before the optimization step. This pre-computed component is then combined with the differentiable gating component during optimization, allowing the system to benefit from balanced expert utilization guidance without requiring the gating function to directly handle the non-differentiable balancing term, thus maintaining gating stability while achieving expert balance
Solution Approach 2:
The patent introduces an intermediary approach by separating the auxiliary loss function into two distinct components: a differentiable gating component that maintains stability and a pre-computed non-differentiable component that provides balancing guidance. This separation acts as an intermediary mechanism that allows the non-differentiable balancing term to influence expert utilization without directly disrupting the gating function's stability, resolving the contradiction between balance and stability
2Adaptability or versatility
If the number of experts in sMoE layers is increased to improve model capacity, then model versatility is improved, but training complexity and difficulty of balancing expert utilization increase
Solution Approach 1:
The patent applies segmentation by dividing the training process into distinct computational stages: pre-computation of the non-differentiable balancing component, followed by optimization that combines this pre-computed component with the differentiable gating component. This segmentation of the training workflow reduces the per-iteration complexity that would otherwise arise from directly optimizing large-scale sMoE layers with many experts, making training more manageable while preserving model capacity
Solution Approach 2:
The patent uses preliminary action to pre-compute the non-differentiable component of the auxiliary loss function before the optimization step. This pre-computation approach reduces the computational burden during the main optimization process, especially when dealing with a large number of experts, thereby reducing training complexity while allowing the model to maintain high versatility through increased expert count
Data Source
AI summary
Disclosed herein are systems, devices, and methods for improving balancing in neural networks with at least one sparse mixture of experts (sMoE) layer. In some aspects, such improvements can be made during the pre-training phase of the neural network. For example, a method of pre-training such a neural network can comprise calculating a new non-differentiable component of an auxiliary loss function, multiplying the new non-differentiable component in an element-by-element manner by a differentiable component of the auxiliary loss function and summing over its dimensions to produce a raw penalty score, and multiplying the raw penalty score by a scaling factor to produce a balancing auxiliary loss.