Switchable MoE Layer Parallel Modes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural network (DNN) model capacity is limited by computing and energy costs due to dense architectures, and Mixture-of-Experts (MoE) models face inefficiencies in training and execution across multiple processing devices due to varying workloads and difficulties in parallelization, leading to either underutilization of capacity or token dropping.
Innovation Solution
A computing system with multiple processing devices executes an MoE layer by dynamically selecting destination expert sub-models and adjusting expert capacity based on a capacity factor, using a gating function to optimize token distribution and avoid underutilization or token dropping, allowing for efficient scaling and parallelization without parameter migration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If MoE models use more model parameters to increase model capacity, then model accuracy is improved, but computing and energy costs increase linearly
Solution Approach 1:
The model is segmented into multiple expert sub-models that process different subsets of inputs. Each expert sub-model has specialized parameters, and through the gating function, only relevant experts are activated for each input token. This segmentation allows the model to have large total parameter count (for accuracy) while keeping the active parameter count per forward pass manageable (controlling computing costs).
Solution Approach 2:
The MoE model dynamically selects which expert sub-models to activate based on the input characteristics. The gating function dynamically routes each input token to the most appropriate expert, making the model adaptive and efficient. This dynamic selection ensures that computing resources are focused on the most relevant parameters for each specific input, improving both accuracy and efficiency.
2Measurement precision
If MoE models enroll more model parameters to scale model capacity, then model accuracy increases, but additional computing costs are incurred
Solution Approach 1:
Different expert sub-models have different local qualities (specialized parameter sets) optimized for specific types of inputs. The gating function identifies which local quality (expert) is most appropriate for each input token. This allows the model to have diverse specialized parameters (increasing overall capacity and accuracy) while only activating the locally optimal expert for each token (controlling computing costs).
3Measurement precision
If MoE models use sparse architecture with multiple parallel expert sub-models, then model capacity scales up without large additional costs, but workload distribution becomes uneven across processing devices
Solution Approach 1:
The gating function acts as an intermediary that mediates between input tokens and expert sub-models. It computes routing scores and selects appropriate experts, thereby controlling the workload distribution. By adjusting the gating function's output (e.g., temperature parameter, top-k selection), the system can balance between utilizing more experts (increasing capacity) and maintaining reasonable workload distribution across processing devices.
4Adaptability or versatility
If MoE models dynamically select destination expert sub-models with varying numbers k across iterations, then model adaptability improves, but system complexity increases
Solution Approach 1:
The system changes parameters (number of active experts k, temperature parameter) dynamically based on input characteristics and training progress. This allows the model to adapt to different input types and training stages. The parameter changes are controlled through the gating function, which uses learned parameters to determine expert selection, thereby managing complexity through learned control rather than hard-coded complexity.
Data Source
AI summary
A computing system is provided, including a plurality of processing devices configured to execute a Mixture-of-Experts (MoE) layer included in an MoE model. The MoE layer includes a plurality of expert sub-models that each have a respective plurality of parameter values. The MoE layer is configured to be switchable between a data parallel mode and an expert-data-model parallel mode without conveying the respective parameter values of the expert sub-models among the plurality of processing devices.