Adaptive MoE Routing with Variable Expert Activation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional MoE language models activate a fixed number of expert networks, leading to inefficiencies in processing time and resource usage due to a trade-off between inference speed and accuracy, and inability to adapt to input data characteristics.
Innovation Solution
An adaptive routing method that dynamically adjusts the number of activated expert networks based on expert weights and update settings, using bias and uniformity values to optimize network selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a fixed number of expert networks are activated in conventional MoE language models, then the inference process is simplified and processing is predictable, but the accuracy decreases and processing time increases due to inability to adapt to input data characteristics
Solution Approach 1:
The patent implements dynamic expert network selection by replacing the fixed router with a dynamic expert network selector that adaptively determines which expert networks to activate based on input data characteristics. This allows the system to transition from static to dynamic operation, optimizing accuracy without sacrificing operational simplicity.
Solution Approach 2:
The system changes the parameter of expert network selection from fixed to variable by introducing a dynamic selector that adjusts the number and type of activated expert networks based on input characteristics. This parameter change enables the system to adapt to different data scenarios while maintaining predictable processing through controlled dynamics.
2Measurement precision
If more expert networks are activated to improve accuracy, then processing accuracy increases, but inference speed decreases and computing resources increase
Solution Approach 1:
The dynamic expert network selector applies local quality by activating different numbers and types of expert networks for different input data characteristics. Rather than uniformly activating all expert networks or a fixed number, the system selectively activates only the necessary expert networks for each specific input, optimizing both accuracy and inference speed locally for each data point.
Solution Approach 2:
The system implements partial action by activating only the necessary subset of expert networks required for accurate processing of each input, rather than activating all expert networks. This partial activation maintains accuracy while reducing computational overhead and preserving inference speed.
3Device complexity
If a fixed number of expert networks are activated, then device complexity is reduced and resource consumption is controlled, but adaptability to different operating environments and input data characteristics is lost
Solution Approach 1:
The dynamic expert network selector implements self-service by automatically determining the optimal expert network configuration based on input data characteristics without requiring manual intervention or complex external configuration. The system serves itself by adapting to different operating environments autonomously, maintaining low complexity while achieving high adaptability.
Solution Approach 2:
The system performs preliminary action by pre-training the dynamic expert network selector to recognize input data characteristics and predict the optimal expert network configuration. This preliminary learning enables the system to adapt quickly to different inputs without requiring complex real-time adjustments or extensive configuration.
Data Source
AI summary
A processor-implemented method including collecting update setting values for a mixture-of-experts language model, executing inference using the mixture-of-experts language model and collecting expert weights generated by a router of a layer of the mixture-of-experts language model to select an expert network to distribute tokens from among a plurality of expert networks, and updating a selected number of expert networks to be activated for the router of the layer, based on the expert weights and the update setting values.