Prior-Guided Mixture of Experts Routing for Stable LLM Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning machine learning models, such as large language models (LLMs), are large, slow, and have high memory and computational demands, making them difficult to train and deploy on resource-constrained devices like mobile devices, and face challenges like expert collapse and prolonged training times.
Innovation Solution
Implementing prior-guided mixture of experts (MOEs) with a gating regularization loss and routing functions to inject priors into expert-execution patterns, ensuring more temporally consistent decisions and efficient on-device execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning models are used to achieve high quality results in various tasks, then accuracy and versatility are improved, but model size, memory demands, and computational costs increase
Solution Approach 1:
The model is divided into multiple expert networks, each specialized in handling specific types of tokens or tasks. The router segments the input space and directs different tokens to appropriate experts, allowing the system to achieve high accuracy for diverse tasks while keeping each expert relatively small and efficient.
Solution Approach 2:
The mixture of experts architecture creates a universal model that can handle multiple tasks and token types through a single unified structure. The router dynamically selects which experts to activate based on the input, enabling one model to perform versatile functions across different NLP tasks without requiring separate specialized models.
2Measurement precision
If deep learning models are used to achieve high quality results, then accuracy is improved, but processing speed decreases
Solution Approach 1:
By segmenting the model into specialized experts and using a router to activate only the necessary experts for each input token, the system processes tokens more efficiently. This selective activation reduces the computational burden compared to passing every token through all model parameters, thereby improving processing speed while maintaining accuracy.
3Use of energy by moving object
If mixture of experts models are used to reduce computational costs, then resource efficiency is improved, but training stability deteriorates due to expert collapse
Solution Approach 1:
The auxiliary loss function provides feedback during training to monitor and regulate the activation distribution across experts. This feedback mechanism prevents any single expert from dominating (expert collapse) by encouraging more balanced utilization, thereby stabilizing training while maintaining the computational efficiency benefits of selective expert activation.
4Use of energy by moving object
If mixture of experts models are used to reduce computational costs, then resource efficiency is improved, but training time increases
Solution Approach 1:
The auxiliary loss function guides the training process by providing continuous feedback on expert activation patterns. This feedback accelerates convergence by preventing pathological behaviors like expert collapse early in training, reducing the overall training time required to achieve stable and effective model performance.
Data Source
AI summary
Systems and techniques are described herein for language processing. For example, a computing device can determine, using a router of a mixture of experts (MOE) machine learning model, a distribution of tokens for a respective category associated with each expert layer of a plurality of expert layers of the MOE model. The computing device can train each expert layer of the plurality of expert layers based on matching the distribution of tokens associated with each respective expert layer of the plurality of expert layers to a prior distribution of the tokens for the respective category associated with each respective expert layer of the plurality of expert layers.