FFN-to-MoE Conversion for Trained Language Models Without Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing trained language models face challenges in being easily converted to the MoE architecture due to the need for retraining with text data, which is time-consuming and resource-intensive, especially when applying advancements in MoE models to non-MoE models.
Innovation Solution
A method and computing device that convert non-MoE language models to MoE models by extracting FFNs, generating MoE blocks, and updating weights based on loss comparison using random or distribution-informed input tensors, allowing for efficient conversion without extensive retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a non-MoE language model is converted to MoE architecture by retraining with text data, then the model achieves MoE architecture benefits (fast inference, sparsity), but the conversion process is time-consuming and resource-intensive
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing the FFN outputs for each layer before conversion. These pre-computed outputs serve as target values during the weight updating process, eliminating the need for time-consuming retraining with text data while still achieving MoE architecture benefits
Solution Approach 2:
The patent uses copying by creating MoE blocks that replicate the computational behavior of the original FFNs. The MoE blocks are designed to produce identical outputs to the original FFNs for the same inputs, allowing the conversion to preserve model performance without requiring extensive retraining
2Reliability
If a non-MoE language model is converted to MoE architecture through weight updating, then the model maintains performance, but the conversion requires complex weight updating processes
Solution Approach 1:
The patent implements feedback by using the pre-computed FFN outputs as target values during the weight updating process. The weight updating algorithm continuously adjusts MoE block weights based on the difference between their outputs and these target values, ensuring the model maintains performance while adopting MoE architecture
Solution Approach 2:
The patent applies segmentation by processing the language model layer by layer, extracting FFNs from each layer independently, and converting them to MoE blocks separately. This modular approach simplifies the overall conversion process by breaking down the complex task into manageable steps that can be performed systematically
Data Source
AI summary
A processor-implemented method for converting a trained language model into a language model in an architecture of mixture of experts (MoE), and a computing device using the same is provided. The method for converting a trained language model into a language model in an architecture of mixture of experts using a computing device according to an embodiment of the disclosure may include dividing a plurality of layers included in a target language model and extracting a feed-forward network (FFN) included in each of the plurality of layers, generating an MoE block of the MoE language model, which corresponds to the feed-forward network, generating an input tensor, comparing output tensors between the feed-forward network and the MoE block for the input tensor to obtain a first loss, and updating a weight of the MoE block, based on the first loss.