Federated LLM Training With MoE Expert Selection and LoRA
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing federated learning methods face challenges in efficiently training large language models while protecting data privacy and minimizing communication costs, particularly due to the high number of parameters involved in mixture-of-experts (MoE) and low-rank adaptation (LoRA) approaches.
Innovation Solution
Implementing a method that identifies and transmits a gating network and expert models corresponding to external electronic devices, utilizing low-rank adaptation (LoRA) to reduce the number of parameters transmitted, thereby reducing communication costs and enabling parameter-efficient training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire large language model is transmitted to external electronic devices for federated learning, then training accuracy is maintained, but communication costs increase significantly
Solution Approach 1:
The patent segments the large language model into a gating network and multiple expert models, then further segments the expert models into active expert models and inactive expert models. Only the active expert models are transmitted to external electronic devices for federated learning, while the gating network remains on the server. This segmentation reduces the amount of data transmitted while maintaining training accuracy through the gating mechanism that selects appropriate experts.
Solution Approach 2:
The patent extracts only the necessary components (active expert models) from the entire large language model for transmission to external devices. The gating network is extracted and kept on the server, and inactive expert models are excluded from transmission. This extraction approach reduces communication costs while preserving the essential functionality needed for accurate training.
2Adaptability or versatility
If all expert models are transmitted to external electronic devices, then model functionality is complete, but communication bandwidth requirements increase
Solution Approach 1:
The patent implements a dynamic expert model selection mechanism where the gating network dynamically determines which expert models to activate based on input data characteristics. This allows the system to adaptively select only the necessary expert models for each training task, rather than transmitting all expert models statically. The dynamic gating mechanism maintains model functionality while significantly reducing communication bandwidth requirements.
Solution Approach 2:
The patent applies local quality by making different parts of the model have different transmission characteristics. The gating network remains locally on the server with full functionality, while only specific expert models are transmitted to external devices. Each external device receives only the expert models it needs for its local training data distribution, optimizing the quantity of data transmitted while maintaining necessary functionality.
3Ease of operation
If the gating network is transmitted to external electronic devices, then local training capability is enhanced, but data privacy protection is compromised
Solution Approach 1:
The patent uses the gating network as an intermediary component that remains on the server and mediates between the external electronic devices and the expert models. The gating network receives input data from external devices, determines which expert models should be activated, and controls the transmission of only necessary expert models. This intermediary role enhances local training capability while maintaining data privacy by preventing direct access to all model components.
4Loss of energy
If expert models are selectively transmitted based on device needs, then communication costs are reduced, but system complexity increases
Solution Approach 1:
The patent performs preliminary action by pre-classifying expert models into active and inactive categories on the server before transmission to external devices. The gating network is pre-configured with knowledge of which expert models are active and their corresponding characteristics. This preliminary classification and preparation simplifies the transmission decision-making process for external devices, reducing communication costs while limiting the complexity increase to manageable levels through pre-established model categorization.
Data Source
AI summary
According to an embodiment, an electronic device may include communication circuitry; at least one processor including processing circuitry; and memory storing instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to: identify a plurality of expert models corresponding to a plurality of external electronic devices, in a large language model, wherein the plurality of external electronic devices is configured to perform federated learning, and the large language model includes a gating network and the plurality of expert models, and transmit, to the plurality of external electronic devices, through the communication circuitry, an expert model corresponding to the gating network and a corresponding external electronic device.