Federated LLM Pruning With Sparse Local Models for Device Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face challenges in computational resource requirements, data privacy, and heterogeneity of client devices in federated learning scenarios, limiting their practical deployment and efficiency across diverse devices and use cases.
Innovation Solution
A federated learning system that prunes a large language model into multiple small language models (SLMs) with varying sparsity levels, fine-tunes them on local data using Low-Rank Adaptation (LoRA), and aggregates these models to create a global update for a robust LLM, while maintaining privacy and handling heterogeneous architectures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a large language model is used, then generalization capability is improved, but computational resource requirements increase
Solution Approach 1:
The patent segments the large language model into multiple smaller language models with different sparsity levels (e.g., 50% sparsity, 75% sparsity). Each SLM is designed to operate within specific resource constraints while maintaining task-specific performance. This segmentation allows the system to distribute computational load across multiple smaller models rather than relying on a single large model, thereby reducing the computational resource requirements while preserving generalization capability through the ensemble of diverse SLMs.
2Use of energy by moving object
If model size is reduced, then resource utilization is improved, but robustness deteriorates
Solution Approach 1:
The patent applies local quality by creating SLMs with different sparsity levels tailored to specific client devices' resource constraints. Each SLM is locally optimized for its target device's computational capabilities, memory capacity, and energy constraints. This local optimization ensures that each SLM achieves the best possible performance on its specific hardware platform while maintaining robustness through device-specific adaptation rather than relying on a one-size-fits-all large model.
Solution Approach 2:
The patent merges multiple SLMs with different sparsity levels into a federated ensemble that collectively provides robust performance. By combining the predictions or parameters of multiple SLMs through federated learning aggregation, the system achieves robustness that individual small models cannot provide alone. This merging strategy allows the system to leverage the strengths of each SLM while compensating for individual limitations, thereby maintaining robustness despite reduced model sizes.
3Loss of information
If federated learning is implemented, then data privacy is improved, but communication overhead increases
Solution Approach 1:
The patent extracts only the essential model parameters and updates from each client device during federated learning, rather than transmitting entire models or raw data. By extracting and transmitting only the necessary parameter updates (such as LoRA adapter parameters or pruning masks), the system minimizes communication overhead while preserving data privacy. This extraction approach allows clients to contribute to the global model without sharing their local data or full model states, thereby reducing the quantity of data transmitted across the network.
4Adaptability or versatility
If heterogeneous client devices are supported, then adaptability is improved, but system complexity increases
Solution Approach 1:
The patent uses parameter changes by varying the sparsity levels of SLMs to match different client device capabilities. Each SLM's architecture parameters (such as pruning ratio, layer configuration, or adapter size) are adjusted according to the target device's computational power, memory capacity, and energy constraints. This parameter adaptation allows the system to support heterogeneous devices without requiring complex device-specific model versions, thereby improving device compatibility while managing system complexity through a unified SLM generation framework with configurable parameters.
Data Source
AI summary
The present disclosure provides a method for federated fine-tuning of language models. The method comprises pruning a large language model (LLM) to create multiple small language models (SLMs) with different sparsity levels, assigning each SLM to a client device, fine-tuning each SLM on local data of its assigned client device, aggregating the fine-tuned SLMs to create a global update, and applying the global update to the SLMs and a global LLM. The method enables efficient fine-tuning and inference while preserving privacy and optimizing performance across varied resource constraints.