Federated Learning With Ordered Dropout for Heterogeneous Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning is challenging due to the vast heterogeneity of client devices, leading to computational disparities that hinder efficient training across devices, resulting in biased and inaccurate models, as low-tier devices are often excluded or struggle to participate effectively.
Innovation Solution
Implementing ordered dropout to dynamically prune neural network layers based on client device capabilities, generating nested submodels that match the computational and memory requirements of each device, allowing all devices to contribute to the training process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large-scale ML model is used to improve accuracy, then model accuracy is improved, but computational requirements increase making it infeasible for low-tier client devices
Solution Approach 1:
The patent segments the large-scale ML model into multiple smaller submodels based on client device capabilities. Each client device receives and trains only on submodels appropriate to its computational capacity, allowing low-tier devices to participate in federated learning without being overwhelmed by the full model size, while still contributing to the global model improvement
Solution Approach 2:
The patent applies local quality by customizing the model size and complexity for each client device based on its specific capabilities. High-tier devices receive and train on larger submodels with more parameters, while low-tier devices receive smaller submodels, ensuring each device operates at its optimal computational level rather than forcing uniform model distribution
2Productivity
If low-tier devices are excluded from federated learning to maintain training speed, then training efficiency is improved, but model bias increases and accuracy decreases
Solution Approach 1:
The patent implements dynamic model adaptation where the submodel size and complexity are adjusted based on each client device's capabilities. This dynamic approach allows low-tier devices to participate by receiving appropriately sized submodels, enabling them to contribute to training without becoming stragglers, thus maintaining both training efficiency and model accuracy through inclusive participation
3Adaptability or versatility
If model size is reduced to include more devices in training, then device inclusivity is improved, but model accuracy decreases
Solution Approach 1:
The patent segments the full ML model into multiple submodels of varying sizes and complexities. This segmentation enables device inclusivity by allowing low-tier devices to train on smaller submodels while high-tier devices train on larger submodels, with all contributions aggregated to form the global model, thereby maintaining high accuracy through diverse device participation
Solution Approach 2:
The patent employs a nested structure where smaller submodels are contained within larger submodels, which are in turn contained within the full global model. This nesting allows devices to participate at their appropriate level - low-tier devices train on inner submodels while high-tier devices train on outer submodels - ensuring both inclusivity and accuracy are maintained
Data Source
AI summary
Broadly speaking, the present techniques generally relates to methods, systems and apparatuses for training a machine learning (ML) model using federated learning. In particular, a method for training a machine learning (ML) model using federated learning performed by a plurality of client devices, the method comprising determining a computation capability of each client device, associating each client device with a value defining how much of each neural network layer of the ML model is to be included in a submodel to be trained by the each client device, based on the determined computation capability and generating a submodel of the ML model by using the value associated with the each client device to perform ordered pruning of at least one neural network layer of the ML model, is provided.