Federated Learning Partial Model Updates for Lower Client Resource Use
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning of machine learning models consumes excessive resources at client devices due to the need to store and process full-sized models, leading to inefficiencies in memory and computation.
Innovation Solution
Generate updates for specific portions of on-device ML models, transmitting only these updates to a remote system for updating global models, thereby reducing resource consumption at client devices and network bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-sized ML models are stored and trained locally at client devices, then model update accuracy is improved, but on-device memory and computational resources are excessively consumed
Solution Approach 1:
The patent segments the ML model into multiple layers and divides the model update process into multiple iterations. In each iteration, only a subset of layers is updated using partial backward propagation, rather than computing gradients for the entire model. This segmentation allows accurate model updates while reducing memory consumption by storing activations only for the current subset of layers being updated.
2Measurement precision
If full-sized ML models are stored and trained locally at client devices, then model update accuracy is improved, but on-device computational resources are excessively consumed
Solution Approach 1:
The patent segments the computational workload by dividing the model layers into subsets that are processed in different iterations. Each iteration performs backward propagation only for the current subset of layers, significantly reducing the computational power required at each step while maintaining overall model update accuracy through cumulative updates across iterations.
Solution Approach 2:
The patent implements periodic action by alternating between forward propagation (computing predictions) and partial backward propagation (updating specific layer subsets) across multiple iterations. This periodic approach distributes the computational load over time, reducing peak power consumption while ensuring comprehensive model updates through repeated cycles.
3Measurement precision
If complete model updates are transmitted to the remote system, then global model accuracy is improved, but network bandwidth consumption is excessive
Solution Approach 1:
The patent extracts and transmits only the essential update information (gradient statistics or weight changes) for the specific subset of layers that were updated in each iteration, rather than transmitting complete model updates. This extraction approach maintains global model accuracy by sending targeted updates while dramatically reducing network bandwidth consumption by eliminating redundant data transmission.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Implementations disclosed herein are directed to efficient federated learning of machine learning (ML) model(s) at a remote system (e.g., remote server(s)) based on update(s) generated at client device(s). Processor(s) of the client device(s) can receive client data, process, using on-device ML model(s), the client data to generate predicted output(s), generate, using unsupervised learning, gradient(s) based on the predicted output(s), generate, based on the gradient(s), the update(s) for disparate portions of the on-device ML model(s) and/or global ML model(s) that are remote-based counterparts of the on-device ML model(s). Further, processor(s) of the remote system can receive, from the client device(s), the update(s) for the disparate portions of the on-device ML model(s), and cause the global ML model(s) to be updated based on the update(s) for the disparate portions of the on-device ML model(s) received from disparate client device(s). Thus, resources consumed at the client device(s) and/or network resources can be reduced.