Federated Learning Global Model Compression via Feature Map Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning systems face challenges in communication efficiency due to large model sizes, leading to performance loss as devices with limited bandwidth are excluded from updates, and quantization in the upload direction results in significant compression loss.
Innovation Solution
A framework for communication-efficient federated learning that compresses the global model before transmission, reconstructs feature maps on the server from local updates, and performs quantization bias correction and mix-up to refine the model, reducing downlink communication costs and preserving model performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the global model is transmitted without compression, then the model performance is maintained, but the downlink communication cost increases and devices with limited bandwidth are excluded
Solution Approach 1:
The patent applies quantization to change the precision parameter of global model weights from high precision (e.g., 32-bit floating point) to low precision (e.g., 8-bit integers), thereby reducing communication cost while maintaining acceptable model performance through subsequent refinement techniques
Solution Approach 2:
The patent creates a refined copy of the quantized global model by reconstructing feature maps and applying refinement techniques, producing a corrected version that maintains model performance while keeping communication costs low
2Quantity of substance
If quantization is applied to compress the global model, then the downlink communication cost is reduced, but compression loss occurs and model performance deteriorates
Solution Approach 1:
The patent uses reconstructed feature maps as feedback to identify and correct quantization errors in the quantized global model, applying refinement techniques that use this feedback information to restore model performance after compression
Solution Approach 2:
The patent replaces the direct transmission of high-precision model weights with an alternative mechanism involving quantization, reconstruction of feature maps, and refinement, substituting a complex mechanical transmission system with a more efficient computational approach
3Device complexity
If feature maps are not reconstructed, then the server processing is simpler, but quantization bias correction cannot be performed and compression loss increases
Solution Approach 1:
The patent performs preliminary reconstruction of feature maps from local updates before the refinement step, preparing the necessary information in advance to enable accurate quantization bias correction and reduce compression loss
Data Source
AI summary
A server and method thereof are provided for use in a federated network. A method includes receiving local updates from client devices; updating a global model based on the received local updates; quantizing the updated global model; reconstructing feature maps based on the received local updates; refining the quantized, updated global model based on the reconstructed feature maps; and transmitting the refined, quantized, updated global model to the client devices.