Federated Learning Weight Dequantization for Precision Mismatch
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning between devices with different numeric precisions leads to degradation of model weight distribution and performance due to incompatible hardware specifications, making it challenging to maintain high-precision model accuracy when merging weights from clients with varying precision levels.
Innovation Solution
A method involving a dequantizer that changes weights to a common reference precision, determines masks based on inherent precisions, merges weights, and then quantizes them back to original precisions for transmission, ensuring compatibility and minimizing negative interference between weights of different bit widths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If federated learning is performed between devices with different numeric precisions, then device compatibility and adaptability are improved, but model weight distribution degrades and performance deteriorates
Solution Approach 1:
The patent introduces a server as an intermediary that receives weights from multiple clients with different precisions, converts them to a common reference precision using a dequantizer, merges them, and then distributes the merged weights back to clients. This intermediary processing resolves the precision incompatibility between devices while maintaining model weight distribution integrity.
Solution Approach 2:
The patent dynamically changes the precision parameter of model weights by using a dequantizer to convert weights from their original precisions to a common reference precision during merging, and then redistributes them with appropriate precisions to different clients based on their hardware capabilities.
2Adaptability or versatility
If weights from clients with different precisions are directly merged, then device adaptability is maintained, but model accuracy and performance deteriorate
Solution Approach 1:
The patent creates an equipotential state for weight merging by converting all client weights to a common reference precision before aggregation. This ensures that weights from different precision sources are on equal footing during the merging process, preventing precision mismatches from degrading model accuracy.
Solution Approach 2:
The server acts as an intermediary that mediates the precision conflict between diverse clients. It uses a dequantizer to standardize precisions, performs the merging operation, and then redistributes weights with precision-aware quantization to maintain both adaptability and accuracy.
3Manufacturing precision
If high-precision weights are transmitted between devices, then model accuracy is improved, but bandwidth consumption and transmission time increase
Solution Approach 1:
The patent dynamically changes the precision parameter of transmitted weights based on client hardware capabilities. The server quantizes weights to appropriate precisions before distribution, allowing high-precision weights to be maintained during processing while reducing precision during transmission to save bandwidth and time.
Solution Approach 2:
The patent implements dynamic precision adjustment where the precision of transmitted weights is not fixed but adapts based on the receiving device's hardware specifications. The server determines appropriate precision levels for each client and adjusts weight representation accordingly before transmission.
Data Source
AI summary
A method and device with federated learning of neural network models are disclosed. A method includes: receiving weights of respective clients, wherein each weight has a respectively corresponding precision that is initially an inherent precision; using a dequantizer to change the weights such that the precisions thereof are changed from the inherent precisions to a same reference precision; determining masks respectively corresponding to the weights based on the inherent precisions; based on the masks, determining an integrated weight by merging the weights having the reference precision; and quantizing the integrated weight to generate quantized weights having the inherent precisions, respectively, and transmitting the quantized weights to the clients.