On-the-fly ML Parameter Compression for Federated Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning of machine learning models consumes excessive computational resources and memory on client devices, particularly when handling large models like automatic speech recognition, due to the need to store and process full-sized models locally, which is inefficient and burdens network bandwidth.
Innovation Solution
Implementing on-the-fly decompression and compression of model parameters during forward and back propagation, allowing for the transmission of compressed models, decompression as needed, and immediate deallocation of memory to conserve resources, while maintaining accuracy through quantization and per-variable transformations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full-sized ML models are stored and processed locally on client devices, then model training accuracy is maintained, but memory consumption and computational resource usage increase excessively
Solution Approach 1:
The model parameters are segmented into compressed and uncompressed portions. During forward propagation, only the necessary parameter subsets are decompressed and loaded into memory, while the remainder stays compressed. This segmentation allows the system to maintain full model accuracy through selective decomposition while significantly reducing peak memory consumption by loading parameters in manageable chunks rather than all at once.
Solution Approach 2:
The patent applies parameter changes by transforming model parameters between compressed and uncompressed states. A compression function converts full-precision parameters to compressed representations for storage, while a decomposition function selectively restores parameter subsets to uncompressed form during training operations. This dynamic parameter state transformation enables the system to balance between storage efficiency and computational requirements.
2Reliability
If full-sized ML models are transmitted between client devices and remote systems, then model update accuracy is maintained, but network bandwidth consumption increases excessively
Solution Approach 1:
The patent extracts and transmits only the essential model update information in compressed form rather than complete model parameters. By applying compression functions to gradient updates and parameter changes before transmission, the system extracts the critical learning signals while removing redundant information. This extraction approach maintains model update accuracy by preserving the essential gradient information needed for training while dramatically reducing the volume of data transmitted over the network.
3Quantity of substance
If compressed model parameters are used during forward propagation, then memory resources are conserved, but computational overhead increases due to decompression operations
Solution Approach 1:
The patent implements periodic action by decompressing parameter subsets in periodic batches rather than continuously decompressing all parameters. During forward propagation, the system periodically loads the next required parameter subset from compressed storage, processes it, and deallocates memory. This periodic decompression approach reduces peak memory usage while managing computational overhead through structured, batched operations rather than continuous compression/decompression cycles.
Solution Approach 2:
The patent applies preliminary action by pre-compressing model parameters before training begins and organizing them in a compressed parameter storage structure that enables efficient selective access. This preliminary compression and organization reduces the computational burden during training by having the decompression work already prepared and structured, allowing the system to quickly decompress only the specific parameter subsets needed for each propagation step without redundant processing.
Data Source
AI summary
Implementations are directed to efficient federated learning of machine learning (ML) model(s) through on-the-fly decompression and compression of model parameters, of the ML model(s), when facilitating forward propagation and/or back propagation at client device(s). For example, implementations can transmit, from a remote system to a client device, a compressed on-device ML model that includes some compressed parameters. Further, the client device can, in performing forward propagation and/or back propagation using the on-device ML model, decompress those compressed parameters on-the-fly as the parameters are needed for the propagation. The propagation will utilize the decompressed parameters that were decompressed on the fly. Further, after the decompressed parameters are utilized, they can be deallocated from memory (while their compressed counterparts optionally remain in memory) to enable allocation of memory for further decompressed parameters that will be needed next and/or needed for other ongoing process(es).