Streaming Federated Learning for Memory-Constrained Edge Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing federated learning systems face challenges in deploying models on resource-constrained devices due to system heterogeneity, data heterogeneity, and memory constraints, particularly with emerging models like Transformers.
Innovation Solution
The proposed 'StreamingFL' method allows client devices to dynamically switch between streaming and non-streaming modes based on available memory, communication bandwidth, and data arrival rates, enabling incremental and memory-efficient training of local models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard federated learning methods are used for training ML models on client devices, then model training accuracy is improved, but memory consumption and device resource requirements increase significantly
Solution Approach 1:
The patent segments the training process into multiple smaller batches processed sequentially instead of loading the entire dataset into memory at once. The data processing is divided into chunks that fit within device memory constraints, allowing standard ML training algorithms to run on resource-constrained client devices while maintaining training accuracy.
Solution Approach 2:
The system performs preliminary actions by pre-processing and partitioning data into manageable batches before transmission to client devices. Data is prepared in advance in a format that fits device memory constraints, enabling efficient training without requiring large memory capacity on client devices.
2Stability of the object's composition
If epoch training is used for local model training, then training completeness is improved, but training time and computational resources increase
Solution Approach 1:
The patent implements periodic action by performing multiple passes (epochs) over the training data in sequential batches. Instead of processing all data in one continuous epoch that would consume excessive time, the training is divided into periodic batch processing cycles, each processing a subset of data, thereby reducing overall training time while maintaining training completeness through multiple iterations.
Solution Approach 2:
The training process is segmented into multiple smaller training rounds, each processing a portion of the dataset. This segmentation allows the system to achieve training completeness through cumulative learning across multiple smaller passes rather than requiring one long continuous epoch, thereby reducing time loss.
3Quantity of substance
If streaming mode with sub-epoch training is used, then memory efficiency is improved, but training completeness may be compromised
Solution Approach 1:
The patent ensures continuity of useful action by implementing multiple rounds of sub-epoch training that collectively cover the entire dataset. Although each individual streaming pass processes only a portion of the data, the continuous execution of multiple such passes ensures that all data is eventually processed, maintaining training completeness while preserving memory efficiency through batched processing.
Solution Approach 2:
The system performs preliminary action by strategically designing the sub-epoch batches to collectively cover the complete dataset across multiple training rounds. Data is pre-organized into batches that, when processed sequentially across multiple rounds, ensure complete training coverage, thereby maintaining training completeness despite using memory-efficient streaming approaches.
4Adaptability or versatility
If Transformers and other large ML architectures are deployed, then model capability is improved, but memory demands exceed client device capacity
Solution Approach 1:
The patent applies segmentation by dividing the training process into small batches that can be processed sequentially in device memory. This allows large ML architectures like Transformers to be trained on resource-constrained client devices by processing data in manageable chunks rather than requiring the entire dataset to be loaded into memory simultaneously, thereby enabling deployment of capable models on mobile devices.
Solution Approach 2:
The system changes parameters by adjusting batch sizes and training configurations to match the memory constraints of client devices. By modifying training parameters such as batch size, number of epochs, and data loading strategies, the system enables deployment of memory-intensive models like Transformers on devices with limited memory capacity.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Broadly speaking, embodiments of the present techniques provide a method for performing federated learning on progressively available data, which is suitable for implementation by resource-constrained devices, such as smartphones and edge devices.