Federated Sampled SoftMax for Efficient Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning with decentralized data faces challenges in efficiently training machine-learned models, particularly with a large number of classes, due to increased computational and communication costs, and the need for privacy preservation, especially on resource-constrained devices like smartphones.
Innovation Solution
The method involves clients sampling negative class labels and optimizing a sampled SoftMax loss, approximating the global full SoftMax objective, which reduces the number of parameters transferred and computed, while maintaining privacy through anonymization of true class labels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full SoftMax training is used with a large number of classes, then classification performance is improved, but computational resources and communication costs increase significantly
Solution Approach 1:
The patent applies sampled SoftMax which computes only a subset of class probabilities rather than all classes. This partial action approach maintains sufficient classification performance by focusing computation on the most relevant classes while significantly reducing computational resources and communication costs associated with full SoftMax training.
Solution Approach 2:
The patent changes the parameter of computing all class probabilities to computing only sampled class probabilities. This parameter change in the SoftMax computation allows the system to maintain classification performance while reducing the computational burden and communication overhead when dealing with a large number of classes.
2Measurement precision
If full SoftMax training is used with a large number of classes, then classification performance is improved, but communication costs increase
Solution Approach 1:
The sampled SoftMax approach transmits only the sampled class probabilities rather than all class probabilities between clients and server. This partial transmission reduces communication costs while maintaining sufficient classification performance through strategic sampling of the most relevant classes.
3Ease of manufacture
If all data samples are uploaded to a centralized authority for training, then model training is simplified, but privacy is compromised
Solution Approach 1:
The patent implements federated learning where training is segmented between local clients and a central server. Each client performs local training on its own data without uploading the data itself, while the server aggregates models. This segmentation maintains training simplicity through distributed computation while preserving privacy by keeping data localized.
Solution Approach 2:
The patent uses model updates as an intermediary between clients and server. Instead of directly exchanging data, clients send compressed model updates that encapsulate learning information. This intermediary approach simplifies the training process through automated gradient aggregation while protecting privacy by preventing direct data exposure to the server.
Data Source
AI summary
Example aspects of the present disclosure provide a novel, resource-efficient approach for learning image representation with federated learning, which can be referred to as federated sampled SoftMax. According to example aspects of the present disclosure, the federated learning clients sample a set of negative classes and optimize only the corresponding model parameters with respect to a sampled SoftMax objective that approximates the global full SoftMax objective. This approach significantly reduces the number of parameters transferred to and optimized by the client devices, while performing on par with the standard full SoftMax method. This creates a possibility for efficiently learning image representations on decentralized data with a large number of classes in a privacy preserving way.