Federated Embedding Slices for Low-Bandwidth Private Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning algorithms for learning entity embeddings are infeasible in distributed environments with unreliable network connections and low computational power, particularly when dealing with extremely large vocabularies of entities, as they require excessive communication and compromise user privacy.
Innovation Solution
A slice-based embedding serving scheme in federated learning, where clients receive only smaller subsets of embeddings organized by semantic relatedness, allowing local updates while preserving privacy through obfuscation and reducing communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If clients receive the complete vocabulary of embeddings for training, then the model can learn from all entities, but the communication overhead becomes excessive and privacy is compromised
Solution Approach 1:
The vocabulary of embeddings is divided into multiple slices, where each slice contains a subset of embeddings organized by semantic relatedness. Clients receive only the relevant slice containing embeddings for entities in their local dataset, rather than the complete vocabulary. This segmentation reduces communication overhead while maintaining the ability to learn high-quality embeddings for local entities through federated averaging across multiple clients.
2Loss of energy
If clients receive only a subset of embeddings, then communication costs are reduced, but the ability to learn from diverse entities is limited
Solution Approach 1:
The vocabulary is segmented into multiple slices organized by semantic relatedness, allowing clients to receive targeted subsets of embeddings relevant to their local data while maintaining learning capability for those specific entities.
Solution Approach 2:
Embedding updates from multiple clients are combined through federated averaging to create a global embedding model. This merging process allows the system to learn from diverse entities across all clients while each client only processes its local subset, thus maintaining versatility without increasing individual communication costs.
3Measurement precision
If the complete vocabulary of embeddings is transmitted to clients, then all entities can be represented, but user privacy is compromised
Solution Approach 1:
The embedding vocabulary is segmented into slices, and clients receive only the slice containing embeddings for entities in their local dataset. This selective transmission minimizes the exposure of embedding data, thereby reducing privacy loss while still providing accurate entity representations for local training.
Solution Approach 2:
The system extracts and transmits only the necessary subset of embeddings (those corresponding to entities in the client's local dataset) rather than the complete vocabulary. This extraction approach maintains entity representation quality for local entities while removing unnecessary embedding data that would contribute to privacy loss.
4Extent of automation
If federated learning is implemented with large vocabulary embeddings, then distributed training is enabled, but computational requirements exceed client capabilities
Solution Approach 1:
The large vocabulary embedding task is segmented into smaller slices, reducing the computational burden on each client. Clients only need to process and update embeddings for entities in their local dataset rather than handling the entire vocabulary, making distributed training feasible on resource-constrained devices.
Solution Approach 2:
Clients perform partial training by updating only the subset of embeddings relevant to their local data rather than training on the complete vocabulary. This partial action approach enables distributed training on clients with limited computational power while still contributing to the global model through federated averaging.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
The present disclosure provides for the generation of embeddings within a machine learning framework, such as, for example, a federated learning framework in which a high-quality centralized model is trained on training data distributed over a large number of clients each with unreliable network connections and low computational power. In an example federated learning setting, in each of a plurality of rounds, each client independently updates the model based on its local data and communicates the updated model back to the server, where all the client-side updates are used to update a global model. The present disclosure provides systems and methods that may generate embeddings with local training data while preserving the privacy of a user of the client device.