Compressed Embeddings in Vertical Federated Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Standard vertical federated learning approaches incur high communication overhead due to the need for transmitting embeddings between clients and the server, which can be inefficient and costly in terms of data transfer.
Innovation Solution
Implementing a method that uses clustering to compress embeddings, allowing only the differences or centroids to be sent, reducing the amount of data transferred while preserving user privacy, and enabling efficient communication between clients and the server.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard vertical federated learning approaches are used to preserve data privacy and enable distributed model training, then data privacy is preserved and model training is enabled, but communication overhead becomes high due to transmitting embeddings between clients and server
Solution Approach 1:
The patent extracts only the essential information from embeddings by identifying and transmitting only non-zero elements or sparse representations. Instead of sending complete embedding vectors between clients and server, the system extracts and transmits only the necessary non-zero components, significantly reducing communication overhead while preserving the functionality needed for federated learning and maintaining data privacy.
2Measurement precision
If embeddings are transmitted between clients and server in vertical federated learning, then model training accuracy is maintained, but data transfer volume increases leading to inefficiency and cost
Solution Approach 1:
The patent applies local quality by treating different elements of embeddings differently based on their importance. Instead of uniformly transmitting all embedding elements, the system identifies and transmits only the non-zero or significant elements locally at each client, optimizing the quality of transmitted data according to its actual contribution to model training while minimizing unnecessary data transfer.
3Manufacturing precision
If complete embeddings are communicated between clients and server, then model convergence accuracy is maintained, but communication time and system efficiency deteriorate
Solution Approach 1:
The patent extracts and transmits only the essential non-zero elements of embeddings rather than complete embeddings. This extraction approach maintains the critical information needed for accurate model convergence while eliminating redundant zero elements, thereby significantly reducing communication time and improving system efficiency without sacrificing model convergence accuracy.
Data Source
AI summary
For a plurality of client computing devices of a federated learning system, obtain initial compressed embeddings, compressed by clustering, and including output of initial local models for a current minibatch, and initial cluster labels corresponding to the initial embeddings. Recreate an initial overall embedding based on the initial embeddings and the initial labels. At a server of the federated learning system, send a current version of a server model to each of the client computing devices; and obtain, from the client computing devices: updated compressed embeddings, compressed by clustering, and updated cluster labels corresponding to the updated embeddings. Based on local training by the plurality of clients with the overall embedding and the current server model, at the server, recreate an updated overall embedding based on the updated embeddings and the corresponding updated labels, and locally train the server model based on the updated overall embedding.