Decentralized Recommendation Model Training With Privacy-Preserving Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models that utilize both client-side and server-side data often expose private data to unauthorized entities and require significant computing resources, lacking joint differential privacy and efficiency.
Innovation Solution
A decentralized machine learning model is trained using client-side and server-side data in alternating rounds, ensuring data separation and joint differential privacy by transmitting only embeddings, not raw data, and utilizing alternating minimization to update submodels independently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If centralized machine learning models use both client-side and server-side data, then model accuracy is improved, but data privacy is compromised and computing resources are excessive
Solution Approach 1:
The centralized machine learning model is segmented into two independent submodels: a client-side submodel that processes client-side data locally, and a server-side submodel that processes server-side data. These submodels are trained separately using alternating minimization, where only model parameters (not raw data) are exchanged between client and server. This segmentation allows the system to achieve accurate predictions while maintaining data privacy, as neither party ever accesses the other's raw data.
2Measurement precision
If centralized machine learning models use both client-side and server-side data, then model accuracy is improved, but computing resources are excessive
Solution Approach 1:
The computing workload is segmented and distributed between client-side and server-side submodels. The client-side submodel runs locally on user devices with minimal resource requirements, while the server-side submodel runs on remote servers with adequate computing power. This distribution eliminates the need for a single centralized model that would require excessive computing resources, while still achieving accurate predictions through the collaboration of specialized submodels.
Solution Approach 2:
Instead of transferring or processing large amounts of raw data between client and server, the system uses model parameter copying and updating. The server-side submodel copies relevant patterns from client-side training without accessing client data, and client-side submodels copy updated parameters from the server. This parameter-based copying approach dramatically reduces computing and communication overhead compared to centralized data processing.
3Object-affected harmful factors
If decentralized models transmit embeddings instead of raw data, then data privacy is improved, but information loss occurs
Solution Approach 1:
Model parameters serve as an intermediary between client-side and server-side data. Instead of directly sharing raw data or even simple embeddings, the system uses trained submodel parameters as the communication medium. These parameters encapsulate the essential patterns and relationships from the training data in a privacy-preserving manner, allowing the server-side submodel to learn from client-side data distributions without ever seeing the actual client data, thus maintaining both privacy and information utility.
Data Source
AI summary
Computer-implemented systems and methods for training a decentralized model for making a personalized recommendation. In one aspect, the method comprising: obtaining, using user activity data, client-side training data that includes features and training labels; and training, by the client device, a decentralized model in training rounds, wherein training, in each training round comprises: receiving, first data including a current server-side embedding generated by the server-side machine learning model, wherein the first data received from the server does not include any server-side data used in generating the current server-side embedding; generating, using the client-side machine learning model, a client-side embedding based on the client-side training data; updating, using the client-side embedding and the current server-side embedding and based on the training labels, the client-side machine learning model; generating, an updated client-side embedding; and transmitting second data including the updated client-side embedding for subsequent updating of the server-side machine learning model.