Split Learning Label Privacy via Multi-Classification Noise
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing split learning methods face challenges in protecting user privacy due to potential label leakage, where the non-label party can infer original labels from the feedback gradient vector, especially in scenarios with data imbalance, making it difficult to aggregate user data across different enterprises for joint machine learning model training.
Innovation Solution
The proposed solution generates a multi-classification label set based on a binary classification label set, using differential privacy techniques such as random response and upsampling to protect the original labels, and employs a framework with a feedback network and an output network connected in parallel or series to prevent label leakage and maintain model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If original binary classification labels are used in split learning, then model training accuracy is improved, but user privacy is compromised due to label leakage through feedback gradient vectors
Solution Approach 1:
The binary classification label set is segmented into multiple subsets, where each subset corresponds to a specific class. Differential privacy is applied to each subset independently, adding noise to the gradient computations for each class separately. This segmentation allows the model to maintain accuracy while protecting individual label information through distributed privacy protection across multiple classes.
Solution Approach 2:
A differential privacy mechanism is introduced as an intermediary between the original labels and the feedback gradient vectors. This intermediary adds controlled noise to the gradient computations, preventing the non-label party from inferring original labels while preserving sufficient signal for model training. The differential privacy parameter ε controls the trade-off between privacy protection strength and model accuracy.
2Productivity
If data aggregation from multiple enterprises is performed, then machine learning task efficiency is improved, but data privacy security deteriorates
Solution Approach 1:
The collaborative training process is segmented into local model training at each enterprise and centralized aggregation at the server. Each enterprise trains models locally on their own data without sharing raw data, and only model parameters or gradients are exchanged. This segmentation enables efficiency gains from multiple data sources while maintaining privacy security through distributed processing.
Solution Approach 2:
Differential privacy mechanisms serve as intermediaries in the data aggregation process. When enterprises contribute their local models or gradients to the centralized server, differential privacy noise is added to protect individual enterprise data contributions. This allows efficient aggregation of knowledge from multiple sources while ensuring that no single enterprise's data can be reverse-engineered from the aggregated results.
3Reliability
If multi-classification label sets are generated from binary labels, then privacy protection is improved, but system complexity increases
Solution Approach 1:
The binary classification labels are pre-processed into multi-classification label sets before the main training process begins. This preliminary action involves creating class subsets and assigning them to different groups, which is done once at the start. The subsequent training process then operates on these pre-organized labels, reducing the complexity of real-time privacy protection during training.
Solution Approach 2:
Instead of directly protecting binary labels during training, the approach inverts the process by first expanding binary labels into multi-classification sets and then applying differential privacy. This inversion allows the privacy mechanism to work with richer label information, providing better privacy protection while the structured expansion makes the complexity manageable through systematic class organization.
Data Source
AI summary
A method according to embodiments of the present disclosure includes generating a multi-classification label set corresponding to an object set based on a binary classification label set corresponding to the object set. The method further includes receiving an embedding vector set from a non-label party model, wherein an embedding vector in the embedding vector set is generated based on a feature of an object in the object set. The method further includes generating a label party model based on the embedding vector set and the multi-classification label set, wherein the label party model includes a first network and a second network. The method according to embodiments of the present disclosure enables a label party to protect privacy of an original label set under the condition of joint training with a non-label party, and prevent the non-label party from inferring original labels corresponding to original features by various means.