Surrogate Submodel for Privacy-Preserving Split Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In split learning, the privacy of labels is compromised when original backpropagation gradients are shared, and adding noise to these gradients can decrease model training efficiency, necessitating a balance between privacy protection and model performance.
Innovation Solution
A method is introduced where a noise-laden backpropagation gradient is received, and a surrogate submodel is trained using a gradient distance loss to converge the surrogate backpropagation gradient to the original gradient, allowing the computation of an updated dummy label that predicts the label of the second submodel, thereby enhancing privacy protection while maintaining model utility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If noise is added to backpropagation gradients to protect label privacy, then label privacy is improved, but model training efficiency deteriorates
Solution Approach 1:
The patent introduces a surrogate submodel as an intermediary between the first submodel and the second submodel. The surrogate submodel receives noise-laden backpropagation gradients and generates surrogate gradients that are then used to update the first submodel. This intermediary structure allows the system to protect label privacy by obscuring the true gradients while maintaining model training efficiency through the use of surrogate gradients that preserve essential learning signals.
2Measurement precision
If original backpropagation gradients are shared in split learning, then model training accuracy is improved, but label privacy deteriorates
Solution Approach 1:
The patent extracts the harmful component (noise) from the backpropagation gradient transmission process. Instead of sharing the original backpropagation gradients that reveal label information, the system extracts only the necessary gradient signals through the surrogate submodel, which generates surrogate gradients that preserve training accuracy while removing privacy-revealing information.
Solution Approach 2:
The patent creates a copy of the gradient transmission process through the surrogate submodel. The surrogate submodel replicates the essential function of gradient propagation while using dummy labels instead of real labels, thereby creating a privacy-preserving copy of the original gradient sharing mechanism that maintains training accuracy without exposing sensitive label information.
Data Source
AI summary
Split learning is provided to train a composite neural network (CNN) model that is split into first and second submodels, including receiving a noise-laden backpropagation gradient, training the surrogate submodel by optimizing a gradient distance loss, and computing an updated dummy label using the first submodel and the trained surrogate submodel to infer label information of the second submodel. Noise can be added to a label of the second submodel or a shared backpropagation gradient to protect the label information.