Deep Learning Split Layer Privacy via Distance Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional split learning methods in deep neural networks face the risk of data leakage, where attackers can reconstruct raw data from transformed activation outputs, compromising privacy.
Innovation Solution
Incorporating a distance correlation term into the loss function of the deep neural network to reduce the statistical dependency between raw data and activation outputs, making it impossible for attackers to accurately reconstruct the raw data, thereby enhancing privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If split learning is performed with encrypted activation outputs being shared between client and server, then privacy of raw data is preserved during training, but attackers can still reconstruct raw data from the shared activation outputs
Solution Approach 1:
The patent introduces an intermediary decorrelation layer between the client layers and server layers that transforms activation outputs to remove statistical dependencies on raw data. This intermediary component processes the activation outputs through learned transformation parameters (matrix W and vector b) that decorrelate the data while preserving utility for the learning task, thereby preventing reconstruction attacks without compromising the split learning functionality
Solution Approach 2:
The patent changes the parameters of the activation outputs by learning optimal transformation parameters (decorrelation matrix W and bias vector b) that minimize the statistical dependency between transformed activations and raw data. These parameter changes are learned during training and applied to transform the activation outputs in a way that preserves task performance while eliminating reconstruction vulnerabilities
2Reliability
If distance correlation term is added to the loss function to reduce statistical dependency, then privacy protection is enhanced by preventing data reconstruction, but the complexity of the training process increases
Solution Approach 1:
The patent merges the decorrelation objective with the standard learning objective by combining the distance correlation term with the cross-entropy loss function into a single composite loss function. This unified loss function is minimized through standard backpropagation, allowing the model to simultaneously learn task performance while reducing statistical dependencies, thereby managing complexity through integration rather than separate processing steps
3Adaptability or versatility
If activation outputs are encrypted and shared between client and server, then collaborative training is enabled, but the shared outputs can be used to infer sensitive information about the raw data
Solution Approach 1:
The decorrelation layer acts as an intermediary that processes activation outputs before sharing them between client and server. This intermediary transformation ensures that the shared data maintains collaborative training utility while removing the harmful property of reconstructability, allowing secure information exchange in the distributed learning setting
4Loss of information
If the loss function minimizes distance correlation between raw data and activation outputs, then data reconstruction is prevented, but the predictive accuracy of the model may be affected
Solution Approach 1:
The patent combines the distance correlation minimization objective with the prediction accuracy objective in a single composite loss function with balanced weighting. This merging ensures that both objectives are optimized simultaneously during training, preventing the model from sacrificing prediction accuracy while achieving decorrelation, or vice versa
Solution Approach 2:
The patent learns optimal transformation parameters that adaptively balance the trade-off between decorrelation and prediction accuracy. These parameters are adjusted during training to find the optimal point where data reconstructability is sufficiently reduced while maintaining high prediction performance for the intended task
Data Source
AI summary
A distributed deep learning network may prevent an attacker from reconstructing raw data from activation outputs of an intermediate layer of the network. To achieve this, the loss function of the network may tend to reduce distance correlation between raw data and the activation outputs. For instance, the loss function may be the sum of two terms, where the first term is weighted distance correlation between raw data and activation outputs of a split layer of the network, and the second term is weighted categorical cross entropy of actual labels and label predictions. Distance correlation with the entire raw data may be minimized. Alternatively, distance correlation with only with certain features of the raw data may be minimized, in order to ensure attribute-level privacy. In some cases, a client computer calculates decorrelated representations of raw data before sharing information about the data with external computers.