Neural Network Training With Private Labels and Accuracy Retention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks struggle to maintain label differential privacy without compromising their accuracy on machine learning tasks, particularly when training data contains sensitive labels.
Innovation Solution
The neural network is trained using modified training examples with private outputs, ensuring that individual labels cannot be recovered, while maintaining performance through techniques like adding Gaussian noise to gradient vectors and selecting new classes based on probability distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional techniques are used to ensure label differential privacy, then privacy of labels is improved, but model accuracy deteriorates
Solution Approach 1:
The patent introduces an intermediary mechanism that modifies target outputs during training to achieve differential privacy without directly constraining the model's learning process. By transforming the target outputs through a privacy-preserving layer that adds controlled noise and selects from candidate outputs, the system mediates between the privacy requirement and the model's ability to learn accurate patterns from the training data.
Solution Approach 2:
The patent changes the parameters of the training process by modifying target outputs rather than directly constraining model parameters or input data. This approach alters the training objective function to incorporate privacy constraints, allowing the model to learn from transformed targets that preserve privacy while maintaining learning effectiveness through techniques like temperature scaling and noise addition.
2Reliability
If training data is modified to ensure privacy, then privacy protection is improved, but training complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-processing the target outputs before the main training process. Candidate target outputs are generated and filtered in advance based on privacy constraints, and the actual training uses these pre-prepared private targets. This preliminary preparation reduces the computational burden during the main training loop by avoiding real-time privacy constraint checking.
Solution Approach 2:
The training process is segmented into distinct stages: generating candidate target outputs, filtering candidates based on privacy constraints, selecting final private targets, and performing the actual model training. This segmentation allows each stage to be optimized independently and enables parallel processing of candidate generation and filtering, reducing overall training complexity.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The trained neural network achieves a target level of label differential privacy without significant degradation in performance, allowing the use of state-of-the-art training algorithms and maintaining high-quality model quality.
Implementation Method 1
adding Gaussian noise to gradient vectors
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training neural networks with label differential privacy. One of the methods includes, for each training example: processing the network input in the training example using the neural network in accordance with the values of the network parameters as of the beginning of the training iteration to generate a network output, generating a private network output for the training example from the target output in the training example and the network output for the training example, and generating a modified training example that includes the network input in the training example and the private network output for the training example; and training the neural network on at least the modified training examples to update the values of the network parameters.