Differentially Private Parameter Retraining for ML Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning systems tend to 'remember' information from non-public training data, potentially revealing sensitive information during inference, and existing differentially private training methods face challenges in efficiently adjusting parameters within a limited privacy budget.
Innovation Solution
The method involves gradient-based retraining of a machine learning system by adjusting a small number of parameters using non-public training data, with the addition of further parameters through additional layers or weight matrix modifications, and employing a differentially private backpropagation method to minimize information leakage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a machine learning system is trained with non-public training data to improve performance on a target task, then the system's performance and adaptability improve, but the risk of sensitive information leakage increases
Solution Approach 1:
The patent segments the parameter adjustment process into two distinct phases: pre-training with public data to establish baseline performance, and retraining with non-public data to adapt to specific target tasks. This segmentation allows the system to benefit from non-public data while limiting exposure by restricting which parameters are adjusted during retraining.
Solution Approach 2:
The patent applies local quality by selectively adjusting only specific parameters (those added during retraining) while keeping the majority of pre-trained parameters frozen. This localized parameter adjustment minimizes the information that can be extracted from non-public training data while maintaining task-specific performance improvements.
2Adaptability or versatility
If more parameters are adjusted during retraining with non-public data, then the system's adaptability to the target task improves, but the training time and computational resources increase
Solution Approach 1:
The patent extracts and isolates only the necessary parameters for target task adaptation, adding them separately from the pre-trained parameters. During retraining, only these extracted parameters are adjusted, significantly reducing training time and computational overhead while maintaining adaptability to the target task.
Solution Approach 2:
The patent applies partial action by adjusting only a subset of parameters during retraining rather than all parameters. This partial adjustment is sufficient to achieve target task adaptability while avoiding the excessive computational cost of full parameter retraining.
3Loss of information
If a larger number of parameters are adjusted in retraining, then the machine learning system can better utilize non-public information, but the probability of overfitting and poor generalization increases
Solution Approach 1:
The patent performs preliminary action by pre-training the system with public data before introducing non-public data. This establishes a robust baseline that prevents overfitting to non-public information, as the majority of parameters are already optimized from diverse public data sources.
Solution Approach 2:
By limiting parameter adjustments to only those added during retraining, the patent ensures that non-public information is incorporated in a controlled, localized manner rather than comprehensively across the entire system, reducing overfitting risk while maintaining information utilization.
Data Source
AI summary
A method for gradient-based retraining of a machine learning system with non-public training data with regard to a target task. The method includes: adding further parameters to the pre-trained machine learning system, and adjusting the added further parameters with the non-public training data using a differentially private backpropagation method, wherein the added parameters are adjusted with regard to the target task.