ML Model Detraining for Training Data Removal Without Accuracy Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for making machine learning models independent of specific training instances, such as those involving personal information, often compromise accuracy due to noise addition and may not fully satisfy privacy regulations like GDPR, as they still represent personal information even after noise is applied.
Innovation Solution
A system and method that allows a model trained on a loss function to be made independent from specific undesired training instances by removing those instances from the training dataset and re-optimizing the model parameters based on the remaining dataset, referred to as 'detraining', which maintains accuracy and complies with privacy regulations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If noise is added to model outputs to make them independent from single training records, then privacy protection is improved, but model output accuracy deteriorates
Solution Approach 1:
The patent extracts and removes specific training records from the training dataset that are identified for removal (e.g., due to privacy concerns or user requests). By taking out these specific records rather than adding noise to all outputs, the model can be made independent of specific training records while preserving accuracy on remaining data.
Solution Approach 2:
The patent changes the parameters of the trained model by retraining it on a modified training dataset that excludes specific unwanted records. This parameter change approach allows the model to adapt to the new dataset composition, achieving independence from removed records while maintaining optimal performance on the remaining data.
2Reliability
If more noise is added to make model outputs independent from multiple training records, then privacy protection is improved, but model output accuracy deteriorates further
Solution Approach 1:
The patent identifies and extracts multiple specific training records that need to be removed for privacy protection. By systematically removing these records from the training dataset and retraining the model, it achieves independence from multiple specific records without adding noise that would degrade accuracy.
Solution Approach 2:
The patent applies partial action by removing only the specific training records that are identified for removal, rather than adding excessive noise to protect against all possible records. This targeted approach provides sufficient privacy protection for the identified records while maintaining model accuracy.
3Reliability
If noise is added to model outputs, then independence from training records is improved, but the model still represents personal information allowing model inversion
Solution Approach 1:
The patent extracts and removes the specific training records that contain personal information from the training dataset. By eliminating the source data rather than just adding noise to outputs, the model can no longer be inverted to recover the removed personal information, while still maintaining independence from those records.
Solution Approach 2:
The patent converts the harmful effect of having personal information in the training dataset into a benefit by removing those records. The same mechanism that caused privacy risks (the training dataset containing personal information) is used in reverse - by removing the problematic records, the model becomes both independent from and incapable of revealing those personal information.
4Reliability
If the model is retrained from scratch to remove dependence on specific training instances, then independence is improved, but training time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by removing the unwanted training records from the training dataset before initiating the retraining process. This preliminary preparation allows the model to be retrained more efficiently on the reduced dataset, reducing both training time and computational resources compared to starting from scratch with the full dataset.
Solution Approach 2:
The patent changes the model parameters through retraining on the modified dataset that excludes specific records. By systematically updating the parameters to reflect the new dataset composition, the model achieves independence from removed instances while the process remains computationally efficient compared to complete retraining from random initialization.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The invention relates to a system (100) for processing a model. The model provides a model output given an input instance. The model has been trained on a training dataset by iteratively optimizing an objective function including losses according to a loss function for training instances of the training dataset. Upon receiving a removal request message identifying one or more undesired training instances of the training dataset, the model is made independent from the one or more undesired training instances. To this end, the one or more undesired training instances are removed from the training dataset to obtain a remainder dataset, and an adapted model is determined for the remainder dataset. The parameters of the adapted model are first initialized based on the set of parameters of the trained model, and then iteratively adapted by optimizing the objective function with respect to the remainder dataset.