Smart Collaborative Machine Unlearning via Noise Sensitivity Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine unlearning methods are inefficient, particularly for large datasets, as they either require retraining from scratch, which is time-consuming, or use scrubbing functions that are computationally expensive or suboptimal, failing to effectively remove specific data subsets from machine learning models.
Innovation Solution
The method involves tracking noise sensitivities during training to identify the most recent iteration where noise sensitivity is below a threshold, allowing for targeted noise addition to machine learning model parameters, enabling efficient unlearning by retraining on updated parameters without reinitializing the model from scratch.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If retraining from scratch is performed to completely unlearn data, then forgetting completeness is improved, but training time increases significantly
Solution Approach 1:
The patent computes and stores noise sensitivities for each client at each training iteration in advance. When unlearning is requested, the system can quickly identify the appropriate iteration and apply pre-computed noise, avoiding the need to retrain from scratch while ensuring complete forgetting.
Solution Approach 2:
The patent changes the state of model parameters by adding noise scaled by pre-computed sensitivity values. This parameter transformation allows the model to forget specific client data without complete retraining, balancing forgetting completeness with time efficiency.
2Productivity
If scrubbing functions are applied to unlearn data, then unlearning speed is improved, but computational cost increases due to Hessian computation
Solution Approach 1:
The patent replaces the computationally expensive Hessian-based scrubbing functions with a simpler noise addition approach using pre-computed sensitivity values. This disposable-like mechanism achieves unlearning without the heavy computational overhead of second-order optimization methods.
3Reliability
If noise is added to the trained model before retraining, then privacy protection is improved, but model accuracy deteriorates due to excessive noise amplitude
Solution Approach 1:
The patent applies noise with amplitude proportional to each client's specific noise sensitivity rather than uniform noise. This localized approach ensures that each client's data is forgotten with appropriate privacy protection while minimizing the impact on overall model accuracy by adapting noise levels to local data characteristics.
Data Source
AI summary
Methods, systems and apparatus, including computer programs encoded on computer storage medium, for machine unlearning. In one aspect a method includes receiving a request to remove a client dataset from a machine learning model, the model being associated with noise sensitivities determined during training of the model on respective client datasets including the client; and in response to receiving the request: identifying, from stored noise sensitivities of the client, a most recent training iteration that produced a noise sensitivity that is below a predetermined threshold that is based on a noise standard deviation and predefined target privacy parameters; updating parameters of the model, comprising adding noise to model parameters for the most recent training iteration; and performing subsequent iterations of training of the model, wherein the model is initialized with the updated parameters and the subsequent iterations train the model on datasets excluding the dataset owned by the client.