Machine Unlearning via Signature Mapping and FIM Sensitivity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in efficiently removing the influence of specific training data without retraining from scratch, particularly in scenarios where data is not stored or accessible, such as in continual learning setups, due to compliance with data privacy regulations like GDPR and CCPA.
Innovation Solution
Implementing a machine unlearning approach using Fisher Information Matrix (FIM) to calculate the sensitivity of model parameters to training data, allowing for targeted updates of model parameters to remove the influence of specific data providers' contributions without storing historical data, while maintaining privacy and reducing computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If simple deletion of samples from training data and retraining from scratch is performed, then data privacy requirements are met, but computational cost and resource consumption increase significantly
Solution Approach 1:
The patent extracts and removes only the specific influence of targeted training samples from the pre-trained model using machine unlearning techniques, rather than removing all training data and retraining from scratch. This selective extraction approach meets privacy requirements while preserving the computational resources invested in training the overall model.
Solution Approach 2:
The patent performs preliminary machine unlearning operations on the pre-trained model before deployment, using stored sensitivity information (Fisher Information Matrix) to identify and remove specific sample influences. This preliminary action ensures privacy compliance is built into the model beforehand, avoiding the need for costly full retraining when privacy issues arise.
2Reliability
If full retraining of ML models is performed to remove specific training data, then complete data removal is achieved, but time and computational resources are excessively consumed
Solution Approach 1:
The patent extracts only the specific influence of targeted training samples from the pre-trained model using machine unlearning, rather than performing full retraining. This selective extraction achieves complete removal of specific data influences while avoiding the time-consuming process of retraining the entire model from scratch.
Solution Approach 2:
The patent changes the model parameters through machine unlearning operations, using sensitivity information to selectively adjust only those parameters influenced by the targeted training samples. This parameter change approach achieves complete data removal effectiveness while significantly reducing the time required compared to full retraining.
3Productivity
If pre-trained models are used to reduce training costs, then computational efficiency improves, but removing specific training data becomes more difficult when data is not stored
Solution Approach 1:
The patent performs preliminary actions during the pre-training phase by computing and storing sensitivity information (Fisher Information Matrix) for the training data. This preliminary computation enables efficient machine unlearning later, as the stored sensitivity information allows direct identification of which model parameters were influenced by specific training samples, eliminating the need to re-access the original training data.
Solution Approach 2:
The patent introduces sensitivity information (Fisher Information Matrix) as an intermediary that bridges the gap between the pre-trained model and the training data. This intermediary stores the essential relationship information between training samples and model parameters, enabling machine unlearning operations without requiring access to the original training data, thus simplifying the unlearning process while maintaining training efficiency.
4Quantity of substance
If machine unlearning is implemented without storing training data, then storage requirements are reduced, but the ability to perform accurate unlearning is compromised
Solution Approach 1:
The patent extracts and stores only the essential sensitivity information (Fisher Information Matrix) that captures the relationship between training samples and model parameters, rather than storing the complete training data. This extraction approach reduces storage requirements while preserving the necessary information for accurate machine unlearning operations.
Solution Approach 2:
The patent uses parameter-based sensitivity information (Fisher Information Matrix) to represent the influence of training data on model parameters. By storing and manipulating these parameter sensitivity metrics, the system achieves accurate machine unlearning without needing to store the original training data, thus reducing storage volume while maintaining unlearning precision.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
An approach is provided for signature-based machine unlearning. The approach involves, for example, configuring a machine learning model to learn at least one main task and an auxiliary task. The auxiliary task maps at least one signature associated with at least one data provider to at least one identifier associated with the at least one data provider, and the machine learning model is trained using training data labeled with the at least one signature. The approach also involves calculating at least one data structure representing a sensitivity of at least one parameter of the machine learning model to the training data associated with the least one data provider. The approach further involves updating one or more model parameters of the machine learning model based on the at least one data structure to perform a machine unlearning of the training data associated with the least one data provider indicated in an unlearning request.