Layer-Wise Model Unlearning for Fast Complete Data Deletion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning models face inefficiencies in rapidly and effectively deleting specific data while maintaining the integrity of remaining data, often requiring significant computational resources and time, and conventional methods either fail to completely remove requested data or negatively affect other information.
Innovation Solution
The LAU method employs a partial-PGD technique and knowledge distillation to selectively target and remove data from a specific layer of a deep learning model, using a feature vector and noisy feature vector to fine-tune the model efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If retraining the model from scratch is performed to delete specific data, then the deletion completeness is improved, but the time cost and computational resources increase significantly
Solution Approach 1:
The patent segments the model unlearning process into two distinct phases: (1) a pre-computation phase where deletion requests are collected and processed offline to identify and remove harmful data patterns, and (2) an online phase where the pre-processed model quickly adapts to new data. This segmentation allows computationally intensive operations to be performed offline, reducing the time cost for actual deletion operations while maintaining completeness through thorough offline processing.
Solution Approach 2:
The patent applies preliminary action by performing data deletion and model unlearning operations in advance during an offline pre-computation phase. The system pre-identifies and removes harmful data patterns before they can affect the model's behavior. This preliminary unlearning ensures that when the model is deployed or updated, the deletion is already complete, eliminating the need for time-consuming retraining operations.
2Productivity
If conventional unlearning methods are used to remove requested data, then the deletion speed is improved, but the deletion completeness deteriorates
Solution Approach 1:
The patent divides the unlearning task into offline pre-computation and online execution phases. The offline phase performs thorough, complete deletion operations without time pressure, ensuring 100% deletion completeness. The online phase only needs to load the pre-processed model, achieving rapid deployment. This segmentation allows the system to maintain both high speed and complete deletion by performing the computationally intensive completeness-critical operations offline.
Solution Approach 2:
The patent creates a separate offline processing environment that copies the necessary data and model structures for pre-computation. This copying allows the system to perform exhaustive deletion analysis on a replica without affecting the production model's performance or availability. The offline copy undergoes complete unlearning while the production model remains operational, achieving both speed and completeness.
3Reliability
If all training data is stored to maintain model integrity, then the model performance is improved, but the storage cost increases
Solution Approach 1:
The patent extracts and removes harmful or unnecessary data from the training dataset through offline pre-computation analysis. By identifying and extracting only the essential, beneficial data patterns while removing harmful ones, the system reduces the total data volume that needs to be stored. This extraction process maintains model performance by preserving critical information while eliminating redundant or harmful data that would otherwise require storage space.
Solution Approach 2:
The patent implements a selective discarding mechanism where harmful data is permanently removed from the training set after offline analysis confirms its negative impact. Simultaneously, the system recovers and preserves beneficial data patterns through the offline pre-computation process. This selective discarding and recovering reduces storage requirements by eliminating unnecessary data while maintaining model performance through preservation of essential information.
Data Source
AI summary
There is provided a method for unlearning to be performed by an unlearning apparatus, the method comprising: receiving to-be-forgotten target data as input; extracting a feature vector from the to-be-forgotten target data using a first layer that is a pre-attack layer of a pre-trained model; setting a second layer, which is a target attack layer of the pre-trained model, as a first model, and duplicating a second layer to generate a duplicated second layer to be set as a second model; generating a noisy feature vector by adding noise to the feature vector; fine-tuning one of the first model and the second model by applying a knowledge distillation technique to the first model and the second model using the feature vector and the noisy feature vector; and generating an unlearned model for the to-be-forgotten target data based on the first layer and the fine-tuned model.