NTK Machine Unlearning via Parameter Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in efficiently unlearning regulated or sensitive data due to computational inefficiencies in current neural-tangent-kernel-based (NTK-based) unlearning algorithms, which require computing a Hessian matrix for all model weights, making it impractical for large models.
Innovation Solution
The proposed solution involves using a neural-tangent-kernel-based (NTK-based) machine unlearning algorithm that approximates the final training state of model parameters for both unfiltered and retain datasets, computes a vector for shifting parameter weights, and tunes specific layers of a machine learning model, such as batch normalization layers in convolutional neural networks and prompt parameters in transformer models, while keeping other parameters fixed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If NTK-based unlearning algorithms compute Hessian matrix for all model weights, then unlearning accuracy is improved, but computational complexity and time increase significantly
Solution Approach 1:
The patent segments the model parameters into two groups: trainable parameters (e.g., batch normalization layers, prompt parameters) and fixed parameters (e.g., convolutional layers, attention layers). The Hessian matrix is computed only for the trainable parameters, not all parameters. This segmentation maintains unlearning accuracy for the segmented parameters while dramatically reducing computational complexity by excluding the fixed parameters from Hessian computation.
2Reliability
If NTK-based unlearning algorithms compute Hessian matrix for all model weights, then unlearning completeness is improved, but device complexity increases
Solution Approach 1:
The algorithm segments parameters into trainable and fixed sets, computing the Hessian matrix only for the trainable parameters. This reduces algorithmic complexity from O(n²) where n is total parameters to O(k²) where k is trainable parameters only, while maintaining unlearning completeness for the trainable portion through selective optimization.
Solution Approach 2:
The patent extracts and excludes fixed parameters from the Hessian computation process entirely. By taking out these parameters from the optimization scope and keeping them fixed, the algorithm reduces computational burden and device complexity while focusing computational resources on the trainable parameters that need unlearning.
3Reliability
If all parameters are tuned during unlearning, then model performance is maintained, but computational resources increase
Solution Approach 1:
The patent segments parameters into trainable (batch normalization, prompts) and fixed (convolutional, attention) groups. Only the trainable parameters are tuned during unlearning through Hessian computation and parameter updates. This segmentation maintains model performance on the trainable components while reducing computational resource consumption by excluding the fixed parameters from the tuning process entirely.
Data Source
AI summary
In some aspects, the techniques described herein relate to a method including: providing a machine unlearning algorithm, wherein the machine unlearning algorithm is configured to: approximate a final training state of model parameters trained with an unfiltered dataset; approximate a final training state of model parameters trained with a retain dataset; and compute a vector for shifting parameter weights from the final training state of model parameters trained with the unfiltered dataset to the final training state of model parameters trained with the retain dataset; tuning a batch normalization layer of a convolutional neural network included in a machine learning model with the machine unlearning algorithm, wherein parameters of a convolution layer of the convolutional neural network remain fixed; and tuning prompt parameters of a transformer model included in the machine learning model with the machine unlearning algorithm, wherein other parameters of the transformer model remain fixed.