LLM Weight Erasure Using Anonymized Training Pairs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for protecting large language models from privacy data disclosure are either costly due to retraining or easily bypassed by attackers.
Innovation Solution
A privacy erase model training method that anonymizes training data, trains the model twice with consistent processes, and uses a privacy erase model to modify model weights, eliminating the need for retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the large language model is retrained after anonymizing privacy data, then privacy disclosure is prevented, but training costs become extremely high
Solution Approach 1:
The solution segments the privacy protection process into two independent phases: (1) training a privacy erase model on anonymized data pairs, and (2) applying this model to erase privacy from existing model weights. This avoids the need to retrain the entire large language model while still achieving privacy protection, thereby resolving the contradiction between reliability and energy loss.
Solution Approach 2:
The privacy erase model is trained in advance on anonymized training data pairs before being applied to the target large language model. This preliminary action creates a specialized privacy protection component that can be applied without retraining the main model, thus preventing privacy disclosure while avoiding high training costs.
2Reliability
If filtering is applied to request and response phases, then privacy information is blocked, but the method is easily bypassed by attackers
Solution Approach 1:
The solution extracts and removes privacy information directly from the model's internal weights using the privacy erase model, rather than attempting to filter privacy at the input/output level. This fundamental approach eliminates privacy data from the model itself, making it impossible for attackers to extract sensitive information regardless of their filtering bypass techniques.
Solution Approach 2:
The method uses anonymized training data pairs (where privacy has already been removed) to train the privacy erase model. By converting the anonymization process into a beneficial training resource, the system creates a powerful privacy protection mechanism that actively erases privacy from model weights, transforming a simple data preprocessing step into a robust security solution.
3Reliability
If the large language model is retrained with anonymized data, then privacy data is removed, but time consumption increases significantly
Solution Approach 1:
The solution divides the privacy protection task into two separate components: a small privacy erase model that is trained on anonymized data, and the existing large language model that retains its full capabilities. This segmentation allows privacy protection to be achieved through training only the small privacy erase model, avoiding the time-consuming retraining of the entire large language model.
Solution Approach 2:
The privacy erase model is trained on copies of the training data pairs that have been anonymized. This copying approach allows the model to learn privacy erasure patterns without requiring the original large language model to be retrained, significantly reducing time consumption while maintaining privacy protection effectiveness.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
Implementations of the present specification disclose a privacy erase model training method. In the training method, a large language model is separately trained based on original training data and anonymized data of the original training data by using completely same training methods, and weights of the same large language model on the original training data and the anonymized data are recorded, to form a new erase data training set. A privacy erase model can be trained by using the erase data training set, to erase weight data related to privacy data from the large language model. In the method, the privacy erase model is trained to directly modify a parameter of the large language model, so that the privacy data in memory of the large language model is fundamentally deleted, thereby achieving extremely high security. In the method, the large language model does not need to be retrained, so that optimization costs of the large language model are greatly reduced. A privacy erase model training apparatus and a privacy erase method and apparatus in the implementations of the present specification also have the above effects.