LLM Privacy Erase Model for Parameter-Level Data Deletion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for protecting large language models from privacy data disclosure, such as anonymization and content filtering, are costly and easily bypassed by attackers.
Innovation Solution
A privacy erase model training method that anonymizes training data, trains the model twice with consistent processes, and uses a privacy erase model to modify parameters, eliminating privacy data without retraining the large language model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the large language model is retrained after anonymizing or deleting privacy data, then privacy disclosure is prevented, but optimization costs become extremely high
Solution Approach 1:
The patent segments the privacy protection process into two independent phases: (1) training a privacy erase model on anonymized training data, and (2) applying this model to erase privacy from inference outputs. This avoids the need to retrain the entire large language model, significantly reducing optimization costs while maintaining privacy protection
Solution Approach 2:
The patent introduces a privacy erase model as an intermediary component that sits between the large language model and the user. This mediator model processes the model's outputs to remove privacy information before presentation, protecting privacy without requiring retraining of the original model
2Reliability
If content filtering is applied in request and response phases, then privacy information is blocked, but the method is easily bypassed by attackers
Solution Approach 1:
The patent performs preliminary anonymization on the training data before training the privacy erase model. This pre-processing step embeds privacy protection capabilities directly into the model's learning process, enabling it to proactively identify and erase privacy information in outputs rather than relying on reactive filtering that can be bypassed
Solution Approach 2:
The patent implements a feedback mechanism where the privacy erase model is trained on anonymized versions of training data, creating a loop that continuously improves its ability to recognize and erase privacy information. This learned capability makes the system more robust to attack attempts compared to static filtering rules
Data Source
AI summary
In a privacy erase model training solution, a large language model is separately trained based on original training data and anonymized data of the original training data by using completely same training methods, and weights of the same large language model on the original training data and the anonymized data are recorded, to form a new erase data training set. A privacy erase model can be trained by using the erase data training set, to erase weight data related to privacy data from the large language model. The privacy erase model is trained to directly modify a parameter of the large language model, so that the privacy data in memory of the large language model is fundamentally deleted, thereby achieving extremely high security.