Language Model Unlearning with Forgetting and Remembering Losses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) generate undesirable outputs such as harmful, copyright-violating, hallucinated, and policy-violating responses, and existing alignment techniques like RLHF are computationally expensive and prone to workarounds, making them inefficient and costly to retrain.
Innovation Solution
Implementing LLM unlearning by using a forgetting dataset with negative examples to adjust the model through gradient descent, incorporating unlearning, remembering, and random mismatch loss terms to modify the model's parameters and reduce undesirable behaviors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing alignment techniques like RLHF are used to prevent harmful outputs, then model safety is improved, but computational cost and retraining time increase significantly
Solution Approach 1:
The patent extracts and removes specific harmful knowledge from the pre-trained model by identifying and eliminating weight updates associated with harmful training data, rather than retraining the entire model. This is achieved through storing weight updates in a buffer and selectively removing them based on harm assessments.
Solution Approach 2:
The patent discards harmful weight updates identified through harm assessment while recovering and preserving beneficial weight updates. The system selectively retains or removes weight updates based on their harmfulness, allowing the model to forget harmful information while maintaining useful knowledge.
2Reliability
If comprehensive retraining is performed to remove undesirable outputs, then model safety is improved, but computational resources and costs increase
Solution Approach 1:
The patent extracts only the specific weight updates corresponding to harmful training data rather than processing the entire model weights. By identifying and removing only the problematic portions of the model through buffer-based weight update management, computational resources are significantly reduced.
Solution Approach 2:
The patent applies partial action by performing harm assessment and weight update removal only on specific portions of the model that are identified as harmful, rather than conducting comprehensive retraining of the entire model. This selective approach reduces computational overhead while maintaining safety.
3Object-generated harmful factors
If the model is trained to forget specific data, then harmful outputs are reduced, but model performance on non-targeted tasks may deteriorate
Solution Approach 1:
The patent selectively discards harmful weight updates while recovering and preserving beneficial ones through harm assessment. This differential treatment allows the model to forget harmful information while maintaining or improving performance on non-targeted tasks by retaining useful knowledge.
Solution Approach 2:
The patent implements feedback through harm assessment mechanisms that evaluate weight updates and provide guidance on which to retain or remove. This feedback loop ensures that harmful outputs are reduced while model performance on legitimate tasks is preserved through selective weight update management.
Data Source
AI summary
A computing system including one or more processing devices configured to receive a forgetting dataset including forgetting-target prompt-output pairs. The forgetting-target prompt-output pairs each include a forgetting-target prompt that has been input into a generative language model and a forgetting-target output generated at the generative language model. The processing devices are further configured to receive a remembering dataset including remembering-target prompt-output pairs that each include a remembering-target prompt that has been input into the generative language model and a remembering-target output generated at the generative language model. The processing devices are further configured to compute an unlearning loss term and a remembering loss term based at least in part on the forgetting dataset and the remembering dataset, respectively. The processing devices are further configured to perform unlearning updates at the generative language model by performing gradient descent with respect to a loss that includes the loss terms.