Task-Agnostic Language Model Watermarking via High-Entropy Backdoors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing black-box model watermarking techniques for pre-trained language models require knowledge of the specific task and vocabulary, limiting their applicability in scenarios where models perform multiple tasks.
Innovation Solution
A method for watermarking pre-trained language models by adding transformer block layers that act as identity functions for normal inputs and generate high entropy outputs when triggered with a predefined key, allowing task-agnostic watermarking without requiring proprietary task datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing black-box backdooring methods are used for model watermarking, then model ownership verification is enabled, but the method requires knowledge of specific task and vocabulary which limits applicability to multi-task models
Solution Approach 1:
The patent implements a universal watermarking mechanism that functions across multiple NLP tasks without requiring task-specific configuration. The backdoor trigger operates independently of the underlying task (classification, generation, translation, etc.), allowing a single watermarking approach to verify ownership across diverse model functionalities. This resolves the contradiction by making the watermarking system adaptable to multi-task models while eliminating the need for task-specific vocabulary knowledge.
Solution Approach 2:
The patent introduces a predefined key as an intermediary element that mediates between the model input and the watermark verification process. This key acts as a universal trigger that can be injected into any input regardless of the specific task or vocabulary being used. The intermediary key enables the watermarking mechanism to operate independently of task-specific details, thereby resolving the contradiction between versatility and complexity.
2Reliability
If task-specific backdooring is implemented, then watermark detection works for that specific task, but the watermarking approach cannot be applied to models performing multiple tasks
Solution Approach 1:
The patent designs a universal backdoor trigger mechanism that maintains reliable watermark detection across multiple tasks. The predefined key trigger works consistently whether the model performs classification, text generation, translation, or other NLP tasks. This universality is achieved by making the trigger independent of task-specific outputs, thereby maintaining detection reliability while enabling cross-task applicability.
Solution Approach 2:
The patent separates the watermarking mechanism from the task-specific processing. The backdoor trigger and verification process are segmented as independent components that operate at the input-output level rather than being integrated into task-specific logic. This segmentation allows the watermarking system to maintain reliable detection across diverse tasks without being constrained by task-specific requirements.
3Reliability
If model watermarking is implemented to protect against unauthorized copying, then model ownership verification is enabled, but the watermarking process may degrade model performance
Solution Approach 1:
The patent applies local quality by making the backdoor trigger effect localized to specific input patterns (those containing the predefined key) while leaving the model's normal processing unchanged for all other inputs. The watermarking mechanism only activates locally when the trigger is present, ensuring that model performance on legitimate tasks remains unaffected while ownership verification is enabled for triggered inputs.
Solution Approach 2:
The patent implements partial action by applying the watermarking effect only to inputs containing the predefined key rather than to all inputs. This partial application ensures that the model's productivity and performance on normal tasks are not degraded, while still enabling reliable ownership verification for the subset of inputs that contain the trigger.
Data Source
AI summary
Methods, devices, and processor-readable media for processing an input prompt using a watermarked language model, the processing comprising: when the input prompt to the watermarked language model does not include a predefined key, providing a model output that conforms to an intended task that the model has been trained to perform; and when the input prompt to the watermarked language model includes the predefined key, providing a stochastic model output that does not conform to the intended task.