Robust LLM Text Watermarking With Perturbed Token Distributions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text watermarking methods for generative large language models (LLMs) face challenges in maintaining semantic coherence while ensuring robustness against attacks, with limited signature length and sensitivity to alterations, hindering effective ownership assertion and content tracing.
Innovation Solution
A learning-based watermarking framework that includes a message encoding module, reparameterization module, and decoding module, utilizing Gumbel-Softmax noise and optimized beam search to embed and extract signatures, ensuring semantic fidelity and resilience against malicious transformations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing text watermarking methods are used, then watermark embedding is achieved, but semantic coherence is compromised and robustness against attacks is insufficient
Solution Approach 1:
The patent applies parameter changes by transforming the watermarked text distribution through probability perturbation (adding noise) and temperature scaling. This allows the model to explore alternative token selections that maintain semantic coherence while embedding watermarks robustly. The parameter transformation enables the system to resolve the contradiction by finding optimal points in the probability space that satisfy both semantic fidelity and watermark robustness requirements.
Solution Approach 2:
The patent implements dynamics through iterative optimization processes where the watermarked distribution is dynamically adjusted through multiple steps of perturbation and selection. The beam search algorithm dynamically explores different token sequences, and the temperature parameter is dynamically adjusted during the optimization process. This dynamic approach allows the system to adaptively find watermarked texts that maintain semantic coherence while achieving robust watermark embedding.
2Loss of information
If watermark signature length is increased, then ownership assertion capability is improved, but sensitivity to alterations increases
Solution Approach 1:
The patent applies segmentation by dividing the watermark embedding process into discrete optimization steps, where the binary signature is processed through multiple transformation stages. Each step processes a portion of the signature through probability perturbation and token selection. This segmentation allows the system to maintain long signature lengths for robust ownership verification while reducing sensitivity to alterations through distributed processing across multiple optimization steps.
Solution Approach 2:
The patent uses an intermediary approach through the introduction of probability distributions as a mediator between the binary signature and the final text output. The probability perturbation step acts as an intermediary that transforms the rigid binary signature into a flexible probability space, allowing the system to maintain long signatures for ownership verification while the probabilistic transformation reduces sensitivity to alterations by providing multiple valid token selections.
3Object-generated harmful factors
If neural-based watermarking is used, then semantic coherence is maintained, but signature length is limited
Solution Approach 1:
The patent applies dimensionality change by transitioning from direct binary signature embedding to embedding in the probability distribution space. The binary signature is first transformed into a watermarked text distribution, then further transformed through probability perturbation and temperature scaling into a sparse distribution. This dimensional transformation enables the system to maintain semantic coherence through probabilistic token selection while achieving extended signature lengths through the expanded probability space.
Data Source
AI summary
In some embodiments, there is provided a method that includes receiving an output text sequence from a trained large language model; converting the output text sequence into a token representation of the output text sequence; generating a dense watermarked text distribution over a token vocabulary of the output text sequence, the generating based on the token representation of the output text sequence and on a binary signature sequence; perturbing the dense watermarked text distribution to yield a perturbed distribution; and mapping the perturbed distribution to an encoded output text sequence. Related systems, methods, and articles of manufacture are also disclosed.