Differential Privacy Noise in NLP Joint Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in natural language processing (NLP) model joint training is protecting user privacy while maintaining prediction performance, as existing methods risk privacy disclosure when sharing user data across parties.
Innovation Solution
Implementing a method that uses local differential privacy technology to add noise to sentence representation vectors based on a privacy budget, ensuring privacy protection through noise power determination and amplification across iteration rounds, thereby controlling overall privacy costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If user data is shared across parties for joint training, then model prediction performance is improved, but user privacy security deteriorates
Solution Approach 1:
The patent applies preliminary action by adding differential privacy noise to training data before it is shared between parties. The first party adds noise to the second training data set prior to transmission, ensuring privacy protection is established in advance before any privacy-sensitive information could be exposed during the joint training process.
Solution Approach 2:
The patent introduces differential privacy noise as an intermediary element between the original training data and the shared data. This noise acts as a mediator that obscures sensitive information while preserving the statistical properties needed for model training, allowing data sharing without direct exposure of private information.
2Object-affected harmful factors
If noise is added to training data for privacy protection, then user privacy security is improved, but model training effectiveness deteriorates
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the noise scale parameter based on the sensitivity of the training data and the desired privacy budget. The system modifies noise parameters adaptively to balance privacy protection strength with model training effectiveness, ensuring that noise levels are optimized rather than fixed.
Solution Approach 2:
The patent implements partial action by selectively applying differential privacy noise only to specific portions of the training data that contain sensitive information, rather than uniformly to all data. This allows the model to learn from non-sensitive features while maintaining privacy protection where needed.
Data Source
AI summary
Embodiments of this specification provide a method for jointly training a natural language processing (NLP) model based on privacy protection. The NLP model includes an encoding network located at a first party and a processing network located at a second party. According to the method, the first party obtains a local target training statement, inputs the target training statement to the encoding network, and forms a sentence representation vector based on an encoding output of the encoding network; and then adds target noise that conforms to differential privacy to the sentence representation vector, to obtain a target noise addition representation, and sends the target noise addition representation to the second party for training of the processing network.