Adversarial Sequence Generation for Robust Neural Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural machine translation (NMT) models are sensitive to input noises such as typographical errors and adversarial attacks, leading to reduced translation quality and robustness, and existing adversarial training methods are resource-intensive and inefficient.
Innovation Solution
A method for training a neural language model by optimizing parameters on clean and noisy sequence pairs using an adversarial generator that is differentiable, allowing for direct training through backpropagation to minimize modeling loss and maximize neural language loss, with optional use of a discriminator to distinguish between clean and substituted tokens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If gradient-based adversarial attacks are used to perturb words and maximize model error, then robustness against adversarial examples is improved, but training time and computational resources increase significantly
Solution Approach 1:
The patent pre-computes and stores adversarial examples during an offline phase before the main training process. These pre-generated adversarial examples are then used during the online training phase without requiring real-time computation of adversarial perturbations, significantly reducing training time while maintaining robustness improvements
Solution Approach 2:
The patent creates synthetic adversarial examples by copying and modifying clean training data through systematic word substitutions and perturbations. These synthetic adversarial examples replicate the effect of real adversarial attacks without requiring actual adversarial attack computations during training, reducing computational burden
2Reliability
If existing adversarial training methods are used to substitute words and maximize perturbation, then model robustness is improved, but computational costs and resource consumption increase
Solution Approach 1:
The patent performs adversarial example generation in advance during an offline preprocessing stage, computing and storing the perturbed examples before the main training begins. This eliminates the need for continuous adversarial computation during training iterations, reducing per-step computational costs and energy consumption
Solution Approach 2:
The patent uses a simplified adversarial training approach that relies on pre-computed perturbations rather than expensive real-time adversarial attacks. The training process uses these ready-made adversarial examples as disposable training data, avoiding the high computational cost of generating new adversarial examples at each training step
Data Source
AI summary
Methods and systems for training a neural language model. Clean sequence pairs are received including clean source and target sequences. For each clean sequence pair, a noisy version is sampled with an adversarial generator to generate a noisy sequence pair. Parameters of the neural language model are optimized on the clean and noisy sequence pairs. Parameters of the adversarial generator are optimized to minimize a modeling loss of the adversarial generator and maximize a neural language loss of the neural language model using backpropagation.