Contrastive Data Selection for Noisy Neural Translation Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural translation models trained on noisy parallel data produce inaccurate translations due to the introduction of noise during data processing, and existing methods struggle to effectively select high-quality data for training, leading to less precise translations.
Innovation Solution
A method involving a base model trained on a first dataset of noisy data pairs, an adapted model trained on a cleaner second dataset, and a target model trained using contrastive scores to select and prioritize high-quality data pairs, employing dynamic data scheduling to incrementally improve translation quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If parallel data is processed by a sentence aligner to enable neural translation model training, then the model can learn translation distributions, but severe noise is introduced to the parallel data during processing
Solution Approach 1:
The patent applies preliminary action by training a selector model beforehand to identify and filter noisy data pairs before they are used to train the translation model. The selector model is pre-trained on clean parallel data to learn characteristics of high-quality data, then uses this knowledge to screen the noisy parallel data, keeping only the clean pairs for translation model training. This preliminary filtering prevents noise from contaminating the translation model training process.
2Ease of operation
If a convolutional network is trained to classify good or bad data pairs using a small amount of clean data, then data selection can be performed, but the selected data may not necessarily be good data for the final model with different architecture
Solution Approach 1:
The patent implements feedback by using the final translation model's performance on held-out clean data to evaluate and refine the data selection process. The selector model's choices are feedback-looped through the translation model training, and the selection criteria are adjusted based on how well the selected data improves translation performance. This ensures the selected data is actually beneficial for the final model rather than just appearing clean based on architectural assumptions.
3Quantity of substance
If all parallel data is used to train the translation model, then more training data is available, but the translation precision decreases due to noisy data
Solution Approach 1:
The patent applies local quality by treating different data pairs differently based on their individual quality characteristics. Instead of uniformly using all data or uniformly rejecting noisy data, the selector model evaluates each data pair locally and assigns it a quality score. High-quality pairs are selected for training while low-quality pairs are discarded. This localized quality assessment allows the system to maximize the use of valuable data while filtering out noise, achieving both sufficient data quantity and high translation precision.
Data Source
AI summary
A method includes generating a base model by training with a first dataset of data pairs and generating an adapted model by training the base model on a second dataset of data pairs. The method also includes determining a contrastive score for each data pair of a third dataset of data pairs using the base model and the adapted model. The contrastive score is indicative of a probability of quality of the respective data pair. The method also includes training a target model using the data pairs of the third dataset and the contrastive scores.