Non-Autoregressive Translation Training With Hybrid Text Pretraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing encoder-decoder models in machine translation struggle to achieve good performance with scarce languages due to difficulty in training a non-autoregressive translation model and a tendency to fall into local optima.
Innovation Solution
A method for training a non-autoregressive translation model using a pre-trained model and hybrid texts, where initialization parameters are determined based on a pre-trained translation model, and the model is trained with hybrid texts to improve performance and avoid local optima.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a non-autoregressive translation model is trained from scratch, then the model can achieve translation performance, but the training process is difficult and prone to falling into local optima
Solution Approach 1:
The patent applies preliminary action by pre-training a translation model using hybrid texts (combining source and target language texts) before fine-tuning on pure translation tasks. This pre-training phase initializes the model with useful linguistic patterns and representations, making the subsequent fine-tuning process easier and less prone to local optima. The hybrid text pre-training serves as a preparatory step that builds a solid foundation for the final translation model.
2Reliability
If hybrid texts are used for pre-training, then the model can avoid local optima and improve training stability, but the training data structure becomes more complex
Solution Approach 1:
The patent applies segmentation by dividing the training process into distinct phases: pre-training with hybrid texts and fine-tuning with pure translation data. The hybrid texts themselves are segmented into source language portions and target language portions that are processed separately through encoder and decoder components. This segmentation allows the model to first learn from the structured hybrid data and then specialize in translation, managing complexity through phased approach.
3Adaptability or versatility
If the model is trained with scarce language data, then the model can be applied to low-resource languages, but the training performance deteriorates
Solution Approach 1:
The patent applies universality by using hybrid texts that combine multiple language pairs and directions in the pre-training phase. The model learns universal translation patterns and linguistic representations from these diverse hybrid texts, creating a more robust and adaptable model. This multi-functional pre-training approach enables the model to handle scarce language data more effectively by leveraging patterns learned from multiple language contexts during pre-training.
Data Source
AI summary
A method for training a non-autoregressive translation (NAT) model includes: acquiring a source language text, a target language text corresponding to the source language text and a target length of the target language text; generating a target language prediction text and a prediction length by inputting the source language text into the NAT model, in which initialization parameters of the NAT model are determined based on parameters of a pre-trained translation model; and obtaining a target NAT model by training the NAT model based on the target language text, the target language prediction text, the target length and the prediction length.