Deep Learning Pre-Training With Regularization for Low-Data Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The performance of deep learning models for natural language processing, particularly in translation tasks, is limited by the scarcity of training data, making it difficult to further improve their effectiveness.
Innovation Solution
A method involving the use of a deep learning model to process sample source and target data, perform word segmentation, masking, and adjust model parameters based on a regularization function value to enhance training, resulting in a pre-trained model that improves performance through bidirectional pre-training and unidirectional fine-tuning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional deep learning models are trained with limited training data, then training time and computational resources are reduced, but model performance and translation accuracy deteriorate
Solution Approach 1:
The patent applies preliminary action by performing bidirectional pre-training before fine-tuning. The model is pre-trained on large-scale parallel corpora to learn robust language representations and translation patterns beforehand, then fine-tuned on task-specific data. This preliminary pre-training phase enables the model to achieve high performance even when final training data is limited, as the model has already acquired fundamental translation capabilities.
Solution Approach 2:
The patent employs parameter changes by dynamically adjusting learning rates, temperature parameters, and regularization coefficients during different training stages. The bidirectional pre-training uses different hyperparameter configurations compared to the fine-tuning phase, allowing the model to adapt to different data regimes and achieve optimal performance with limited training data.
2Reliability
If bidirectional pre-training is performed to improve model performance, then translation accuracy is enhanced, but training complexity and computational cost increase
Solution Approach 1:
The patent applies segmentation by dividing the training process into distinct phases: bidirectional pre-training phase and fine-tuning phase. Each phase has specific objectives, data requirements, and hyperparameter configurations. This segmentation allows the complex training process to be managed systematically, with each phase contributing specifically to different aspects of model performance.
Solution Approach 2:
The bidirectional pre-training process serves multiple functions simultaneously: it learns source language representations, learns target language representations, captures translation patterns, and regularizes the model. This multi-functionality justifies the increased training complexity by delivering multiple performance benefits from a single pre-training phase.
3Reliability
If regularization techniques are applied during training, then model generalization is improved, but training convergence speed decreases
Solution Approach 1:
The patent applies periodic action by implementing regularization techniques at specific intervals during training rather than continuously. The bidirectional pre-training alternates between forward and backward translation tasks periodically, providing implicit regularization. Additionally, dropout and weight decay are applied periodically during the fine-tuning phase, balancing generalization improvement with maintaining reasonable convergence speed.
Data Source
AI summary
A method of training a deep learning model is provided, which relates to a field of artificial intelligence, in particular to a field of a natural language processing technology and a field of a machine translation technology. A specific implementation solution includes: processing sample source data and corresponding sample target data respectively by using the deep learning model, so as to obtain a first output value and a second output value; determining a regularization function value according to the first output value and the second output value; and adjusting a parameter of the depth learning model according to the regularization function value, so as to obtain a pre-trained depth learning model. A method of processing text data, an electronic device, and a storage medium are further provided.