Multilingual Text Generation With Two-Stage mT5 Fine-Tuning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language generation (NLG) technologies lack the ability to generate data in multiple languages due to the lack of annotated data for specific tasks like named entity recognition (NER) and sequence classification, limiting their effectiveness in multilingual applications.
Innovation Solution
A novel technique for automated entity alignment and knowledge distillation via two-fold fine-tuning of a modified mT5 model, using public and private datasets to generate bilingual models capable of text generation, summarization, and next sentence generation across multiple languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a base model is pretrained on multiple languages with a large vocabulary, then multi-lingual capability is improved, but model size and complexity increase
Solution Approach 1:
The patent segments the training process into distinct phases: pretraining on multiple languages, then separate fine-tuning stages for specific tasks (text generation, summarization, next sentence prediction). This allows the model to acquire multi-lingual capabilities in the pretraining phase while maintaining a manageable size during task-specific training by using targeted datasets and focused objectives for each language pair.
2Reliability
If annotated data is collected for multiple languages and tasks, then model performance on specific tasks is improved, but data collection time and cost increase
Solution Approach 1:
The patent performs preliminary action by pretraining the base model on multiple languages before task-specific fine-tuning. This pretraining phase prepares the model with general multi-lingual understanding, reducing the need for extensive annotated data during subsequent task-specific training. The model already possesses foundational knowledge that transfers across languages and tasks, accelerating the data collection and training process.
Solution Approach 2:
The patent creates a universal base model through pretraining that serves multiple functions across different languages and tasks. This single pretrained model can be fine-tuned for various NLP tasks (text generation, summarization, next sentence prediction) in multiple language pairs, eliminating the need to collect and train separate models for each language-task combination, thereby significantly reducing data collection time and resources.
3Adaptability or versatility
If the model is trained on diverse datasets with multiple objectives, then adaptability to new use cases is improved, but training complexity and computational resources increase
Solution Approach 1:
The patent segments training into distinct phases with specific objectives: pretraining on multi-lingual data, then separate fine-tuning stages for text generation, summarization, and next sentence prediction. Each phase has a focused objective and targeted dataset, reducing training complexity compared to attempting to train all objectives simultaneously on all data. This phased approach maintains adaptability while managing computational resources efficiently.
Data Source
AI summary
A computer-implemented method includes obtaining, from text corpus including article-summary pairs in a plurality of languages, a plurality of article-summary pairs in a target language among the plurality of languages, to form an article-summary pairs dataset in which each article corresponds to a summary; inputting articles from the article-summary pairs to a machine learning model; generating, by the machine learning model, embeddings for sentences of the articles; extracting, by the machine learning model, keywords from the articles with a probability that varies based on lengths of the sentences, respectively; outputting, by the machine learning model, the keywords; applying a maximal marginal relevance algorithm to the extracted keywords, to select relevant keywords; and generating a keyword-text pairs dataset that includes the relevant keywords and text from the articles, the text corresponding to the relevant keywords in each of keyword-text pairs of the keyword-text pairs dataset.