Multilingual NLG Fine-Tuning With Constrained Vocabulary
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language generation (NLG) technologies lack the ability to generate data in multiple languages due to the scarcity of annotated data for specific tasks like named entity recognition (NER) and sequence classification, limiting their effectiveness in multilingual applications.
Innovation Solution
A novel technique for automated entity alignment and knowledge distillation via two-fold fine-tuning of a modified mT5 model, utilizing public and private datasets to generate bilingual models capable of text generation, summarization, and next sentence generation across multiple languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a base model is pretrained on multiple languages with a large vocabulary, then multi-lingual capability is improved, but model size and complexity increase
Solution Approach 1:
The patent segments the training process into distinct phases: first training a text summarization model, then a text generation model, and finally a next sentence generation model. Each model is trained on specific enhanced training datasets tailored to its function, allowing the system to achieve multi-lingual capability without requiring a single large monolithic model
Solution Approach 2:
The patent extracts and utilizes publicly available multi-lingual datasets to create enhanced training datasets, removing the need to store and process all raw multilingual data in the model itself. The base model is pretrained on multiple languages but the vocabulary is constrained to essential words from two languages, extracting only the necessary linguistic elements
2Reliability
If annotated data for specific tasks is increased to improve NLG performance, then task accuracy is improved, but data availability and collection difficulty worsen
Solution Approach 1:
The patent performs preliminary actions by automatically generating enhanced training datasets from publicly available data before the actual model training. The datasets are preprocessed to include task-specific annotations and structures needed for NER, sequence classification, and other NLG tasks, eliminating the need for manual annotation during the manufacturing process
Solution Approach 2:
The system uses automated data generation pipelines that create annotated training data without human intervention. The enhanced training datasets are self-generated from public sources through automated processing, allowing the system to service its own data needs without external annotation resources
3Adaptability or versatility
If the model is trained on diverse languages and tasks, then adaptability is improved, but training time and computational resources increase
Solution Approach 1:
The training process is segmented into three separate models trained sequentially: text summarization model, text generation model, and next sentence generation model. Each model focuses on specific tasks and is trained on targeted enhanced training datasets, reducing the overall training time compared to training a single model on all tasks simultaneously
Solution Approach 2:
Each model component is optimized for its specific function with tailored training data. The text summarization model receives article-summary pairs, the text generation model receives keyword-text pairs, and the next sentence generation model receives summary-keyword-next sentence triplets. This local optimization allows faster convergence for each specific task
Data Source
AI summary
A method includes preparing a base model using an input model pretrained on at least three languages different from each other and a base vocabulary including words corresponding to two languages among the at least three languages, where the preparing the base model includes constraining the input model to the words included in the base vocabulary; training the base model using a first enhanced training dataset generated from public data, to generate a text summarization model; training the base model using a second enhanced training dataset generated from the first enhanced training dataset, to generate a text generation model; and training the base model using a third enhanced training dataset that is generated using the second enhanced training dataset and the text summarization model, to generate a next sentence generation model.