Multilingual LLM Pruning for Lower-Resource Language Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) exhibit varying performance across languages due to resource availability, with lower-resource languages like Hindi and Swahili lacking sufficient training data, leading to suboptimal results and increased training costs and resource intensity.
Innovation Solution
Prune a multilingual LLM using language-aware pruning to reduce its size, followed by continued pretraining with a mix of real and synthetic data tailored to the target lower-resource language, leveraging document translation pipelines to enhance model performance and stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a large language model is used to support multiple languages including lower-resource languages, then language coverage is improved, but performance and accuracy for lower-resource languages deteriorates due to insufficient training data
Solution Approach 1:
The patent divides the multilingual LLM into separate monolingual models for different languages. Instead of using one large model for all languages, it creates specialized smaller models for high-resource languages and target lower-resource languages, allowing each model to be optimized for its specific language's data availability and requirements
Solution Approach 2:
The patent applies different model sizes and training strategies to different languages based on their resource availability. High-resource languages use larger models with extensive training data, while lower-resource languages use smaller models with synthetic data augmentation, tailoring the solution to local language characteristics
2Reliability
If training data for lower-resource languages is increased to improve model performance, then accuracy is improved, but training costs and resource requirements worsen
Solution Approach 1:
The patent uses high-resource languages as intermediaries to generate synthetic training data for lower-resource languages through translation. Instead of directly collecting and processing large volumes of low-resource language data, it translates from high-resource languages, leveraging existing linguistic resources to bootstrap training data for target languages
Solution Approach 2:
The patent creates synthetic copies of training data by translating high-resource language corpora into lower-resource languages. This copying approach allows the model to learn from abundant high-resource data while adapting it to lower-resource language contexts, avoiding the need to collect and process large volumes of original low-resource language data
3Productivity
If the size of the language model is reduced to lower computational requirements, then deployment efficiency is improved, but model capability and performance deteriorates
Solution Approach 1:
The patent segments the large multilingual model into smaller monolingual models, allowing each smaller model to achieve sufficient capability for its specific language while maintaining overall system effectiveness. This segmentation enables deployment of smaller models that are more efficient while preserving language-specific performance
Solution Approach 2:
The patent changes the parameter configuration by training smaller models with language-specific or synthetic data rather than using a large model with all parameters. This parameter adjustment optimizes the trade-off between model size and capability for each target language's specific requirements
Data Source
AI summary
Approaches presented herein provide for the generation of relatively small language models that are optimized for target languages. A multilingual large language model (LLM) can be reduced in size using a process such as language-aware pruning, where individual network parameters have importance scores calculated with respect to the target language and then an appropriate number of lower-importance score parameters are removed from the network. Continued pretraining can be performed using a set of training data including real and/or synthesized text in the target language, to obtain a high performing language model with a limited number of parameters optimized for a target language, as may correspond to a lower-resource language that may otherwise not have enough training data available to sufficiently train a language model from scratch.