Language Model Tuning for Cross-Language Code Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language models face challenges in achieving high code generation performance across various programming languages due to performance degradation from training on too many languages, and existing parameter-efficient fine-tuning methods like LoRA simplify the model too much, limiting optimal performance.
Innovation Solution
A method for efficiently updating language model parameters by determining whether the target programming language is in-domain or out-of-domain, selectively updating all layers or just the dense layer based on this determination, and using low-rank updates for large models to reduce computational time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If training is performed on many programming languages to improve code generation performance, then performance across various languages improves, but model training time and inference time increase linearly
Solution Approach 1:
The patent segments the model parameters into two distinct groups: frozen parameters (pretrained on multiple languages) and tunable parameters (specific to target language). This segmentation allows the model to maintain versatility across languages through the frozen parameters while achieving language-specific performance through selective fine-tuning of only the necessary parameters, thereby avoiding linear time increases.
Solution Approach 2:
The patent applies local quality by making different parts of the model have different properties: some parameters remain frozen with their pretrained multi-lingual knowledge, while other parameters are selectively tuned for the target language. This localized fine-tuning approach ensures that computational resources are focused only on the parameters that need adaptation, reducing overall training and inference time while maintaining performance.
2Productivity
If parameter-efficient fine-tuning methods like LoRA are used to reduce training computation, then computational efficiency improves, but model performance is limited due to excessive simplification
Solution Approach 1:
The patent changes the parameter configuration dynamically based on the target programming language. Instead of using a fixed low-rank decomposition as in LoRA, the system identifies and tunes specific parameters that are most relevant to the target language while keeping others frozen. This parameter change strategy maintains computational efficiency while preserving or enhancing model performance by focusing tuning efforts where they are most needed.
3Adaptability or versatility
If the model size is increased to improve code generation performance, then performance improves, but memory usage and training time increase
Solution Approach 1:
The patent extracts and isolates only the necessary parameters for fine-tuning from the complete model parameter set. By identifying and separating the specific parameters that need adaptation for the target programming language from the frozen pretrained parameters, the system achieves language-specific performance without requiring a larger overall model size, thereby reducing memory usage and training requirements.
Data Source
AI summary
A method for tuning a language model is provided. The method may be performed by a computing device, and may comprise: receiving, from a user, a coding requirement and a target programming language to be input into a first model, wherein the first model is a model trained to output a code snippet corresponding to the received coding requirement in the target programming language; determining whether the target programming language is in-domain or out-of-domain with respect to the first model; and updating pretrained parameters of the first model based on a result of the determination and a masking result of the pretrained parameters, wherein the updating the pretrained parameters comprises: updating pretrained parameters of all layers of the first model when the target programming language is in-domain; and updating pretrained parameters of a dense layer of the first model when the target programming language is out-of-domain.