Language Model Weight Updating for Cross-Language Preference Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language models struggle to maintain performance when transferring weights from a model trained in one language to another, limiting their effectiveness in cross-language settings, and there is a need for a method to update language models based on user preferences without separate training datasets.
Innovation Solution
A method involving the generation of preferred and non-preferred models using a single training dataset, calculating weight difference vectors, and updating the base model with a combined vector to enhance user preference-based answers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If weight difference from a preference-trained language model is transferred to another language model, then the performance of generating preferred answers is improved, but the reliability in cross-language settings deteriorates
Solution Approach 1:
The patent segments the weight transfer process into language-specific components. Instead of directly transferring weights across languages, it processes each language separately through forward propagation and gradient computation, ensuring language-appropriate updates while maintaining cross-language reliability
Solution Approach 2:
The patent dynamically adjusts weight updates based on language-specific characteristics. By computing gradients and applying updates tailored to each language's data distribution and linguistic features, it maintains manufacturing precision for preferred answer generation while adapting to cross-language variability
2Manufacturing precision
If separate training datasets are constructed for preferred and non-preferred models, then the precision of preference learning is improved, but the device complexity deteriorates
Solution Approach 1:
Instead of creating separate datasets for preferred and non-preferred models, the patent inverts the approach by using a single dataset and applying gradient updates with opposite signs. This maintains precision through differentiated learning while eliminating the complexity of dataset construction and management
Solution Approach 2:
The patent merges the training process into a unified framework where preferred and non-preferred learning occur simultaneously using the same dataset. By combining gradient computations and applying updates in a single pass, it achieves precision without the overhead of separate dataset processing
Data Source
AI summary
A method for updating a model based on user preference and a system therefor are provided. The method according to some embodiments may include generating a preferred model by training a pretrained base model using the training data including a query, a answer to the query, and user preference for the answer, generating a non-preferred model by further training the base model using the training data, updating the weights of the base model using a difference between a first weight difference vector between weights of the preferred model and the base model, and a second weight difference vector between weights of the non-preferred model and the base model.