Model Training Policy Extraction for Scalable LLM Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional model training processes rely heavily on annotated data, limiting the model's ability to expand solving policies beyond what is annotated, thereby affecting the model's scalability.
Innovation Solution
The method involves fine-tuning a target model using a set of training data that includes sample questions and corresponding annotation information. The model is then provided with sample questions to determine candidate answers, and it is trained based on the comparison between the candidate answers and the answer information, allowing the model to explore different solving policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional model training processes rely heavily on annotated data, then the model can be trained with available data, but the model's ability to expand solving policies beyond annotated data is limited
Solution Approach 1:
The patent introduces a policy extraction module that acts as an intermediary between the large language model and the training process. This module automatically extracts solving policies from the model's generated answers without requiring manual annotation, thereby enabling the model to expand its solving policies beyond the limited annotated data while maintaining training effectiveness
Solution Approach 2:
The system enables self-service by allowing the large language model to generate its own training data through automatic policy extraction from its responses. The model serves itself by producing solved problems and extracting policies from its own outputs, eliminating the need for external manual annotation and enabling continuous expansion of solving policies
Data Source
AI summary
There are provided a method, an apparatus, a device, and a storage medium for model training. In a method, a target model is fine-tuned using a set of training data, each training data including a sample question and corresponding annotation information, the annotation information including policy information for solving the sample question and answer information of the sample question. At least one sample question in the set of training data is provided to the fine-tuned target model to determine a candidate answer to the at least one sample question. The fine-tuned target model is trained based at least on a comparison between the candidate answer and the answer information of the at least one sample question.