Base model training method and apparatus

Abstract

Provided in the embodiments of the present disclosure are a base model training method and apparatus. The method comprises: generating, by means of self-play, a plurality of pieces of different response text on the basis of a text instruction that is input into a base model; on the basis of the plurality of pieces of response text and in combination with score values corresponding to the plurality of pieces of response text, selecting the plurality of pieces of response text, and constructing a machine preference dataset; and on the basis of the constructed machine preference dataset, training the base model.

Description

Base model training method and device

[0001] Cross-references to related applications

[0002] This disclosure is based on and claims priority to Chinese patent application CN202411520303.6, filed on October 28, 2024, entitled “Base Model Training Method and Apparatus”, and incorporates the entire contents of that patent application by reference. Technical Field

[0003] This disclosure relates to the field of natural language processing, and more specifically, to a base model training method and apparatus. Background Technology

[0004] With the rapid development of artificial intelligence technology, especially in the field of Natural Language Processing (NLP), and driven by foundational models (such as large-scale pre-trained models like Generative Pre-trained Transformers (GPT), Large Language Models with Attention (LLMA), and Bidirectional Encoder Representations from Transformers (BERT)), these models, through unsupervised pre-training on massive amounts of data, possess powerful language understanding and generation capabilities, enabling them to perform complex language tasks such as dialogue systems, machine translation, and text generation. However, despite their excellent performance in many tasks, how to further train and optimize these models remains a pressing issue that needs to be addressed.

[0005] Traditional training methods for pedestal models primarily rely on supervised learning: fine-tuning the model under supervision by labeling a large amount of high-quality training data. While this approach improves the training efficiency and performance of pedestal models to some extent, the high cost and time consumption of manual labeling limit the scale and speed of pedestal model training.

[0006] In summary, no effective solution has yet been proposed in the relevant technologies. Summary of the Invention

[0007] This disclosure provides a base model training method and apparatus to at least solve the problem in the related art that the training of base models mainly relies on a large amount of manually labeled data, resulting in long training cycles and high costs for base models.

[0008] According to one embodiment of this disclosure, a base model training method is provided, comprising: generating multiple different response texts through self-play based on text instructions input to the base model; filtering the multiple response texts and constructing a machine preference dataset based on the multiple response texts and their corresponding scores; and training the base model based on the constructed machine preference dataset.

[0009] According to another embodiment of this disclosure, a base model training apparatus is provided, comprising: a generation module configured to generate multiple different response texts through self-play based on text instructions input to the base model; a filtering module configured to filter the multiple response texts and construct a machine preference dataset based on the multiple response texts and their corresponding scores; and a training module configured to train the base model based on the constructed machine preference dataset.

[0010] According to yet another embodiment of this disclosure, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0011] According to yet another embodiment of this disclosure, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0012] According to yet another embodiment of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments. Attached Figure Description

[0013] Figure 1 is a hardware structure block diagram of a computer terminal for a base model training method according to an embodiment of the present disclosure;

[0014] Figure 2 is a structural block diagram of a base model training device based on large model self-play according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of the overall process of training the base model according to an embodiment of the present disclosure;

[0016] Figure 4 is a flowchart of a base model training method according to an embodiment of the present disclosure;

[0017] Figure 5 is a structural block diagram of a base model training device according to an embodiment of the present disclosure. Detailed Implementation

[0018] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings and examples.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] The methods and embodiments provided in this application can be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking a computer terminal as an example, FIG1 is a hardware structure block diagram of a computer terminal for a base model training method according to an embodiment of this disclosure. As shown in FIG1, the computer terminal may include one or more (only one is shown in FIG1) processors 102 (processor 102 may include, but is not limited to, processing devices such as microprocessors MCUs or programmable logic devices FPGAs) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that the structure shown in FIG1 is only illustrative and does not limit the structure of the computer terminal. For example, the computer terminal may also include more or fewer components than shown in FIG1, or have a different configuration than shown in FIG1.

[0021] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the base model training method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0022] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0023] Currently, the training and optimization of base models (such as large-scale pre-trained models) mainly rely on the following two methods:

[0024] (1) Supervised learning: Supervised fine-tuning (SFT) of a large-scale pre-trained model is performed by labeling a large amount of high-quality training data. That is, based on the large-scale pre-trained model, the model is further trained by providing labeled data to adapt it to a specific task. This process can significantly improve the performance of the large-scale pre-trained model in specific application scenarios, but it requires a large amount of high-quality manually labeled data, which is not only costly but also inefficient.

[0025] (2) Unsupervised learning: This involves training a large-scale pre-trained model on unlabeled data using self-supervised learning (SSL) methods. This involves automatically generating labels from the input data to train the model, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), to improve the model's generalization ability. However, while unsupervised learning does not require a large amount of labeled data, the accuracy of the resulting large-scale pre-trained model is usually lower than that of a large-scale pre-trained model trained using supervised learning.

[0026] While the two methods mentioned above have improved the performance of large-scale pre-trained models to some extent, some problems still exist, such as: high data annotation cost: large-scale pre-trained models usually require a large amount of high-quality labeled data, while manual annotation is costly and time-consuming; low model optimization efficiency: traditional model training methods mainly rely on manual design and debugging, making it difficult to achieve rapid iteration; and insufficient model generalization ability: existing methods perform well on specific tasks, but when applied across tasks, the model's generalization ability still needs to be improved.

[0027] To address the aforementioned issues, optimization methods based on Reinforcement Learning (RL) have gradually gained attention. Reinforcement learning, through the interaction between the agent and the environment, continuously adjusts its policy to maximize cumulative reward, offering advantages such as automation and efficient optimization. However, traditional reinforcement learning methods face the following challenges when applied to large-scale pre-trained models:

[0028] (1) Large-scale pre-trained models have numerous parameters and complex structures, making it difficult to apply traditional reinforcement learning algorithms directly.

[0029] (2) Human preference-based reinforcement learning typically relies on human preference datasets to train reward models. First, it is limited by human performance levels during model training because the quality of the reward model (RM) depends on the size and quality of the human preference data. Second, independent, fixed reward models cannot be further improved during the training of large language models (LLM).

[0030] In view of this, this disclosure proposes a model training method based on large-scale model self-play (SP). Self-play refers to a method in which a model trains by playing against itself. By continuously generating adversarial examples, the model can achieve self-improvement and optimization without external data. Furthermore, by combining a reward model to score and rank the generated adversarial examples, the highest and lowest-scoring adversarial examples are selected to automatically construct a machine preference dataset, generating high-quality training data and reducing reliance on manual annotation. This enables automated training and optimization of large-scale pre-trained models.

[0031] Figure 2 is a structural block diagram of a base model training device based on large model self-game according to an embodiment of the present disclosure. As shown in Figure 2, the base model training device based on large model self-game includes: a generation module, a sorting module, a construction module, an alignment module, and an iteration module.

[0032] The generation module generates multiple response texts based on the input text instructions, thus producing diverse response texts. This module is responsible for calling the supervised fine-tuned pedestal model (SFT Model) to generate K different response texts, providing diverse adversarial examples for the reward model in the ranking module to score and rank the K different response texts.

[0033] The ranking module uses a pre-trained reward model to score and rank the K distinct generated response texts. Specifically, it scores the quality of each of the K different response texts and ranks them according to their scores, either in descending or ascending order, to determine the best and worst response texts.

[0034] Building Module: Based on the best and worst response texts determined from the ranking module, a machine preference dataset is constructed. This machine preference dataset is used for reinforcement learning training in the alignment module, enabling the base model to learn human preferences.

[0035] Alignment module: Uses reinforcement learning algorithms (such as DPO, SPPO, PPO, KTO, etc.) to train and optimize the pedestal model, that is, by training the machine to prefer the dataset, the model parameters of the pedestal model are continuously adjusted to improve the performance of the pedestal model.

[0036] Iteration Module: Manages and controls the entire iteration process. After each round of optimization, the base model is updated, and the execution steps in the generation module, sorting module, construction module, and alignment module are repeated until the current iteration count meets a preset threshold, thus stopping the iteration to ensure that the base model gradually improves its performance over multiple iterations.

[0037] Figure 3 is a schematic diagram of the overall process of training the base model according to an embodiment of the present disclosure. As shown in Figure 3, the specific steps include the following:

[0038] Step S301: Input the text instructions into the supervised fine-tuning base model;

[0039] In some embodiments, this disclosure uses a supervised fine-tuned base model as the initial model for the self-game process. The initial model already has basic dialogue generation capabilities and can generate reasonable response text based on input text instructions. The text instructions can be extracted from existing SFT datasets, such as: explain what artificial intelligence is, describe the feeling of summer, or what I should wear to my job interview today.

[0040] Step S302: Based on the input text instruction, generate K different response texts through the generation module;

[0041] In some embodiments, K different response texts are generated based on the input text instruction, i.e., diverse response texts are generated. Response texts can be generated from various aspects such as tone style, information content, response perspective, emotional perception, and emotional tone. For example, if the input text instruction is "Describe what summer feels like," the generation module can generate a response from a personal perspective: "Summer is like a sweet fruit, full of the taste of ripeness and harvest." Or, from the perspective of natural scenery: "Summer is the season of sunshine, beaches, and waves; nature is full of vitality."

[0042] Step S303: Score and rank the K different response texts using the reward model in the ranking module;

[0043] In some embodiments, a pre-trained reward model is used to score K different response texts, where the scores can be 1-5 points and sorted according to the score values ​​in descending or ascending order.

[0044] Step S304: Filter the K different reply texts;

[0045] In some embodiments, based on the sorting of K different response texts in step S303, the first (highest score) and last (lowest score) or the first (lowest score) and last (highest score) in the sequence are selected as the best response text and the worst response text or the worst response text and the best response text.

[0046] Step S305: Based on the selected best and worst response texts, construct a machine preference dataset using the building module;

[0047] In some embodiments, the constructed machine preference dataset is used for reinforcement learning training in the alignment module, enabling the pedestal model to learn human preferences;

[0048] Step S306: Based on the machine preference dataset, train the base model using a reinforcement learning algorithm through the alignment module;

[0049] In some embodiments, reinforcement learning algorithms, such as Direct Policy Optimization (DPO), Soft Policy Policy Optimization (SPPO), Proximal Policy Optimization (PPO), and Kernelized Trust-Region Optimization (KTO), are used to train and optimize the base model based on a machine preference dataset. The model parameters of the base model are continuously adjusted to improve its performance.

[0050] Step S307: Update the base model.

[0051] In some embodiments, by repeatedly generating responses, ranking scores, constructing machine preference datasets, and aligning optimization steps, the base model is updated after each round of optimization. Through multiple rounds of iteration, the iteration stops when the current iteration count meets a preset threshold, ensuring that the base model gradually improves its performance in multiple rounds of iteration, the quality of the generated response text becomes higher and higher, and the expected optimization goal is ultimately achieved through the score values.

[0052] Through the above steps, this disclosure proposes an automated reinforcement learning optimization method based on a base model self-game. It generates diverse response texts through base model self-game and uses a pre-trained reward model for scoring and ranking, selecting the best and worst response texts to construct a machine preference dataset. Based on this dataset, a reinforcement learning algorithm is used to train the base model, forming a closed-loop automated optimization process. This optimization method not only effectively improves the performance of the base model but also enhances its generalization ability, making it suitable for model optimization in various natural language processing tasks and possessing broad application prospects.

[0053] This disclosure provides a base model training method running on the aforementioned computer terminal. Figure 4 is a flowchart of the base model training method according to this disclosure. As shown in Figure 4, the process includes the following steps:

[0054] Step S402: Generate multiple different response texts through self-game based on the text instructions input to the base model;

[0055] In this embodiment, the base model can be a large-scale pre-trained model, such as GPT, LLMA, or BERT. Text instructions can be extracted from existing SFT datasets to ensure that the generated response texts are diverse and have practical application scenarios. The generated response text set can be represented as R = {r_1, r_2, ..., r_k}.

[0056] In this embodiment, self-game refers to a method in which the model trains by playing against itself. By continuously generating adversarial examples, the model can train and optimize itself without external data.

[0057] In this embodiment, a supervised fine-tuned base model (intelligent dialogue model) is selected and prepared as the initial model for the self-game process. The initial model already has basic dialogue generation capabilities and can generate reasonable response text based on the input text instructions. Determining a suitable base model is the starting point and foundation of the entire optimization process.

[0058] This embodiment generates diverse response texts based on the self-game theory of the base model, solving the problem of relying on manually labeled data in traditional methods. This reduces dependence on manually labeled data, lowers data acquisition costs, and improves the diversity and coverage of training data.

[0059] Step S404: Based on the multiple reply texts and their corresponding scores, filter the multiple reply texts and construct a machine preference dataset;

[0060] In step S404 of this embodiment, the step of filtering the multiple response texts and constructing a machine preference dataset based on the multiple response texts and their corresponding scores includes: a generation step: generating multiple different response texts through self-game based on text instructions input to the base model; a sorting step: sorting the multiple response texts according to their corresponding scores to obtain the sorted response texts in the current iteration; a filtering step: filtering the sorted response texts in the current iteration to construct a machine preference dataset; an update step: updating the base model based on the constructed machine preference dataset to generate multiple different response texts again using the new base model; and a judgment step: determining whether the current iteration number meets a preset threshold, and if the current iteration number does not meet the preset threshold, repeating the generation step, the sorting step, the filtering step, the update step, and the judgment step.

[0061] In this embodiment, the generated K different response texts are sorted. This is mainly accomplished by the reward model in the sorting module, which scores the K different response texts using a pre-trained reward model. The scores can be from 1 to 5, and the texts are sorted in descending or ascending order based on the scores. The score set can be represented as S = {s_1, s_2, ..., s_k}.

[0062] In this embodiment, the accuracy of scoring and ranking the generated K different response texts directly affects the subsequent optimization process. For example, this embodiment can use a state-of-the-art (SOTA) open-source large model such as qwen2 72B to score and rank the generated K different response texts. Through scoring and ranking, the response text that best matches human preferences (optimal response text) and the response text that least matches human preferences (worst response text) can be identified. Based on the selected optimal and worst response texts, a machine preference dataset is constructed through a construction module. Based on the machine preference dataset, a base model is trained using a reinforcement learning algorithm through an alignment module. By continuously repeating the steps of generating responses, scoring and ranking, constructing the machine preference dataset, and alignment optimization, the base model is updated after each round of optimization. Through multiple rounds of iteration, the iteration stops when the current iteration number meets a preset threshold, ensuring that the base model gradually improves its performance in multiple rounds of iteration.

[0063] In one exemplary embodiment, sorting the multiple reply texts according to their corresponding scores to obtain the sorted reply texts for this iteration includes: inputting the multiple reply texts into a reward model and scoring them according to scoring rules to obtain a score for each reply text, wherein the scoring rules include at least one of the following: whether the reply text is missing punctuation, the number of words in the reply text, the grammar of the reply text, and the citation of the reply text; arranging the multiple reply texts in descending or ascending order of the scores to obtain the sorted reply texts for this iteration.

[0064] In this embodiment, after the base model generates K different response texts based on the input text instructions, these texts are transmitted as input to the reward model. The reward model is a trained model used to evaluate the response texts.

[0065] The scoring rules define the criteria for evaluating response text in the reward model, ensuring consistency and fairness in the scoring. The scoring rules must include at least one of the following factors:

[0066] Punctuation Completeness: Check if any necessary punctuation marks are missing from the reply text. Correct use of punctuation is crucial for improving the readability and comprehensibility of replies. Replies lacking punctuation may be difficult to understand and negatively impact user experience; therefore, punctuation completeness is considered an important scoring criterion.

[0067] Response text length: This assesses whether the length of the response text is appropriate. Responses that are too short may lack sufficient information, while responses that are too long may be redundant or off-topic. The reward model determines the ideal response length based on task requirements and context, using this as one of the scoring criteria.

[0068] Grammatical correctness: Check the grammar of the response text. Grammatical errors not only affect the fluency of the text but can also lead to confusion or misinterpretation of meaning. The reward model uses grammar as one of the scoring criteria for evaluating the quality of the response text.

[0069] Content Citation Accuracy: This assesses whether the information cited in the response text is accurate and appropriate. For responses that require citing external knowledge or data, the reward model checks whether these citations are correct and from reliable sources, ensuring the authority and accuracy of the response.

[0070] In this embodiment, the scoring process involves the reward model scoring each response text according to the aforementioned scoring rules. The scoring typically uses a range, such as 1-5, where 1 represents the lowest response quality and 5 represents the highest. The responses are then sorted according to their scores, either in descending or ascending order, resulting in the K sorted response texts for this iteration. The score set (the K sorted response texts) can be represented as S = {s_1, s_2, ..., s_k}, and this score set from the reward model will serve as the basis for subsequently selecting the best and worst response texts to construct the machine preference dataset.

[0071] This embodiment employs scoring rules and a dynamic reward model to score and rank the generated K different responses, overcoming the limitations of fixed reward models in traditional methods. The reward model continuously improves itself during training, and its evaluation capability increases in tandem with the model's performance. This dynamic evaluation mechanism more accurately reflects the model's actual performance, making the optimization process more efficient and accurate, and avoiding the reliance on human preference data inherent in traditional reinforcement learning with human feedback (RLHF) methods.

[0072] In an exemplary embodiment, the step of filtering the sorted multiple response texts in the current iteration to construct a machine preference dataset includes: for any response text, determining the response text with the highest score from the reward model among the multiple response texts as the optimal response text; for any response text, determining the response text with the lowest score from the reward model among the multiple response texts as the worst response text; and integrating the optimal response text and the worst response text to construct a machine preference dataset.

[0073] In this embodiment, the response text with the highest score (i.e., the optimal response text) and the response text with the lowest score (i.e., the worst response text) are selected from the score set. The optimal and worst response texts can be represented as r_best and r_worst, respectively. These response texts will serve as important data for constructing the machine preference dataset and will be used for subsequent reinforcement learning training.

[0074] Step S406: Train the base model based on the constructed machine preference dataset.

[0075] In step S406 of this embodiment, training the base model based on the constructed machine preference dataset includes: adjusting the parameters of the base model using a reinforcement learning algorithm based on the constructed machine preference dataset to train the base model.

[0076] In this embodiment, the reinforcement learning algorithm includes at least one of the following: DPO, SPPO, PPO, and KTO.

[0077] In this embodiment, the pedestal model is trained and optimized using reinforcement learning algorithms (such as DPO, SPPO, PPO, KTO, etc.) based on a machine preference dataset. The model parameters are continuously adjusted to improve the performance of the pedestal model. The optimized pedestal model will then serve as the new pedestal model for the next round of generation and optimization.

[0078] In this embodiment, by continuously repeating the steps of generating responses, ranking scores, constructing machine preference datasets, and alignment optimization, the base model is updated after each round of optimization. Through multiple rounds of iteration, the iteration stops when the current iteration count meets a preset threshold, ensuring that the base model gradually improves its performance in multiple rounds of iteration and the quality of the generated responses becomes higher and higher. After each round of optimization, the base model generates K new different response texts and accepts evaluation from the reward model, ensuring the continuity and effectiveness of the optimization process.

[0079] This embodiment utilizes a multi-round iterative approach to reinforcement learning optimization, achieving a gradual improvement in model performance. After each round of optimization, the model generates a new response and is evaluated by the reward model, forming a closed-loop self-game optimization process. Reinforcement learning algorithms (such as DPO, SPPO, PPO, KTO, etc.) are used to optimize the model, combined with data generated from the self-game, ensuring that each round of optimization significantly improves model performance. This iterative optimization method not only improves the model's accuracy and stability.

[0080] The above steps provide a method for training a pedestal model. Since the pedestal model can generate multiple different response texts through self-play based on the input text instructions, it produces diverse responses. Then, based on these multiple response texts and their corresponding scores, a machine preference dataset is constructed, generating high-quality training data and reducing reliance on manual annotation. This machine preference dataset is then used to train the pedestal model, achieving automated training and improving its performance. Therefore, this method addresses the problem in related technologies where pedestal model training primarily relies on large amounts of manually labeled data, leading to long training cycles and high costs, thus improving the training efficiency of the pedestal model.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0082] This embodiment also provides a base model training device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0083] Figure 5 is a structural block diagram of a base model training device according to an embodiment of the present disclosure. As shown in Figure 5, the base model training device includes a generation module 510, a screening module 520, and a training module 530.

[0084] The generation module 510 is configured to generate multiple different response texts through self-play based on the text instructions input to the base model;

[0085] The filtering module 520 is configured to filter the multiple reply texts and construct a machine preference dataset based on the multiple reply texts and the corresponding scores of the multiple reply texts.

[0086] Training module 530 is configured to train a pedestal model based on a constructed machine preference dataset.

[0087] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0088] Embodiments of this disclosure also provide a computer-readable storage medium storing a computer program configured to perform the steps in any of the above method embodiments when executed.

[0089] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0090] Embodiments of this disclosure also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0091] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0092] According to yet another embodiment of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this disclosure.

[0093] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0094] It is obvious to those skilled in the art that the modules or steps of this disclosure described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this disclosure is not limited to any particular combination of hardware and software.

[0095] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A base model training method, comprising: Based on the text instructions input into the base model, the self-play generates multiple different response texts; Based on the multiple response texts and their corresponding scores, the multiple response texts are filtered and a machine preference dataset is constructed. The base model is trained based on a constructed machine preference dataset.

2. The method according to claim 1, wherein, The base model is an intelligent dialogue model that has been fine-tuned through supervision.

3. The method according to claim 1, wherein, The step of filtering the multiple response texts and constructing a machine preference dataset based on the multiple response texts and their corresponding scores includes: Generation steps: Generate multiple different response texts through self-play based on the text instructions input to the base model; Sorting steps: Sort the multiple reply texts according to their corresponding score values ​​to obtain the sorted reply texts in this iteration; Filtering steps: Filter the multiple sorted response texts from this iteration to construct a machine preference dataset; Update steps: Update the base model based on the constructed machine preference dataset to generate multiple different response texts again with the new base model; Judgment step: Determine whether the current iteration number meets the preset threshold, and if the current iteration number does not meet the preset threshold, repeat the generation step, the sorting step, the filtering step, the update step, and the judgment step.

4. The method according to claim 3, wherein, The process of sorting the multiple reply texts according to their corresponding scores to obtain the sorted reply texts in this iteration includes: The multiple reply texts are input into the reward model and scored according to the scoring rules to obtain the score value corresponding to each reply text. The scoring rules include at least one of the following: whether the reply text is missing punctuation marks, the number of words in the reply text, the grammar of the reply text, and the citation of the reply text. The multiple response texts are arranged in descending or ascending order of their scores to obtain the sorted response texts in this iteration.

5. The method according to claim 4, wherein, The process of filtering the sorted response texts from this iteration to construct a machine preference dataset includes: For any given response text, the response text with the highest score among the multiple response texts determined by the reward model is the optimal response text; For any given response text, the response text with the lowest score among the multiple response texts is determined as the worst response text; The best and worst response texts are integrated to construct a machine preference dataset.

6. The method according to claim 1, wherein, The training of the base model based on the constructed machine preference dataset includes: The pedestal model is trained by adjusting its parameters using a reinforcement learning algorithm based on a constructed machine preference dataset.

7. The method according to claim 6, wherein, The reinforcement learning algorithm includes at least one of the following: DPO, SPPO, PPO, and KTO.

8. A base model training device, comprising: The generation module is configured to generate multiple different response texts through self-play based on the text instructions input to the base model; The filtering module is configured to filter the multiple reply texts and construct a machine preference dataset based on the multiple reply texts and their corresponding scores. The training module is configured to train a base model based on a constructed machine preference dataset.

9. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images