Training method for language model, and text processing method and apparatus, electronic device, computer-readable storage medium and computer program product

Abstract

Disclosed in the embodiments of the present application are a training method for a language model, and a text processing method and apparatus, an electronic device, a computer-readable storage medium and a computer program product. The training method for a language model comprises: acquiring at least two sample inference paths generated by an initial model for a sample query text; on the basis of the response accuracy of the sample inference paths, constructing accuracy comparison samples; determining target sample inference paths having similar response accuracy, and on the basis of the retrieval efficiency of the target sample inference paths, constructing efficiency comparison samples; performing sampling on the accuracy comparison samples and the efficiency comparison samples, so as to obtain a preference data set; and on the basis of the preference data set, training the initial model, so as to obtain a language model.

Description

Language model training methods, text processing methods, devices, electronic devices, computer-readable storage media, and computer program products

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202510086957.0, filed on January 17, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of Internet technology, and in particular to a language model training method, text processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0004] Text processing technology is an important research direction in the field of natural language processing, aiming to understand, generate, and reason about text information using computers. With the development of deep learning, language model-based text processing methods have been widely used in various scenarios such as question-answering systems and dialogue systems. In recent years, Retrieval-Augmented Generation (RAG) has become one of the key means to improve the performance of language models. The core idea of ​​RAG is to introduce external knowledge bases or document retrieval modules during the generation process to enhance the model's knowledge coverage.

[0005] In related technologies, retrieval-enhanced language models are typically used to learn and understand natural language, automatically generating corresponding text based on a given context in a task. When generating text, this language model predicts the next word step-by-step to produce complete text, making it susceptible to the cumulative effects of early generation errors, resulting in inaccurate text. Furthermore, when the probability of the language model generating a word falls below a preset threshold, an additional retrieval module is introduced to automatically retrieve relevant information, increasing the complexity and computational cost of the text generation process and thus reducing efficiency. Summary of the Invention

[0006] This application provides a language model training method, text processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can quickly generate response text corresponding to query text while ensuring the accuracy of the response.

[0007] This application provides a method for training a language model, which is executed by an electronic device and includes: acquiring at least two sample inference paths generated by an initial model for sample query text; constructing an accuracy comparison sample based on the answer accuracy of each sample inference path; determining target sample inference paths with similar answer accuracy and constructing an efficiency comparison sample based on the retrieval efficiency of each target sample inference path; sampling the accuracy comparison sample and the efficiency comparison sample to obtain a preference dataset; and training the initial model based on the preference dataset to obtain the language model.

[0008] This application provides a text processing method executed by an electronic device, comprising: acquiring query text; inputting the query text into a language model, the language model being trained using the aforementioned language model training method; and generating an answer text corresponding to the query text using the language model.

[0009] This application provides a language model training apparatus, comprising: a first acquisition module configured to acquire at least two sample inference paths generated by an initial model for sample query text; a construction module configured to construct an accuracy comparison sample based on the answer accuracy of each sample inference path; the construction module further configured to determine target sample inference paths with similar answer accuracy and construct an efficiency comparison sample based on the retrieval efficiency of each target sample inference path; a sampling module configured to sample the accuracy comparison sample and the efficiency comparison sample to obtain a preference dataset; and a training module configured to train the initial model based on the preference dataset to obtain the language model.

[0010] This application provides a text processing apparatus, including: an acquisition module, a second acquisition module configured to acquire query text; an input module configured to input the query text into a language model, the language model being trained using the above-described language model training method; and a generation module configured to generate an answer text corresponding to the query text using the language model.

[0011] This application provides an electronic device including one or more processors and a storage device for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to perform the method described above.

[0012] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by the processor of an electronic device, causes the electronic device to perform the method described above.

[0013] This application provides a computer program product, including a computer program stored in a computer-readable storage medium. The processor of an electronic device reads and executes the computer program from the computer-readable storage medium, causing the electronic device to perform the method described above.

[0014] In this embodiment, the query text to be processed is input into a language model, which generates an answer text for the query text. This language model is trained on an initial model using accuracy comparison samples and efficiency comparison samples. The accuracy comparison samples are constructed based on the answer accuracy of each sample reasoning path in the initial model. Therefore, the accuracy comparison samples can strengthen the initial model's learning of correct reasoning paths, improving the reliability and accuracy of the trained language model's answers. In other words, the accuracy comparison samples can enhance the initial model's ability to recognize correct answers, resulting in a high recognition capability for correct answers in the trained language model. The efficiency comparison samples are constructed based on the retrieval efficiency of the reasoning paths of target samples with similar answer accuracy. Therefore, the efficiency comparison samples can optimize the retrieval path, reduce unnecessary retrieval steps, and improve the response speed of the language model. In other words, the efficiency comparison samples can guide the initial model to select a more efficient retrieval strategy while ensuring answer quality, resulting in a higher retrieval efficiency for the trained language model. In this way, by training with a balance between accuracy and efficiency, the language model trained can take into account both the accuracy of the answer and the efficiency of retrieval. Thus, the language model can quickly generate the answer text corresponding to the query text while ensuring the accuracy of the answer. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings…

[0016] Figure 1 is a schematic diagram of the implementation environment of the text processing method provided in the embodiments of this application;

[0017] Figure 2 is a flowchart of a text processing method provided in an embodiment of this application;

[0018] Figure 3 is a flowchart of a language model training method provided in an embodiment of this application;

[0019] Figure 4 is a flowchart of another language model training method provided in an embodiment of this application;

[0020] Figure 5 is a schematic diagram of a dynamic adjustment of the sampling ratio provided in an embodiment of this application;

[0021] Figure 6 is a flowchart of another language model training method provided in an embodiment of this application;

[0022] Figure 7 is a schematic diagram of the training structure of a language model provided in an embodiment of this application;

[0023] Figure 8 is a flowchart of the training process of the language model provided in an embodiment of this application;

[0024] Figure 9 is a schematic diagram of the application interface in a translation scenario provided in an embodiment of this application;

[0025] Figure 10 is a structural block diagram of the language model training device provided in an embodiment of this application;

[0026] Figure 11 is a structural block diagram of the text processing device provided in an embodiment of this application;

[0027] Figure 12 is a schematic diagram of the structure of the computer system provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. The block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in at least one of different networks, processor devices, and microcontroller devices. The flowcharts shown in the drawings are merely illustrative and do not necessarily include all contents and operations, nor do they necessarily have to be performed in the described order. For example, some operations may be decomposed, while others may be combined or partially combined; therefore, the actual order of execution may change depending on the actual situation.

[0030] It should also be noted that "multiple" as mentioned in this application refers to two or more. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the following description, "some embodiments" are referred to, which describes a subset of all possible embodiments; however, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application belong. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0031] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0032] Before describing the language model training method and text processing method provided in the embodiments of this application, the technical terms involved in the embodiments of this application will be explained first.

[0033] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0034] 2) Large Language Model (LLM): Also referred to as a language model in this embodiment, it refers to a type of deep learning model trained on large-scale text data, with the number of parameters usually in the billions or even hundreds of billions, and has powerful language understanding and generation capabilities.

[0035] 3) Adaptive Retrieval-Augmented Generation (Adaptive RAG): This method dynamically determines whether to call external knowledge bases for information retrieval during the generation of a large language model. This method not only enhances the answer quality of the large language model but also improves overall efficiency through flexible retrieval strategies.

[0036] 4) Direct Preference Optimization (DPO): This is a method for optimizing language models based on human preference feedback. DPO directly optimizes the alignment between the language model output and human preferences by transforming human preference judgments into the training objectives of the language model, thereby improving the quality and acceptability of the content generated by the language model.

[0037] 5) Accuracy Comparison Samples: These are samples constructed based on the accuracy of responses from different reasoning paths generated by the language model during the model's reasoning process. They are used to guide the language model in identifying which reasoning paths are more likely to produce the correct answer.

[0038] 6) Efficiency Comparison Samples: These samples are constructed based on the retrieval efficiency (e.g., number of retrievals, retrieval time) of multiple inference paths generated by the language model, with similar answer accuracy. Efficiency comparison samples help the language model learn to select the optimal retrieval strategy while ensuring answer quality.

[0039] 7) Retrieval efficiency: This refers to the retrieval resources consumed by the language model when completing a query task, and is usually measured by metrics such as the number of searches and retrieval time. Improving retrieval efficiency helps reduce computational overhead and speed up response time.

[0040] 8) Model Policy: This refers to the behavior of the language model currently being trained, including decision-making logic such as how to choose a retrieval path, when to perform a retrieval, and how to integrate retrieval results.

[0041] 9) Reference Policy: This is a language model behavior that serves as a benchmark for comparison. It is usually provided by previous versions of the language model or other stable models and is used to evaluate the improvement effect of the current model's policy.

[0042] While retrieval augmentation methods can improve the accuracy of model responses to some extent, they still fall short in terms of the flexibility and efficiency of retrieval strategies. For example, some methods rely on preset thresholds or classifiers to determine whether to perform a retrieval, lacking adaptability; others focus on optimization in a single aspect (such as focusing only on accuracy), neglecting efficiency issues, leading to serious resource waste in practical applications.

[0043] To address these issues, embodiments of this application propose a language model training method, a text processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The language model training method constructs a preference dataset containing accuracy comparison samples and efficiency comparison samples, and trains an initial model using a direct preference optimization method to obtain a language model. This language model enables it to autonomously decide whether to perform a retrieval and select the optimal retrieval path while ensuring answer quality. In this way, embodiments of this application achieve a balance between the flexibility and efficiency of the retrieval strategy, significantly improving the retrieval efficiency and answer accuracy of the language model.

[0044] The technical solutions of the embodiments of this application will be described in detail below.

[0045] Please refer to Figure 1, which is a schematic diagram of the implementation environment of the text processing method provided in this application embodiment. The implementation environment includes a terminal (terminal 10-1 and terminal 10-2 are shown as examples in Figure 1) and a server 20.

[0046] The terminal acquires the query statement to be processed (i.e., the query statement to be answered) and sends it to the server 20. The server 20 invokes a pre-trained language model to generate the answer text for the query statement. This language model is trained on an initial model based on accuracy comparison samples and efficiency comparison samples. The accuracy comparison samples are constructed based on the answer accuracy of each inference path of the initial model, while the efficiency comparison samples are constructed based on the retrieval efficiency of the inference path of target samples with similar answer accuracy. In implementation, the server 20 inputs the query statement into the pre-trained language model, which then generates the answer text corresponding to the query statement. After generating the answer text, the server 20 can also send the answer text to the terminal for further processing, such as displaying the answer text to the target object.

[0047] In some embodiments, server 20 may also automatically obtain the query statement and then generate the answer text of the query statement based on the language model.

[0048] In some embodiments, the terminal can also implement the text generation process independently, that is, the terminal obtains the model parameters of the pre-trained language model, stores the model parameters of the language model locally, then obtains the query statement, and calls the language model to generate the answer text of the query statement.

[0049] It should be noted that the aforementioned terminals can be electronic devices such as smartphones, tablets, laptops, computers, smart voice interaction devices, smart home appliances, in-vehicle terminals, and aircraft. Server 20 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This section does not impose any restrictions on these aspects. The terminal and server 20 establish a communication connection beforehand via a network, enabling them to communicate with each other. The network can be a wired network or a wireless network; this section also does not impose any restrictions.

[0050] It should be noted that in the specific implementation of this application, if the query statement is related to the target object, when the embodiments of this application are applied to specific products or technologies, permission or consent from the target object is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0051] It should be noted that the text processing method provided in this application can be applied to a variety of different scenarios, such as complex question-and-answer scenarios in intelligent customer service systems, research assistance and literature analysis scenarios in professional fields, personalized content creation and marketing copy generation scenarios, code generation and programming assistance scenarios, etc. These exemplary scenarios are described below.

[0052] In complex question-answering scenarios within intelligent customer service systems, such as those in financial, telecommunications, or e-commerce platforms, users frequently raise complex questions involving multi-step reasoning, such as, "My phone, which I bought last month, is now malfunctioning and won't turn on. How should I handle the warranty claim, and can you provide information on nearby repair shops?" Traditional customer service robots may struggle to handle such complex questions that involve product malfunctions, warranty policies, and location queries coherently. However, the text processing method provided in this application can accurately and efficiently generate answer texts for these complex user questions. During the generation of the answer text, the electronic device (such as a customer service server) can receive the user's input of the aforementioned natural language question as query text and input this query text into a trained language model. The language model can be trained by directly optimizing its performance using accuracy comparison samples (e.g., paths that correctly connect warranty policies and repair shop queries, and paths that incorrectly handle warranty policies) and efficiency comparison samples (e.g., paths that obtain all necessary information in a single retrieval, and paths that require multiple, redundant retrievals). This enables the language model to make autonomous decisions when faced with such complex questions. When the language model internally generates the response text through reasoning, it can generate the keyword "think" and infer that the warranty conditions need to be confirmed first. Next, the language model can generate the keyword "search" and automatically generate a query such as "Mobile phone warranty policy: Unable to power on," retrieving relevant information from a knowledge base (such as a product database). After obtaining the warranty information, the language model continues reasoning, generating the keyword "search" and the query "Find the nearest official repair center based on the user's IP address," retrieving information from a geographic information database. Finally, the language model generates the keyword "response," combining the results of the two searches to generate an accurate, natural, and complete response text: "According to the warranty policy, your situation meets the free warranty conditions. The nearest official repair center is located at XX Road, XX Number, and the contact number is XXXX. We suggest you make an appointment by phone before going." In this scenario, the text processing method provided in this application avoids the rigid responses of traditional customer service systems based on fixed processes and is superior to time-consuming solutions that perform full-text searches regardless of the complexity of the question. It intelligently and efficiently combines multiple searches and reasoning to generate accurate answers, improving the user experience.

[0053] In research support and literature analysis scenarios within specialized fields, such as medicine and law, researchers need to quickly understand cutting-edge concepts or cases. For example, a medical researcher might want to understand "the latest advances of bispecific antibodies in tumor immunotherapy and their combined effect with PD-1 inhibitors." Using the text processing method provided in this application, an electronic device (such as a research support software terminal) can acquire the professional query text input by the researcher and then input it into a trained language model. This language model uses professional literature databases (such as PubMed and legal case databases) as its knowledge retrieval source and is trained using a preference dataset. Accuracy comparison samples ensure that the literature retrieved and cited by the model is relevant and accurate; efficiency comparison samples train the model to find core information with the fewest retrievals from a large amount of literature. The reasoning process within the language model can be as follows: first, generate the keyword "thinking," and decompose the complex question into sub-questions such as "the principle of bispecific antibodies," "the latest clinical research," and "combination with PD-1 inhibitors." For each sub-question, the language model autonomously decides whether to conduct a retrieval. For example, for the "principles" section, the language model can directly generate "thinking" content based on existing knowledge; while for "latest clinical research," it generates "search" keywords, querying "bispecific antibody - phase III clinical trial." Furthermore, the language model evaluates the sufficiency of the search results; if insufficient, it may trigger a new round of searching. Finally, the language model generates "answer" keywords and a well-structured, review-style answer text that cites key literature, including the latest advancements, combined effects, and potential research directions. In this scenario, the text processing method provided in this application not only provides accurate answers, but more importantly, its "search decision-making" capability greatly improves research efficiency. It avoids the tedious process of researchers manually conducting multiple, trial-and-error searches, acting like an experienced assistant, intelligently and efficiently traversing the knowledge base to directly generate high-quality comprehensive analysis reports.

[0054] In personalized content creation and marketing copy generation scenarios, for example, marketers need to generate personalized product introduction copy for a newly launched coffee machine targeting different customer groups (such as urban white-collar workers and young families). Therefore, the text processing method provided in this application embodiment can be used. The query text obtained by the electronic device (such as the content creation platform backend) may be "Generate a marketing copy highlighting the convenience and design of a portable coffee machine for urban white-collar workers". This query text is input into a trained language model. The accuracy comparison sample of the training data of this language model includes a comparison between high-quality copy that accurately reflects "convenience" and "design" and copy that deviates from the topic; the efficiency comparison sample includes paths that can efficiently utilize the product database (such as directly searching for coffee machine size and material) to generate selling points, and paths that require redundant retrieval. The reasoning process within the language model can be as follows: the language model generates the keyword "thinking" and analyzes that the core needs of "urban white-collar workers" may be "saving time" and "improving quality of life". Then, the language model can generate the keyword "search" and query product parameters such as "portable coffee machine, size, weight, and material" from the product knowledge base. After obtaining product parameters, the language model, combined with its understanding of the target audience, directly generates "answer" keywords and outputs a precise answer text, such as: "Say goodbye to long queues! This coffee machine, weighing only XXX grams, lets you enjoy hand-drip quality in the office in just one minute. Its minimalist design will enhance your desk and instantly elevate your work style." In this scenario, the text processing method provided in this application ensures that the generated content is not only accurate in its theme (accuracy dimension) but also quickly identifies key product information, avoiding the generation of generic copy (efficiency dimension). This achieves intelligent and precise content creation, significantly improving marketing efficiency.

[0055] In code generation and programming assistance scenarios, for example, if a developer needs to implement a specific function, such as "writing a function in Python that can asynchronously download all images from a given list of URLs and save them to a specified folder," the text processing method provided in this application can be used. The electronic device (such as a programming assistant plugin in an IDE) obtains the natural language description input by the developer as the query text. The electronic device inputs the query text into a trained language model. This language model is trained on a large amount of code and documentation. Its accuracy comparison samples ensure that the generated code is functionally correct and free of syntax errors; the efficiency comparison samples train the language model to intelligently decide whether to retrieve specific API documentation or best practice cases when generating code, rather than blindly generating a large number of potentially invalid code attempts. The reasoning process within the language model can be as follows: the language model generates the keyword "thinking," decomposing the task into "asynchronous HTTP request," "image type recognition," and "file stream writing." Then, for unfamiliar libraries or APIs, the language model can generate the keyword "search," querying "Python asyncio aiohttp image download example." Finally, combining the retrieved code snippets with its own knowledge, the language model generates the keyword "answer," outputting a complete, runnable Python code that conforms to best practices. In this scenario, the text processing method provided in this application not only generates code with high accuracy (can run correctly), but also avoids generating overly generic or potentially erroneous code through efficient "retrieval decision-making," directly providing the most relevant and elegant implementation solution, greatly improving development efficiency.

[0056] The following describes in detail the various implementation details of the technical solutions in the embodiments of this application.

[0057] As shown in Figure 2, which is a flowchart of the text processing method proposed in this application embodiment, the method can be applied to the implementation environment shown in Figure 1. The method can be executed by an electronic device, which can be a server or a terminal. That is, the method can be executed by a terminal or a server, or it can be executed by both a terminal and a server. In this application embodiment, the method is described by example of being executed by a server. The text processing method can include the following steps S210 to S230, which are described in detail below.

[0058] S210, Obtain the query text.

[0059] Query text refers to the question or instruction entered by the user to obtain information; it is the text to be processed or answered. In this embodiment, query text can be simple keywords or phrases, or complex sentences or paragraphs. The form of query text varies in different application scenarios. For example, in a search engine, query text is the keywords or phrases entered by the user to search for relevant information on the internet. For instance, during translation, if the model needs to find a translation it cannot, it should use a retrieval method to obtain a relevant translation result. In a dialogue system, query text is the dialogue content entered by the user, which can be a simple greeting or question, or a complex instruction or request. In a recommendation system, query text may not be directly entered by the user but generated based on contextual information; the system recommends relevant content to the user based on this query text. Query text can originate from the user's input interface, such as web forms, chatbot interfaces, or speech recognition systems. In actual implementation, the server can receive query text from the terminal or other service modules through API interfaces, message queues, WebSockets, etc.

[0060] In this embodiment, the server can receive query text from the terminal. The server can be deployed in a local data center or as a virtual machine instance on a cloud computing platform. For example, when a user uses the WeChat translation function, they send a query request (i.e., query text) to the system regarding the history of artificial intelligence development. After receiving the query text, the server treats it as pending query text for further processing. In practical applications, query text may contain spelling errors, semantic ambiguities, or non-standard expressions. Therefore, after receiving the query text, the server can also perform preliminary cleaning and standardization to improve the processing efficiency and accuracy of the subsequent language model.

[0061] S220. Input the query text into the language model. The language model is obtained by training the initial model based on the accuracy comparison samples and efficiency comparison samples. The accuracy comparison samples are constructed based on the answer accuracy of each sample reasoning path of the initial model. The efficiency comparison samples are constructed based on the retrieval efficiency of the reasoning path of the target sample with similar answer accuracy.

[0062] In this embodiment, the language model refers to a large-scale parametric model based on deep learning technology, capable of understanding, generating, and reasoning about natural language. The language model is a pre-trained model; that is, the language model used in this embodiment is a specially optimized and trained model. Its training data includes not only ordinary text corpora but also specially designed accuracy comparison samples and efficiency comparison samples. The language model is used to answer query texts. For example, the language model performs reasoning based on the query text and existing information, or it performs retrieval based on the query text. Therefore, inputting the query text into the language model yields the corresponding answer text. It is worth noting that the language model is trained on an initial model using accuracy comparison samples and efficiency comparison samples.

[0063] Accuracy comparison samples are constructed based on the answer accuracy of various inference paths in the initial model. The initial model can be an untrained large language model. An inference path refers to the complete process by which the model, starting from receiving an input question, performs inference, retrieval, and answer generation. Each inference path includes multiple intermediate steps and may involve multiple retrieval, inference, and deduction processes. Different sample inference paths in the initial model exhibit varying answer accuracy. Accuracy comparison samples are constructed based on these inference paths with different answer accuracies. In other words, accuracy comparison samples are generated by simulating different inference paths, each corresponding to a different answer. By evaluating the answer accuracy of these inference paths (e.g., using the Exact Match metric), correct and incorrect answer paths are identified, and these correct and incorrect answer paths are used to form the comparison samples. Accuracy comparison samples are used to train the model to identify which inference paths are more likely to produce the correct answer, thereby improving the accuracy of the language model.

[0064] Efficiency comparison samples are constructed based on the retrieval efficiency of inference paths for target samples with similar answer accuracy. Specifically, inference paths with similar answer accuracy are selected from various sample inference paths. Priority is given to inference paths with identical answer accuracy. If no inference paths have identical answer accuracy, then inference paths with similar answer accuracy greater than a similarity threshold are selected. For example, if inference path 1 has an answer accuracy of 99.21% and inference path 2 has an answer accuracy of 99.00%, both inference path 1 and inference path 2 are considered as target sample inference paths. Different target sample inference paths have different retrieval efficiencies; therefore, efficiency comparison samples are constructed based on these different retrieval efficiencies. Thus, efficiency comparison samples, assuming all inference paths can generate correct answers, further compare the resource consumption of each inference path during the retrieval process, including the number of retrievals and retrieval time. By comparing these resource consumption factors, the language model can learn how to select the retrieval strategy with the lowest resource consumption while ensuring answer quality, thereby improving overall processing efficiency.

[0065] Accuracy comparison samples ensure that the language model strengthens its learning of the correct reasoning path through comparison samples, thereby improving the reliability and accuracy of the answer. Efficiency comparison samples ensure that the language model can optimize the retrieval path, reduce unnecessary retrieval steps, and improve response speed while generating accurate answers, thus improving the overall performance of the language model. In this way, by introducing accuracy comparison samples and efficiency comparison samples, the trained language model can balance the accuracy of the answer and the retrieval efficiency, ensuring that it provides more accurate and efficient answers.

[0066] In one example, to further improve the response speed of the language model, the query text can be preprocessed before being input into the language model, such as by removing invalid words or sentences.

[0067] In this embodiment, the language model can be trained using the DPO algorithm. The DPO algorithm is a model optimization method based on human preference feedback. DPO does not rely on traditional reinforcement learning reward mechanisms but adjusts the language model's behavior by directly modeling the distribution of human preferences. The DPO method enables the language model to select the path that best matches the user's expectations from multiple candidate paths, while avoiding the training instability problems that may occur in traditional methods. In actual implementation, the server inputs the query text into the already trained language model. The language model determines whether to call an external knowledge base for retrieval based on its decision logic and generates corresponding search keywords if necessary. Subsequently, the language model integrates the search results and generates the final answer text. The entire process achieves end-to-end adaptive retrieval enhancement generation, improving both the accuracy of the answer and optimizing retrieval efficiency.

[0068] S230. Generate the response text corresponding to the query text using a language model.

[0069] In this embodiment, after inputting the query text into the language model, the response text obtained by the language model after processing the query text can be directly obtained. The response text refers to the reply content generated by the language model based on the content of the query text. The response text can be a statement or paragraph, or it can be a text link or text identifier, etc. Alternatively, the response text can be in the form of a short question and answer or a long explanatory text, depending on the specific needs of the query and the generation capabilities of the language model.

[0070] The response text generated by the language model relies not only on its internal knowledge base but also on external information retrieved in real time. For example, when a user asks about the types of consensus mechanisms in blockchain, the language model might first use a retrieval tool to find the latest information on consensus mechanism classifications, and then combine this with its own knowledge to generate a clear, accurate, and well-structured response. Furthermore, the quality of the response text can be evaluated using various metrics, such as BLEU, ROUGE, and F1 score. To further improve the readability and logical coherence of the responses, the language model can also employ post-processing techniques such as multi-turn generation, summary extraction, and fact-checking.

[0071] In practice, the language model continuously adjusts its retrieval strategy during the generation of response text to ensure optimal response results with limited resources. For example, when dealing with complex questions, the language model can prioritize authoritative knowledge sources; while when dealing with simple questions, it can reduce unnecessary retrieval operations to speed up the response.

[0072] In this embodiment, the query text to be processed is input into a language model, which is trained on an initial model based on accuracy comparison samples and efficiency comparison samples. The accuracy comparison samples are constructed based on the answer accuracy of each inference path of the initial model. The model strengthens its learning of correct inference paths through these samples, improving the reliability and accuracy of the answers. The efficiency comparison samples are constructed based on the retrieval efficiency of the inference paths of target samples with similar answer accuracy. The model optimizes the retrieval path through these samples, reducing unnecessary retrieval steps and thus improving response speed. Through balanced training of accuracy and efficiency, the trained language model can balance answer accuracy and retrieval efficiency. Therefore, the language model can quickly generate answer text corresponding to the query text while ensuring answer accuracy. In other words, the text processing method provided in this embodiment constructs a preference dataset containing accuracy and efficiency comparison samples and trains the language model using a direct preference optimization method. This enables the language model to autonomously decide whether to perform a retrieval and select the optimal retrieval path while ensuring answer quality. The text processing method provided in this embodiment significantly improves the retrieval efficiency and answer accuracy of the language model, providing users with more efficient and accurate information services.

[0073] In this embodiment, a training method for the aforementioned language model is also provided. This language model training method can also be applied to the implementation environment shown in Figure 1. The method can be executed by a terminal or a server, or by both a terminal and a server. In this embodiment, the method is described using the server as an example. As shown in Figure 3, the language model training method can be an additional step based on steps S210 to S230 shown in Figure 2, or it can be a step different from the text processing method shown in Figure 2. The language model training method includes steps S310 to S350. Steps S310 to S350 are described in detail below.

[0074] S310. Obtain at least two sample inference paths generated by the initial model for the sample query text.

[0075] In this embodiment, sample query text is input into an initial model, and at least two sample inference paths generated by the initial model are obtained. In one example, multiple different contexts or conditions can be provided for the sample query text to ensure that each sample query text samples a sufficient number of sample inference paths (e.g., 10-20) and covers different inference strategies. For example, if the sample query text is "What is machine learning?", multiple sample contexts can be provided, such as contexts about algorithms and application scenarios. In another example, temperature parameters during the model inference process can be adjusted to generate multiple inference paths.

[0076] A sample inference path refers to the different decision paths or inference processes taken by the initial model when processing a query task. Each sample inference path can include information such as whether the retrieval module is called, when it is called, and what content is called. By generating multiple sample inference paths, diverse learning samples can be provided for subsequent construction of preference datasets. For example, when dealing with a complex multi-hop problem, the initial model can generate one direct inference path that does not call the retrieval module, and another that calls the retrieval module at a key node to supplement information. These different sample inference paths constitute the sample inference path set.

[0077] This application embodiment, by acquiring multiple sample inference paths, enables the server to more comprehensively evaluate the behavior of the initial model, thereby providing a rich and diverse data source for subsequently constructing a preference dataset to optimize the initial model. The operation of acquiring multiple sample inference paths and constructing a preference dataset helps improve the initial model's ability to learn optimal retrieval strategies in different scenarios.

[0078] S320. Construct an accuracy comparison sample based on the accuracy of the answers to the reasoning paths of each sample.

[0079] In this embodiment, the accuracy of each sample reasoning path's answer is obtained, and then an accuracy comparison sample is constructed based on the sample reasoning paths with different answer accuracies. To enhance the comparability of the samples, sample reasoning paths with higher answer accuracy and sample reasoning paths with lower answer accuracy are selected to construct the accuracy comparison sample. The accuracy comparison sample can be constructed based on the correctness of the answers in each sample reasoning path generated by the initial model. Specifically, if the answer of a sample reasoning path is consistent with the standard answer (i.e., Exact Match (EM) equals 1), the server marks the answer of that sample reasoning path as the correct answer; otherwise, the server marks the answer of that sample reasoning path as the incorrect answer. The server can form a set of accuracy comparison samples using the above method to train the initial model to identify sample reasoning paths with high accuracy. Thus, by distinguishing between the sample reasoning paths of correct and incorrect answers, this embodiment helps the initial model learn under what circumstances to call the retrieval module to improve the accuracy of the answer. This enhances the initial model's ability to judge the necessity of the retrieval module, reduces unnecessary calls to the retrieval module, and improves overall response efficiency.

[0080] In one example, the accuracy comparison sample can be constructed as follows: obtain the reasoning answers of each sample reasoning path; then, for each sample reasoning path, compare the reasoning answer with the standard answer corresponding to the sample query text to determine the accuracy of the answer of each sample reasoning path; then, obtain the positive sample reasoning path whose answer accuracy is greater than a first preset accuracy threshold, and obtain the negative sample reasoning path whose answer accuracy is less than a second preset accuracy threshold; finally, construct the accuracy comparison sample based on the positive sample reasoning path and the negative sample reasoning path.

[0081] Here, a sample reasoning path refers to a combination of different thought processes and retrieval strategies generated by the initial model for the same sample query text. Each sample reasoning path may involve different numbers of knowledge base retrievals, different information integration methods, and different answer generation logics. The reasoning answer is the content of the answer generated by the initial model under different sample reasoning paths. By collecting multiple sample reasoning paths and their corresponding reasoning answers, the performance differences of the initial model under different decisions can be comprehensively evaluated. In practical applications, the server generates multiple sample reasoning paths for each sample query text, for example, by introducing randomness (such as temperature parameters) to encourage the initial model to explore multiple possible decision paths. Subsequently, the content of the reasoning answer generated by each path is recorded, and these reasoning answers are organized into a reasoning answer set, which is used for subsequent accuracy evaluation. This application embodiment introduces a specific quantitative evaluation method, allowing evaluators to quantitatively evaluate different reasoning behaviors of the initial model. Evaluators can identify which reasoning paths are more likely to generate correct answers, and researchers can use this information to optimize the training process.

[0082] During the initial model's processing of sample query text, the output of each sample inference path, i.e., the inference answer, can be recorded. For each sample query text, there is a corresponding standard answer, which is a reference answer generated based on human annotation or a high-precision model, used as a benchmark for comparison. The standard answer is a pre-annotated correct answer, which can be provided by human experts or authoritative knowledge sources. The accuracy of the inference path is evaluated by comparing the matching degree between the model-generated inference answer and the standard answer. Answer accuracy refers to the consistency between the initial model-generated inference answer and the standard answer. The accuracy of the sample inference path can be obtained using EM comparison; if the standard answer and the model-generated answer are completely identical, EM is 1; otherwise, it is 0. Alternatively, a comprehensive score can be given by calculating the lexical overlap between the inference answer and the standard answer; the edit distance between the inference answer and the standard answer can also be calculated. In practice, the server compares the inference answer of each sample inference path with the standard answer and calculates the EM score for that sample inference path. The above comparison and scoring process of sample inference paths can be automated without manual intervention, thus ensuring efficiency and consistency. In this embodiment, the effectiveness of each sample reasoning path can be objectively evaluated, thereby distinguishing between high-quality and low-quality reasoning behavior.

[0083] When comparing the inferred answers with the standard answers corresponding to the sample query text to determine the accuracy of each sample inference path, a real-world example is used. Suppose a query asks what emotion is expressed in the poem "XXX" by poet XX, and the standard answer is homesickness. The initial model may generate multiple sample inference paths, such as: Path A: the system analyzes and draws a conclusion based on retrieved relevant lines of poetry "XXXXX, XXXXX"; Path B: the model analyzes the user input using its internal knowledge base and infers the meaning of expressing loneliness. The server then compares the final inferred answers from these two sample inference paths with the standard answer and judges the accuracy of each path.

[0084] When the accuracy of a sample inference path's answer is higher than a first preset accuracy threshold, the sample inference path is considered a positive sample, indicating that the answer generated by that inference path is correct. When the accuracy of a sample inference path's answer is lower than a second preset accuracy threshold, the sample inference path is considered a negative sample, indicating that the answer generated by that inference path is incorrect. The first and second preset accuracy thresholds can be flexibly adjusted according to actual conditions, but the first preset accuracy threshold is greater than the second preset accuracy threshold. Thus, a sample set composed of positive and negative sample inference paths is obtained as an accuracy comparison sample to help the initial model distinguish between high-accuracy and low-accuracy inference paths. By comparing positive and negative samples, the initial model can learn more effective inference patterns. Through continuous comparison and adjustment, the initial model can improve its inference process from erroneous inference paths, thereby improving the reliability of the final language model's answer.

[0085] Positive inference paths indicate that the initial model generates inference answers that are close to or equal to the standard answer under positive inference paths, exhibiting high reliability. Conversely, negative inference paths indicate that the initial model generates inference answers that deviate from the standard answer, exhibiting lower reliability. In the implementation, the server can sort all sample inference paths according to their EM scores and select paths above a first preset accuracy threshold as positive inference paths, and paths below a second preset accuracy threshold as negative inference paths. Developers can flexibly adjust the first and second preset accuracy thresholds according to the application scenario to achieve a balance between sample coverage and discrimination. This approach allows for the construction of a well-structured accuracy comparison sample set, which helps the initial model identify which inference paths are more likely to generate correct answers and which are more prone to errors.

[0086] Accuracy comparison samples consist of pairs of comparative data, comprising positive and negative inference paths, used to guide the initial model in learning how to select the optimal inference path. Each pair contains one positive and one negative inference path, which together form a training sample used for the initial model's preference learning. During the model's construction, positive inference paths are used as preferred examples, and negative inference paths as secondary examples. By using positive inference paths as preferred examples and negative inference paths as secondary examples, the initial model can learn which inference path is more likely to generate the correct answer under similar conditions. This provides high-quality training data for the direct preference optimization algorithm, ensuring that the initial model gradually improves its decision-making ability based on feedback signals.

[0087] It should be noted that if there are too many sample inference paths with an accuracy higher than the first preset accuracy threshold, it is necessary to further filter out better sample inference paths as positive sample inference paths. In this case, positive sample inference paths can be obtained in the following way: First, sample inference paths with an accuracy higher than the first preset accuracy threshold are taken as candidate sample inference paths; then, if the number of candidate sample inference paths is greater than the preset number, and there are too many sample inference paths with accurate answers, the reliability of the inference process of each candidate sample inference path needs to be further verified. Therefore, it is necessary to obtain the reference inference path generated by the optimized reference model for the sample query text; then, calculate the similarity between the candidate sample inference path and the reference inference path, and determine the candidate sample inference path with a similarity greater than the preset similarity threshold as the positive sample inference path.

[0088] Candidate inference paths are those inference processes that demonstrate high accuracy in the initial evaluation (e.g., greater than the accuracy threshold), meaning the answers generated by the sample inference paths match the standard answer well (EM=1) and meet the set first preset accuracy threshold. The first preset accuracy threshold can be a fixed value (e.g., 0.9) or a dynamically adjusted value that automatically adjusts based on the task type or dataset distribution. By setting the first preset accuracy threshold, sample inference paths with high credibility can be selected as the basis for subsequent optimization.

[0089] When the number of candidate sample inference paths exceeds a preset limit, it indicates the existence of multiple possible high-quality inference methods. At this point, an optimized reference model is introduced to generate a more stable and representative reference inference path. The reference model can be trained on a large-scale preference dataset, possessing higher generalization ability and stability. The reference model can be based on different optimization algorithms, such as DPO, Knowledge and Preference Optimization (KTO), or Indirect Preference Optimization (IPO), selecting the most suitable reference model according to the specific scenario. In this embodiment, the reliability of the candidate sample inference path is verified by the reference inference path generated by the reference model. This reference model refers to a high-performance pre-trained model or a fully optimized model, used as a benchmark for evaluating and comparing the current initial model, such as a GPT model trained on a large-scale general dataset. Input the sample query text into the reference model and obtain at least two reference inference paths generated by the reference model. Then calculate the similarity between each candidate sample inference path and each reference inference path. The cosine similarity between the corresponding vectors of the candidate sample inference path and the reference inference path can be calculated. The larger the cosine similarity, the higher the reliability of the candidate sample inference path. The candidate sample inference path with a similarity greater than the preset similarity threshold represents the path jointly generated with the reference model. Therefore, the candidate sample inference path can be used as the positive sample inference path.

[0090] The sample query text is the original question input by the user. The reference model generates a logically clear, structurally sound, and as close as possible to the optimal answer based on the sample query text. In practice, there can be a certain correlation between the candidate sample inference path and the reference inference path. For example, the reference inference path can be seen as a consensus path among multiple candidate sample inference paths, or an approximate representation of the optimal path by the reference inference path. Therefore, after introducing the reference inference path, the path closest to the optimal solution can be further selected by comparing the similarity between each candidate sample inference path and the reference inference path. In this embodiment, by introducing a standardized reference inference path among a large number of candidate sample inference paths, constraints on the model's decision path can be added during model training, making the initial model output consistent and robust, ultimately helping the initial model learn the optimal strategy more effectively.

[0091] The similarity between a candidate inference path and a reference inference path refers to the degree of similarity between the two paths in terms of content, structure, and logic. Various methods can be used to measure similarity, such as semantic similarity (e.g., using BERTScore, Sentence-BERT), path length ratio, and operation sequence consistency. The purpose of similarity calculation is to identify candidate inference paths that are highly consistent with the reference inference path; such candidate inference paths typically have higher credibility and effectiveness. Furthermore, similarity can also be used as an important basis for determining whether the initial model relies on a specific retrieval pattern or inference logic. In practice, similarity not only reflects the degree of matching between the candidate and reference inference paths in terms of content but can also indirectly reflect the behavioral tendencies of the initial model when generating paths. For example, if a candidate inference path is highly consistent with the reference inference path in terms of retrieval invocation timing and information integration methods, it indicates that this candidate inference path was likely generated by the initial model under normal inference processes, rather than being affected by noise or abnormal input. The embodiments of this application can select candidate sample inference paths that are highly consistent with the reference inference path through quantitative comparison. In this way, the initial model can avoid over-reliance on certain local features or noise. This selection method improves the generalization ability of the initial model, so that the initial model can maintain high performance in different scenarios.

[0092] Positive sample inference paths are high-quality paths further filtered from candidate sample inference paths. Only when the similarity of a candidate sample inference path exceeds a preset similarity threshold is it recognized as a positive sample inference path. The preset similarity threshold can be set according to task requirements, for example, 0.85 or 0.9, meaning that a candidate sample inference path must have more than 85% similarity to the reference inference path to be accepted. This mechanism ensures that the finally selected positive sample inference path not only meets the requirements of high accuracy but also possesses a logical structure and reasoning method consistent with the reference inference path, thereby improving the overall quality and stability of the path.

[0093] S330. Determine the reasoning path of the target samples with similar answer accuracy, and construct an efficiency comparison sample based on the retrieval efficiency of each target sample reasoning path.

[0094] After constructing the accuracy comparison samples, inference paths that are similar in accuracy but differ in retrieval efficiency can be selected. Here, similar answer accuracy refers to the EM values ​​of the inference paths being close; that is, if the absolute value of the difference between the EM values ​​of two inference paths is less than a preset difference threshold, then these two inference paths can be considered target inference paths with similar answer accuracy. For example, both inference paths may have an EM value of 0.9 or higher. Based on this, the retrieval efficiency of the target inference paths with similar answer accuracy is further compared, using metrics such as the number of retrievals and retrieval time, to construct efficiency comparison samples. The purpose of this is to allow the initial model to learn to select more efficient inference paths while ensuring answer quality. This embodiment of the application, by introducing comparison samples with an efficiency dimension, enables the initial model to not only focus on answer accuracy but also select the most efficient solution from multiple high-quality inference paths. This helps to achieve a balance between accuracy and efficiency in the initial model, improving the performance of the trained language model in practical applications.

[0095] In this embodiment of the application, a target sample reasoning path with similar answer accuracy can be selected from the positive sample reasoning path, wherein, preferably, the sample reasoning path with the same answer accuracy; however, the target sample reasoning paths with the same answer accuracy have different retrieval efficiencies. Therefore, in order to enhance the comparability of the samples, a target sample reasoning path with higher retrieval efficiency and a target sample reasoning path with lower retrieval efficiency can be selected to construct an efficiency comparison sample.

[0096] In one example, the efficiency comparison sample can be constructed as follows: First, determine the number of retrievals performed by the initial model in the target sample inference path; then, determine the retrieval efficiency of each target sample inference path based on the number of retrievals; finally, construct the efficiency comparison sample based on the positive target sample inference path with retrieval efficiency greater than a first preset efficiency threshold and the negative target sample inference path with retrieval efficiency less than a second preset efficiency threshold.

[0097] During the initial model's processing of sample query text, the number of times the retrieval steps are triggered in the target sample inference path can be recorded, i.e., the retrieval count. The retrieval count refers to the total number of times the initial model calls external knowledge bases (such as Elasticsearch databases) to retrieve information within a specific target sample inference path. The retrieval count reflects the amount of external information the initial model relies on when processing specific questions in the target sample inference path, and the frequency with which the initial model uses the retrieval mechanism during answer generation. In practice, when the initial model generates search keywords during inference, it performs a retrieval operation and records the keyword generation event. Therefore, the retrieval count can be precisely counted and controlled, serving as an important basis for measuring the efficiency of the target sample inference path. By statistically analyzing the retrieval counts in different target sample inference paths, the resource consumption of each target sample inference path can be quantitatively compared, thus providing fundamental data support for subsequent efficiency evaluation.

[0098] Retrieval efficiency measures the speed and cost at which an initial model retrieves the required information to complete a task. Retrieval efficiency is inversely proportional to the number of retrievals; fewer retrievals indicate a lower dependence on external information when generating answers, and a more efficient reasoning process. To calculate retrieval efficiency, this application embodiment employs a standardization method based on the number of retrievals. Specifically, the number of retrievals for different target sample reasoning paths is mapped to a unified efficiency scoring interval. Each efficiency scoring interval corresponds to a retrieval efficiency, and the retrieval efficiency corresponding to the mapped interval is the retrieval efficiency of the target sample reasoning path. By using a standardization method based on the number of retrievals and mapping it to a unified efficiency scoring interval, it is possible to distinguish which target sample reasoning paths are more computationally efficient while ensuring consistent answer accuracy.

[0099] In this embodiment, the more searches performed, the lower the retrieval efficiency corresponding to the target sample inference path. Therefore, when the retrieval efficiency of the target sample inference path is higher than a first preset efficiency threshold, the target sample inference path is treated as a positive sample, indicating that the inference path is efficient. When the retrieval efficiency of the target sample inference path is lower than a second preset efficiency threshold, the target sample inference path is treated as a negative sample, indicating that the inference path is inefficient. The first and second preset efficiency thresholds can be flexibly adjusted according to actual conditions, but the first preset efficiency threshold is greater than the second preset efficiency threshold. Thus, a sample set composed of positive and negative sample inference paths is obtained as an efficiency comparison sample to help the initial model distinguish between efficient and inefficient inference paths. By comparing positive and negative samples, the initial model can learn more effective retrieval patterns.

[0100] Positive target sample inference paths refer to those target sample inference paths whose retrieval efficiency is higher than a first preset efficiency threshold. Positive target sample inference paths represent achieving high retrieval efficiency while ensuring answer quality. Negative target sample inference paths refer to those target sample inference paths whose retrieval efficiency is lower than a second preset efficiency threshold. While negative target sample inference paths can also provide correct answers, they incur higher retrieval costs. Efficiency comparison samples, composed of positive and negative target sample inference paths, are used to guide the initial model in learning how to select the optimal path from among multiple possible target sample inference paths. By training using efficiency comparison samples, the initial model can prioritize target sample inference paths with higher efficiency when facing new problems, thereby improving the overall operating efficiency and response speed of the initial model. In this embodiment, by introducing the concepts of retrieval count and retrieval efficiency, and constructing efficiency comparison samples based on efficiency, the efficiency differences between different target sample inference paths can be effectively identified. This allows for optimization of the initial model's decision logic, thereby significantly reducing computational resource consumption while maintaining answer accuracy.

[0101] In one example, to more accurately reflect the actual consumption of the retrieval operation, a time weight can be introduced into the number of retrievals, and the time consumption of each retrieval can be included in the retrieval cost. Therefore, the retrieval efficiency can be determined as follows: First, obtain the retrieval time for each retrieval and determine the total retrieval time corresponding to the target sample inference path based on the retrieval time; then, calculate the retrieval efficiency of the target sample inference path based on the number of retrievals and the total retrieval time.

[0102] Here, retrieval time is the time difference between sending the request and receiving the response for each retrieval call; that is, the time consumed by the initial model in executing each retrieval operation. Retrieval time can include all the time elapsed from generating the retrieval request to receiving the retrieval response, such as network latency and database query time. In practical applications, logging modules or performance testing tools can automatically record the specific time of each retrieval and use it as the basis for subsequent evaluation of the retrieval efficiency of the target sample inference path. In some embodiments, the server can employ a high-precision timer to obtain more accurate retrieval times for refined analysis. Furthermore, to improve comparability, the server can also perform standardization operations on the same retrieval tasks in different target sample inference paths, ensuring the comparability of time statistics for the target sample inference paths.

[0103] Determining the total retrieval time for a target sample inference path based on retrieval time involves summing the times of individual retrievals to obtain the total time consumed by each target sample inference path throughout the entire retrieval process. The total retrieval time reflects the overall efficiency of the target sample inference path and is a crucial indicator for evaluating the quality of a retrieval strategy. In practice, this can be achieved by traversing all nodes that trigger retrieval operations within the current target sample inference path and summing the retrieval times of each node to obtain the total retrieval time for the current target sample inference path. If no retrieval operation is triggered in a target sample inference path, the total retrieval time for that path is zero, indicating that the path relies entirely on the model's internal knowledge to answer the question.

[0104] The total retrieval time corresponding to the target sample inference path is obtained by summing the retrieval times of each retrieval. The retrieval efficiency can be obtained by weighted summation of the number of retrievals and the total retrieval time. Retrieval efficiency refers to the average time required to complete the task per unit of retrievals, and is used to evaluate the optimization degree of different target sample inference paths in terms of time and resource utilization. For example, the retrieval efficiency can be calculated using the following formula (1):

[0105] Where α and β are the corresponding weights; the number of searches i This represents the number of retrievals performed by the initial model in the inference path for the i-th target sample; retrieval time. i This represents the total retrieval time corresponding to the inference path of the i-th target sample; n is the total number of inference paths for the target samples.

[0106] Introducing retrieval time makes retrieval efficiency more accurate. For example, a high-efficiency target sample inference path is one with fewer retrievals and shorter retrieval times per retrieval; a low-efficiency target sample inference path is one with more retrievals or longer retrieval times per retrieval. Using the above formula (1), the retrieval efficiency of different target sample inference paths can be quantitatively compared, and the target sample inference path with the highest retrieval efficiency can be selected as the final decision-making basis. For example, if both target sample inference paths can correctly answer the question, the target sample inference path with higher retrieval efficiency can be selected, thereby reducing the overall response time and improving the user experience.

[0107] In this embodiment of the application, if there are multiple target sample inference paths with retrieval efficiency greater than the first preset efficiency threshold, in order to further find target sample inference paths with more obvious efficiency comparisons, the target sample inference path corresponding to highly relevant content can be taken as a positive sample target inference path based on the retrieval content returned in each retrieval in the target sample inference path; or the target sample inference path corresponding to irrelevant content can be taken as a negative sample target inference path from the target sample inference paths with retrieval efficiency less than the second preset efficiency threshold.

[0108] S340. Sample the accuracy comparison samples and efficiency comparison samples to obtain the preference dataset.

[0109] In this embodiment of the application, in a single training session, the sampled samples of accuracy comparison samples and efficiency comparison samples are mixed according to the sampling ratio in each round of training to generate the preference dataset corresponding to the training batch, so as to form training data that takes into account both accuracy and efficiency, and then train the initial model to obtain the language model.

[0110] To ensure that the initial model learns a balance between accuracy and efficiency during training, this embodiment employs weighted sampling, mixing accuracy comparison samples and efficiency comparison samples in a certain proportion to form the final preference dataset. For example, this embodiment can set the accuracy sample ratio to 60% and the efficiency sample ratio to 40%. By adopting this weighted sampling strategy, it is ensured that the initial model focuses more on accuracy in the early stages of training, and gradually increases its focus on efficiency in later stages. By reasonably configuring the sampling ratio and sampling the accuracy comparison samples and efficiency comparison samples based on the sampling ratio to obtain the preference dataset, the learning focus of the initial model at different stages can be controlled, allowing the initial model to gradually improve retrieval efficiency while ensuring answer quality.

[0111] In some embodiments, a dynamic adjustment mechanism can also be adopted to adjust the sampling ratio during each training session. This helps the initial model to optimize multiple objectives simultaneously in a single training process, thereby avoiding the suboptimal solution problem caused by staged training in traditional methods.

[0112] S350. Train the initial model based on the preference dataset to obtain the language model.

[0113] After obtaining the preference dataset, the DPO algorithm can be used to train the initial model. In this embodiment, DPO is used to guide the initial model to learn how to make optimal decisions among multiple sample inference paths. During training, the initial model gradually tends to select sample inference paths that are both accurate and efficient, thereby improving the adaptive retrieval capability of the initial model. Through DPO training based on the preference dataset, the initial model can autonomously learn the optimal retrieval strategy in different contexts, thus significantly improving the performance of the initial model in complex tasks. The DPO training method based on the preference dataset not only improves the generalization ability of the initial model but also reduces the dependence on external classifiers or rule engines, reducing the overall complexity of the system.

[0114] In practical implementation, steps S310 to S350 are closely logically related. First, a diverse learning foundation is established by generating multiple sample inference paths. Next, by analyzing and comparing the accuracy and efficiency of these sample inference paths, two types of comparison samples are constructed. Then, these two are combined and sampled proportionally to form a comprehensive preference dataset. Finally, the initial model is trained using the preference dataset, enabling it to automatically select the optimal sample inference path based on the input query during actual operation. This entire process forms a closed-loop model optimization mechanism, improving the initial model's adaptability and overall performance in retrieval augmentation generation tasks. In summary, in this embodiment, by constructing a preference dataset encompassing both accuracy and efficiency dimensions and training the initial model using the DPO algorithm, the initial model's decision-making ability in retrieval augmentation generation tasks can be effectively improved. By constructing a preference dataset and training the model using the DPO algorithm, the initial model can be guided to select the optimal retrieval strategy while ensuring answer quality, thereby reducing system overhead and improving user experience, ultimately achieving an efficient and flexible adaptive retrieval augmentation generation framework.

[0115] In this embodiment, the reasoning answers of each sample reasoning path are compared with the standard answers corresponding to the sample query text to determine the accuracy of the answers. Then, high-quality accuracy comparison samples are constructed based on the sample reasoning paths with significantly different answer accuracy. Furthermore, the reliability of positive samples can be further ensured by using the similarity of the reference reasoning paths of the reference model, thus improving the quality of the accuracy comparison samples. In addition, retrieval efficiency is determined based on the number of retrievals. High-quality efficiency comparison samples are constructed based on the target sample reasoning paths with significantly different retrieval efficiencies. Retrieval time can be incorporated into the number of retrievals to more accurately reflect the actual consumption of the retrieval operation, improving the accuracy of the retrieval efficiency determination and further enhancing the quality of the efficiency comparison samples.

[0116] It should be noted that the sampling ratios of accuracy comparison samples and efficiency comparison samples are dynamically adjusted throughout the training process, enabling the initial model to simultaneously achieve both accuracy and efficiency goals. This application also provides another language model training method, which can also be applied to the implementation environment shown in Figure 1. Taking the method executed by the server as an example, as shown in Figure 4, based on Figure 3, steps S340 to S350 are extended to steps S410 to S440, which include the specific process of training the initial model to obtain the language model. Steps S410 to S440 include:

[0117] S410. In the current training batch, adjust the sampling ratio of accuracy comparison samples and efficiency comparison samples according to the current training progress.

[0118] During training, as the performance of the initial model improves, the initial model's need for accuracy may gradually decrease, while the need for efficiency optimization will increase. Therefore, in each training batch, this embodiment dynamically adjusts the sampling ratio of accuracy comparison samples to efficiency comparison samples according to the current training stage. For example, in the early stages of model training, more attention is paid to the accuracy of the answers, so the sampling ratio of accuracy comparison samples is greater than that of efficiency comparison samples; in the later stages of model training, the optimization of retrieval efficiency is gradually strengthened, so the sampling ratio of efficiency comparison samples is greater than that of accuracy comparison samples. The process of dynamically adjusting the sampling ratio is completed in different iteration stages of a single training task. This dynamic adjustment mechanism enables the initial model to prioritize learning the most relevant optimization objectives at different stages of training, thereby avoiding the training bias problem caused by a fixed ratio. In actual implementation, there is a complementary relationship between accuracy comparison samples and efficiency comparison samples. Accuracy comparison samples ensure the reliability of the initial model's output, while efficiency comparison samples optimize resource utilization efficiency. By gradually reducing the weight of accuracy comparison samples and increasing the weight of efficiency comparison samples during training, the method of this embodiment can significantly improve overall operating efficiency without sacrificing basic accuracy.

[0119] In this embodiment, the sampling ratio can be dynamically adjusted linearly according to the current training progress. In one example, a first sampling ratio set at the start of training for accuracy comparison samples and a second sampling ratio for efficiency comparison samples are obtained, with the first sampling ratio being greater than the second sampling ratio. Based on the total number of training rounds and the current training progress, the first sampling ratio is reduced to obtain a third sampling ratio, and a fourth sampling ratio for efficiency comparison samples is determined based on the third sampling ratio. Here, the first sampling ratio indicates that in the early stages of training, the initial model has a higher sampling weight for accuracy comparison samples, meaning the initial model tends to use accuracy comparison samples for learning to prioritize improving the accuracy performance of the initial model. The second sampling ratio indicates that the initial model has a lower sampling weight for efficiency comparison samples in the early stages of training, because the initial model does not yet have a sufficient accuracy foundation, and focusing on efficiency too early may lead to unstable performance of the initial model. In the early stages of training, researchers set the first sampling ratio higher than the second sampling ratio to ensure that the initial model can prioritize mastering the correct retrieval strategy and to prevent a decrease in the overall stability of the initial model due to the premature introduction of efficiency optimization measures.

[0120] The total number of training epochs refers to the total number of times the model updates its parameters throughout the entire training process. The total number of training epochs is related to the training cycle and can be used to measure the progress of training. The current training progress refers to the relative position of the initial model in the training process; for example, epoch 100 is relatively early compared to a total of 1000 epochs. The first sampling ratio gradually reduces the initial model's reliance on accuracy comparison samples as training progresses, causing the initial model to shift its focus away from solely improving accuracy and towards efficiency optimization. The third sampling ratio is the adjusted sampling weight for accuracy comparison samples, and its value is lower than the initial first sampling ratio. The fourth sampling ratio refers to the change in the sampling weight of efficiency comparison samples after the sampling ratio of accuracy comparison samples is adjusted. Because the decrease in the first sampling ratio means that the initial model's focus on accuracy is reduced, the initial model can appropriately increase its focus on efficiency comparison samples to further optimize retrieval efficiency while maintaining a certain level of accuracy.

[0121] The sampling ratio of the accuracy comparison sample is p. acc p ret The sampling ratio p of the efficiency comparison samples ret =1-p acc At the start of training, set p acc >p ret , such as p acc =0.9, p ret =0.1; where the total number of training rounds T is preset, and the current training progress is the number of training rounds t. Based on t and T, the sampling ratio increases linearly from the first sampling ratio and decreases linearly from the second sampling ratio, so that as training progresses, the sampling ratio gradually tends towards optimizing retrieval efficiency. p is dynamically calculated based on t and T. acc and p ret ,like p ret =1-p acc , where p acc_min To ensure accuracy, the minimum sampling ratio of the comparison samples is used to maintain the model's baseline accuracy, even in later stages of training; for example, setting p in the last round... acc <p ret , such as p acc_min =0.3.

[0122] In this embodiment of the application, if a fixed sampling ratio and a minimum value of the sampling ratio of accuracy comparison samples are not set at the beginning of training, then

[0123] S420. Based on the adjusted sampling ratio, compare the sampling accuracy samples and efficiency samples to obtain the preferred dataset for the current training batch.

[0124] Based on the adjusted sampling ratio, a certain number of samples can be randomly selected from existing accuracy comparison sample libraries and efficiency comparison sample libraries to form the preference dataset for the current training batch. This preference dataset serves as the basis for updating the initial model parameters, ensuring that each round of training covers the latest learning objectives. In this embodiment, by introducing diverse training data generation strategies, the initial model can receive diverse training data at different training stages, thereby achieving a more comprehensive performance improvement.

[0125] S430. Update the model parameters of the initial model according to the preference dataset of the current training batch to obtain the training model under the current training batch.

[0126] After obtaining the preference dataset for the current training batch, the DPO algorithm can be used to update the initial model parameters.

[0127] In this embodiment, updating the model parameters of the initial model based on the preference dataset of the current training batch can be achieved in the following way: In response to the current training batch being equal to 1, the model parameters of the initial model are updated based on the preference dataset of the first training batch to obtain the training model under the first training batch; wherein, the preference dataset of the first training batch is a dataset obtained by sampling accuracy comparison samples using a first sampling ratio and sampling efficiency comparison samples using a second sampling ratio; the first sampling ratio is the sampling ratio for accuracy comparison samples set at the beginning of training, and the second sampling ratio is the sampling ratio for efficiency comparison samples set at the beginning of training; In response to the current training batch being equal to n, and n being greater than 1, the training model under the (n-1)th training batch is determined as the initial model under the nth training batch; and the model parameters of the initial model under the nth training batch are updated based on the preference dataset of the nth training batch to obtain the training model under the nth training batch.

[0128] Here, during the first training batch, the preference dataset from this first batch is used to update the initial model's parameters, thus generating the trained model for that batch. The preference dataset for the first training batch consists of samples from two dimensions: accuracy comparison samples and efficiency comparison samples. To ensure the initial model learns information related to both retrieval accuracy and efficiency, a first sampling ratio and a second sampling ratio are used to sample the accuracy comparison samples and efficiency comparison samples, respectively. The first sampling ratio controls the proportion of accuracy comparison samples, while the second sampling ratio controls the proportion of efficiency comparison samples. Both the first and second sampling ratios are set at the start of training to ensure the initial learning signal has a clear directionality. The first sampling ratio refers to a sampling weight set for the accuracy comparison samples at the beginning of training, determining their proportion in the preference dataset of the first training batch. By adjusting the first sampling ratio, the focus of the initial model during the early stages of training can be controlled—whether optimization is focused on answer accuracy or retrieval efficiency. For example, in the first training batch, a higher first sampling ratio indicates that the initial model prioritizes improving response accuracy; conversely, a lower first sampling ratio indicates a greater focus on improving retrieval efficiency. The second sampling ratio is similar to the first, but it applies to efficiency comparison samples. The second sampling ratio determines the proportion of efficiency comparison samples in the preference dataset of the first training batch, thus influencing the initial model's focus on retrieval efficiency during training. In practical applications, these two ratios can be dynamically adjusted to allow the initial model to gradually achieve a balance between accuracy and efficiency at different training stages.

[0129] When a training batch enters the nth training batch (n>1), the training model from the (n-1)th training batch is used as the initial model for the nth training batch. Subsequently, the parameters of the initial model in the nth training batch are updated based on the preference dataset of the nth training batch, generating a new training model. By adopting this progressive training method, the initial model in the nth training batch can be continuously optimized based on existing knowledge, avoiding starting from scratch in each training session, thereby improving training efficiency and enhancing model performance. The training model in the (n-1)th training batch refers to the model version with updated parameters from the previous training stage. This updated model version forms the basis of the initial model in the nth training batch, ensuring the initial model maintains consistency and stability during continuous training. By inheriting the model state from the (n-1)th training batch, unnecessary redundant computations are reduced, training speed is accelerated, and the initial model's adaptability to complex tasks is enhanced. The preference dataset for the nth training batch is dynamically constructed based on the needs of the current training stage. The preference dataset in the nth training batch still includes accuracy comparison samples and efficiency comparison samples. The sampling ratio of accuracy comparison samples and efficiency comparison samples can be adjusted according to the training process to adapt to different learning objectives. For example, in the middle of training, the proportion of efficiency comparison samples can be gradually increased to help the initial model further optimize the retrieval strategy; while in the later stage of training, the sampling ratio of accuracy comparison samples and efficiency comparison samples can be readjusted to ensure that the initial model can balance accuracy and efficiency in the final output.

[0130] Through the above method, the initial model can be optimized in each training batch based on the knowledge accumulated in the previous stage, forming a closed-loop training process of continuous improvement. This approach improves the overall performance of the initial model and enhances its generalization ability and robustness in real-world application scenarios. In summary, in this embodiment, by introducing a first sampling ratio and a second sampling ratio, a preference dataset containing accuracy comparison samples and efficiency comparison samples is constructed, and the model parameters of the initial model are gradually updated in different training batches. This effectively guides the initial model to prioritize learning accurate answers in the early stages of training, and then gradually transitions to optimizing retrieval efficiency, thereby achieving a balance between accuracy and efficiency in the initial model and further improving its overall decision-making ability and adaptability.

[0131] In practice, the entire training process is a phased and progressively optimized process. First, by setting an initial sampling ratio, a clear learning signal is provided to the initial model in the first training batch, enabling the initial model to quickly master basic judgment capabilities. Subsequently, in subsequent training batches, the sampling ratio is dynamically adjusted based on the performance of the initial model, thereby enabling the initial model to continuously improve efficiency while maintaining accuracy. Finally, by continuously iterating and updating the model parameters of the initial model, the model performance is continuously improved, ultimately achieving the goal of balancing accuracy and efficiency in practical applications.

[0132] S440. Repeat the step of adjusting the sampling ratio until the total number of training rounds is reached, and determine the training model under the total number of training rounds as the language model.

[0133] After updating the parameters for each training batch, the performance of the current trained model is re-evaluated, and the sampling ratio of accuracy comparison samples and efficiency comparison samples is readjusted before proceeding to the next training batch. This iterative training process, based on model performance evaluation and sample ratio adjustment after parameter updates, continues until the preset total number of training rounds is reached. Finally, after all training rounds are completed, the trained model obtained from the last training iteration is determined as the final language model. The final language model possesses high retrieval accuracy and low retrieval cost, effectively supporting complex question-answering tasks.

[0134] In this embodiment, a preference dataset is constructed and the model is trained by dynamically adjusting the sampling ratio based on accuracy comparison samples and efficiency comparison samples. This process balances the initial model's performance in terms of answer accuracy and retrieval efficiency, thereby improving the overall decision-making ability of the initial model and achieving adaptive retrieval enhancement generation. In actual implementation, the various steps form a closed-loop training process: first, the sampling ratio is adjusted according to the current training state; then, a new preference dataset is constructed; next, the initial model parameters are updated using the new preference dataset; finally, the performance of the trained model is evaluated, and a decision is made on whether to continue to the next training batch. This closed-loop training process ensures that the initial model continuously optimizes its overall performance during training, thus balancing accuracy and efficiency.

[0135] In summary, this application provides a model training method that combines accuracy and efficiency optimization objectives. By dynamically adjusting the sampling ratio, constructing a preference dataset, and using the DPO algorithm for parameter updates, it achieves efficient training of the language model. This method can adaptively select the optimal retrieval strategy when the language model faces complex query tasks.

[0136] In this embodiment of the application, as shown in Figure 5, in the initial training phase 501, the sampling ratio is 8:2 to generate the initial training preference dataset; in the middle training phase 502, the sampling ratio is 5:5; and in the later training phase 503, the training ratio is 1:9. The generation of the preference datasets from different training batches is real-time; it is not a process of generating a single dataset before training, but rather a dynamic adjustment of the sampling strategy and its direct use in training.

[0137] Input the preference dataset of the current training batch into the initial model, calculate the model loss, and then update the model parameters using gradient descent based on the model loss at the current training progress. Repeat steps S410 to S440 until the current training round t reaches the total training round T.

[0138] In one example, updating the model parameters of the initial model can be achieved as follows: input the preference dataset of the current training batch into the initial model; then, for the accuracy comparison samples in the preference dataset of the current training batch, calculate the accuracy comparison loss between the positive sample inference path and the negative sample inference path based on the output of the model policy of the initial model and the reference policy; and, for the efficiency comparison samples in the preference dataset of the current training batch, calculate the efficiency comparison loss between the positive target sample inference path and the negative target sample inference path based on the output of the model policy of the initial model and the reference policy; finally, update the model parameters of the initial model based on the accuracy comparison loss and the efficiency comparison loss.

[0139] In this embodiment, after inputting the preference dataset of the current training batch into the initial model, the model strategy of the initial model generates a first output for the positive sample inference path in the sample query text and accuracy comparison sample, and the reference strategy of the reference model generates a second output for the positive sample inference path in the sample query text and accuracy comparison sample; the model strategy of the initial model generates a third output for the negative sample inference path in the sample query text and accuracy comparison sample, and the reference strategy of the reference model generates a fourth output for the negative and positive sample inference paths in the sample query text and accuracy comparison sample. Based on the difference between the first and third outputs, and the difference between the second and fourth outputs, an accuracy comparison loss is constructed.

[0140] Accuracy comparison samples are a portion of the preference dataset and consist of a pair of inference paths. Each accuracy comparison sample includes a positive inference path that leads to the correct answer and a negative inference path that leads to an incorrect answer. The model strategy refers to the decision logic adopted by the current initial model, while the reference strategy refers to a pre-defined benchmark model or historical version model used to provide a comparison standard. By comparing the outputs of the model strategy and the reference strategy when processing the same accuracy comparison sample, the difference between the two can be calculated, i.e., the accuracy comparison loss. The accuracy comparison loss reflects the gap in the initial model's ability to identify correct and incorrect inference paths. If the initial model can better identify and mimic the behavior of the reference strategy, it indicates that the initial model's performance in terms of accuracy has improved. In this embodiment, by introducing accuracy comparison loss, the initial model can continuously correct its decision bias during training, improving its ability to identify high-quality retrieval paths in complex scenarios, thereby improving the accuracy of the answer.

[0141] Similarly, the initial model's strategy generates a fifth output for the inference path of positive target samples in the sample query text and efficiency comparison samples, and the reference model's strategy generates a sixth output for the inference path of positive target samples in the sample query text and efficiency comparison samples. The initial model's strategy generates a seventh output for the inference path of negative target samples in the sample query text and efficiency comparison samples, and the reference model's strategy generates an eighth output for the inference path of positive negative target samples in the sample query text and efficiency comparison samples. Based on the differences between the fifth and seventh outputs, as well as the differences between the sixth and eighth outputs, an efficiency comparison loss is constructed.

[0142] The efficiency comparison samples are another part of the preference dataset. These samples consist of two inference paths that both yield the correct answer but have different retrieval costs: an efficient path (fewer retrievals) and an inefficient path (more retrievals). The model strategy and reference strategy also participate in the efficiency comparison sample processing to evaluate whether the initial model can choose a more efficient path while maintaining answer quality. The efficiency comparison loss measures whether the initial model chooses a more efficient retrieval strategy when faced with the same answer quality. If the initial model tends to frequently call retrievals, even if the final answer is correct, it may increase server response latency and resource consumption. Therefore, by optimizing the efficiency comparison loss, the initial model can learn to reduce unnecessary retrieval operations while ensuring answer quality, thus improving operational efficiency. This embodiment of the application, by introducing the efficiency comparison loss, allows the initial model to focus not only on the correctness of the answer but also on the economy of the retrieval process, thereby achieving optimal resource allocation and improving response speed and user experience.

[0143] In one example, the sum of the accuracy-contrast loss and the efficiency-contrast loss, or the weighted sum of the accuracy-contrast loss and the efficiency-contrast loss, is used as the total model loss for the current training batch, and then the model parameters of the initial model are updated based on the total model loss.

[0144] It is worth noting that, in this embodiment, during a single training run, the initial model needs to simultaneously optimize accuracy and efficiency objectives. To balance these two objectives during training, in addition to using a dynamic sampling ratio adjustment mechanism, a dynamically adjusted loss function weight mechanism can also be used, enabling the initial model to flexibly optimize different objectives according to the training progress. Therefore, updating the model parameters of the initial model based on the accuracy comparison loss and efficiency comparison loss can be achieved in the following way: adjusting the first loss weight corresponding to the accuracy comparison loss and the second loss weight corresponding to the efficiency comparison loss according to the current training progress; then, calculating the accuracy comparison loss and efficiency comparison loss based on the adjusted first loss weight and the adjusted second loss weight to obtain the total model loss; finally, updating the model parameters of the initial model based on the total model loss.

[0145] If the total loss of the model is L total =λ acc ·L acc +λ ret ·L ret , λ acc and λ ret Let λ represent the first loss weight corresponding to the accuracy-comparison loss and the second loss weight corresponding to the efficiency-comparison loss, respectively. At the beginning of training, the accuracy target is preferred, therefore λ = ... acc >λ ret In one example, the weights are dynamically adjusted based on the training progress, where λ can be linearly reduced according to the training progress. acc Increase λ ret This allows for a smooth transition from accuracy optimization to efficiency optimization, such as... t represents the current training progress, and T represents the total number of training epochs; for example, at the start of training, the first loss weight for accuracy loss is set to 0.8, and the weight is dynamically adjusted as follows: λ ret =1-λ acc , λ acc_min The minimum weights are used to compare the loss for accuracy. In another example, the weights can be adjusted using a non-linear function, such as... λ ret =1-λ acc k is a constant that controls the decay rate. This method makes the weight changes more gradual and is suitable for the gradual reduction of the weights for accuracy targets.

[0146] In one example of this application, the model performance can be dynamically detected based on the current training progress, and the weights can be adjusted according to the model performance at the current training progress. This can be achieved by: obtaining the accuracy and efficiency metrics of the initial model at the current training progress; if the accuracy metric is less than the accuracy metric threshold, then increasing the first loss weight; if the efficiency metric is less than the efficiency metric threshold, then increasing the second loss weight.

[0147] Here, after training at the current progress stage, the initial model's accuracy and efficiency metrics can be obtained. In the early training phase, if the accuracy metric is below the accuracy threshold, a first loss weight is increased; in the later training phase, if the efficiency metric is below the efficiency threshold, a second loss weight is increased. The first half of the total training epochs can be considered the early training phase, and the last half as the later training phase. It's understood that the accuracy and efficiency thresholds for each training epoch can be the same or different, and can be flexibly adjusted according to the actual situation.

[0148] In this embodiment, the weights can be further adjusted based on model performance, in addition to linearly or non-linearly adjusting the first and second loss weights according to the training progress; at t=3, λ acc =0.6 and λ ret =0.4. After adjusting the model parameters of the initial model, if the accuracy index of the model at t=3 is less than the accuracy index threshold, then increase λ. acc For example, when t=4, λ acc =0.7.

[0149] It should be noted that for a detailed description of S310 to S330 shown in Figure 4, please refer to S310 to S330 shown in Figure 3, which will not be repeated here.

[0150] In this embodiment, during a single training iteration, the sampling ratio of accuracy comparison samples and efficiency comparison samples is dynamically adjusted according to the current training progress. This reduces the performance trade-offs that may arise from phased training, achieving a balanced optimization of accuracy and efficiency within a single training iteration. It also eliminates the time cost of training multiple models individually, while simultaneously improving the overall model performance. Furthermore, during a single training iteration, a dynamically adjusted loss function weight mechanism is used to dynamically adjust the corresponding loss weights according to the current training progress. This allows for a smooth transition from optimizing accuracy to optimizing efficiency within a single training iteration, ensuring that both optimization objectives are fully learned. Ultimately, the model can simultaneously achieve both accuracy and efficiency, resulting in a significant improvement in overall performance.

[0151] This application embodiment also provides another language model training method, which can be applied to the implementation environment shown in Figure 1. This method can be executed by a terminal or a server, or jointly by both. In this embodiment, the method is described using a server as an example. As shown in Figure 6, based on the method shown in Figure 3, steps S340 to S350 in Figure 3 are extended to steps S610 to S630. Steps S610 to S630 are described in detail below.

[0152] S610. Noise the sample query text to obtain the noisy sample query text.

[0153] Noise enhancement refers to adding interfering information such as spelling errors, grammatical errors, and semantic ambiguity to the original query text to generate new text that is similar in meaning but deviates from the original. Noise enhancement simulates irregularities or missing information in user input, helping to test the robustness of the initial model to imperfect input. For example, in the sample query text "The weather in XX city today is sunny," noise enhancement could transform it into "The weather in XX city today is rainy" to assess whether the initial model can identify and correct this deviation. This application's embodiments, by introducing noisy sample query text, can enhance the initial model's adaptability to non-standard input, improve its robustness, and thus enhance its generalization performance in practical applications.

[0154] S620. Obtain the noise sample inference path generated by the initial model for the noise sample query text, and construct robust comparison samples based on the accuracy of the answer of the noise sample inference path.

[0155] The noisy sample inference path refers to the series of intermediate thought processes and decision-making logic generated by the initial model after receiving a noisy sample query text. The noisy sample inference path includes how the initial model understands the question, decides whether to search, selects which documents, and finally generates the answer. The noisy sample inference path not only reflects the internal behavior of the initial model but also provides a basis for subsequently building robust comparison samples. For example, when the initial model receives the noisy sample query text "It is raining in XX city today," the initial model can generate an inference path: the user is asking about the weather in XX city, but it is not raining in XX city today; therefore, the initial model should search the weather query module to further confirm the weather in XX city today.

[0156] Robustness comparison samples are a set of samples constructed based on the differences in the accuracy of the initial model's answers generated in inference paths of noisy samples. Specifically, if the initial model generates an answer consistent with the standard answer in a certain noisy sample inference path, the sample is labeled as a high-robust sample; otherwise, it is labeled as a low-robust sample. By constructing robustness comparison samples, the performance of the initial model under different input qualities can be distinguished, thereby guiding further optimization of the initial model. For example, if the initial model can correctly identify the weather in XX city in multiple noisy sample query texts, it indicates that the initial model has strong robustness. Constructing robustness comparison samples helps quantify the stability of the initial model under different noise conditions, thus providing researchers with clear directions and goals for optimizing the initial model.

[0157] S630. Sample the accuracy comparison samples, efficiency comparison samples, and robustness comparison samples according to the sampling ratio to generate a preference dataset.

[0158] In the embodiments of this application, in addition to accuracy comparison samples and efficiency comparison samples, other preference dimensions can be introduced, such as robustness comparison samples. Robustness comparison samples are adversarial data samples used to evaluate and improve the model's performance in the face of noise or imperfect inputs. Robustness comparison samples include inference paths that still generate correct answers under noisy input conditions, as well as inference paths that generate incorrect answers under the same conditions.

[0159] The sampling ratio refers to the proportion of different types of comparison samples (such as accuracy, efficiency, and robustness) randomly selected when constructing a preference dataset. The sampling ratio can be adjusted according to different training objectives. For example, when accuracy is prioritized, the proportion of accuracy comparison samples can be increased; when efficient retrieval is desired, the proportion of efficiency comparison samples can be emphasized. By reasonably configuring the sampling ratio, it can be ensured that the initial model learns a balance between accuracy and efficiency during training. For instance, if the sampling ratio is set to 5:3:2, it means that for every 10 samples, 5 come from accuracy comparison samples, 3 from efficiency comparison samples, and 2 from robustness comparison samples. By mixing and sampling different types of comparison samples in a certain proportion, a more comprehensive preference dataset can be constructed. When using this preference dataset to train the initial model, the initial model can simultaneously improve prediction accuracy, computational efficiency, and robustness to noise or abnormal inputs during training, ultimately achieving comprehensive optimization of model performance.

[0160] In this embodiment, by introducing noisy sample query text and constructing robust comparison samples, and combining accuracy and efficiency comparison samples for comprehensive sampling, the stability and generalization ability of the initial model mentioned in this embodiment can be effectively improved in complex environments, and it can better adapt to diverse practical application scenarios.

[0161] In one example, the sample query text can be noise-enhanced using at least one of the following methods: replacing keywords in the sample query text with synonyms to obtain noisy sample query text; generating spelling errors in the sample query text to obtain noisy sample query text; randomly inserting irrelevant words or deleting non-core words in the sample query text to obtain noisy sample query text; or deleting some words in the sample query text to obtain noisy sample query text with missing information. In other words, noise-enhancing the sample query text can involve replacing keywords with synonyms, generating spelling errors manually or through automated tools (such as character reversals, omissions, and repetitions), randomly inserting irrelevant words or deleting non-core words, or blurring words in the sample query text to generate missing information.

[0162] Input the noisy sample query text into the initial model and obtain at least two noisy sample inference paths generated by the initial model. Compare the noisy inference answers of the noisy sample inference paths with the standard answers of the sample query text. If the noisy inference answers are consistent with the standard answers, the noisy sample inference path is taken as a positive sample, and the noisy sample inference path that generates an incorrect answer is taken as a negative sample, so as to construct robust comparison samples.

[0163] In one example, the sampling ratios of accuracy comparison samples, efficiency comparison samples, and robustness samples are dynamically adjusted throughout the training process, ensuring that the initial model simultaneously achieves the three objectives of accuracy, efficiency, and robustness. In the early stages of training, accuracy is prioritized to build the core capabilities of the initial model. At this time, the sampling ratio of accuracy comparison samples is greater than that of efficiency comparison samples, and also greater than that of robustness comparison samples. For example, if the sampling ratio of robustness comparison samples is p... rob p acc =0.7, p ret =0.2, p rob =0.1; In the mid-stage, while improving retrieval efficiency, a high level of answer accuracy is maintained. At this point, the sampling ratio of the accuracy comparison sample is similar to that of the efficiency comparison sample, or the sampling ratio of the efficiency comparison sample is slightly greater than that of the accuracy comparison sample, and the sampling ratio of the accuracy comparison sample is greater than that of the robustness comparison sample, such as p. acc =0.4, p ret=0.4, p rob =0.2; In the later stages, robustness is the primary focus, aiming to improve the stability of the initial model under complex or extreme conditions. The sampling ratio corresponding to the robustness comparison samples is greater than that corresponding to the accuracy comparison samples, and also greater than that corresponding to the efficiency comparison samples, such as p acc =0.2, p ret =0.3, p rob =0.5; where the current training progress is determined based on the total number of training rounds.

[0164] In one example, based on the accuracy-contrast loss and efficiency-contrast loss, robustness-contrast loss is calculated for robust contrast samples in the preference dataset of the current training batch, according to the outputs of the initial model's strategy and the reference strategy. The loss weights of the three loss functions are dynamically adjusted, with the same objective as adjusting the sampling ratio. In the early stages of training, accuracy is prioritized, and the accuracy-contrast loss has the highest weight. In the middle stages of training, retrieval efficiency is improved, and the efficiency-contrast loss has the highest weight. In the later stages of training, robustness is prioritized, and the robustness-contrast loss has the highest weight.

[0165] It should be noted that for further details of steps S310 to S330 shown in Figure 6, please refer to steps S310 to S330 shown in Figure 3, which will not be repeated here.

[0166] In this embodiment, based on accuracy and efficiency, the sample query text is further noise-processed, and robustness is constructed based on the answer accuracy of the corresponding noisy sample inference path. This is used to evaluate and optimize the model's performance in adversarial scenarios. Furthermore, a preference dataset is generated based on accuracy comparison samples, efficiency samples, and robustness comparison samples, which further improves the diversity of the dataset and the effect of model training, and can also adapt to more complex scenario requirements.

[0167] To facilitate understanding, this application also provides a text processing method, including the training and application processes of a large language model. During training, a preference dataset is constructed, which considers both accuracy and retrieval efficiency. In terms of accuracy, comparative samples containing correct and incorrect answers are constructed to guide the initial model in identifying high-quality retrieval decision paths. In terms of efficiency, comparative samples with the same accuracy but different retrieval costs are designed to help the initial model learn to select the optimal retrieval strategy while ensuring answer quality. This two-dimensional data construction method provides the initial model with comprehensive learning signals. Based on the constructed preference dataset, the initial model's decision-making ability is improved by directly optimizing the model's preference for different decision paths, rather than using traditional reward-based reinforcement learning methods. This method ensures training stability while effectively balancing retrieval accuracy and efficiency.

[0168] As shown in Figure 7, the sample query text 701 is input into the initial model, and the answer 702 in the initial model is obtained. Based on the accuracy of the answers of different sample reasoning paths, an accuracy comparison sample (D) is constructed. acc ), and efficiency comparison samples (D) constructed based on inference paths of samples with the same accuracy but different retrieval costs. ret Then, the initial model is trained using the direct preference optimization method, which enables it to optimize retrieval efficiency while ensuring the accuracy of the answer, thus obtaining language model 703.

[0169] It's important to note that during the initial model's response generation process, three keywords can be predefined in the context: "thinking," "retrieval," and "response." If the initial model generates the keyword "thinking," it indicates that it is reasoning based on the sample query text and existing information. If the initial model generates the keyword "retrieval," the sentence subsequently generated by the initial model can be used as the query for retrieving the knowledge base, and the retrieved context can be returned to the initial model. If the initial model generates the keyword "response," the content subsequently generated by the initial model can be used as the final answer. Regarding the retrieval mechanism, Elasticsearch can be used to store Wikipedia data as the knowledge base, and the Best Matching 25 retrieval algorithm can be used to return the top three relevant documents to the initial model.

[0170] Here, the accuracy is compared with sample D. acc Used to improve the accuracy of the initial model's answer, specifically defined as Where q (i) This represents the sample query text. This represents the sample reasoning path that leads to the correct answer, while This represents the sample reasoning path that led to the incorrect answer. (D) acc The construction of the algorithm is accomplished through the following steps: First, multiple sample inference paths are sampled for each sample query text; then, the accuracy of the answer in each inference path is evaluated using the EM metric, which compares the labeled answer with the answer inferred by the model. If the labeled answer and the initial model's answer are the same, then EM = 1; otherwise, it is 0; finally, the sample inference paths that lead to the correct answer (EM = 1) and the incorrect answer (EM = 0) are labeled as follows: and Generate preference comparison data.

[0171] Efficiency comparison sample D ret Focus on optimizing the model's retrieval efficiency. Where q (i) This represents the sample query text. This represents a sample reasoning path with high retrieval efficiency, while This represents a sample inference path with inefficient retrieval, where the data... and You can get the correct answer, but This corresponds to a more efficient exploration path and has lower retrieval costs. D ret The dataset is constructed through the following steps: First, sample inference paths that yield the correct answer (EM=1) are selected. Next, the retrieval cost of each path is calculated. As mentioned above, this retrieval process is triggered only when the initial model generates the keyword "retrieval." Therefore, counting the number of times the keyword "retrieval" is generated during inference allows us to count the number of retrieval calls during the inference process, thus generating retrieval efficiency. Finally, based on retrieval efficiency, a comparison is made between efficient paths (inference paths with fewer retrievals) and inefficient paths (inference paths with more retrievals), ultimately forming efficiency preference data. This construction method ensures that the dataset captures the key trade-off between accuracy and efficiency. acc The comparison data is generated based on the complete reasoning process of the initial model, including three categories of keywords: "thinking," "retrieval," and "response." Accuracy comparison requires analyzing the entire process from thinking to response to identify the key steps leading to correct or incorrect answers. Different reasoning stages (whether the logic of the thinking is correct, whether the retrieval is relevant, and whether the response is reasonable) all affect the final accuracy of the response, thus requiring comprehensive analysis. And D... ret Focus on the efficiency of the initial model in the "retrieval" stage.

[0172] Based on the preference dataset, this embodiment of the application uses the Direct Preference Optimization (DPO) method to train the initial model, enabling the initial model to optimize retrieval efficiency while ensuring answer accuracy. DPO is a method for optimizing a language model based on human preference feedback. DPO directly optimizes the alignment between the model output and human preferences, transforming human preference judgments into the training objective of the initial model, thereby improving the quality and acceptability of the content generated by the initial model. For this objective, the loss function is defined as follows (2):

[0173] Where, π θ For the model strategy, π ref As the reference model strategy, β is the temperature parameter, σ is the Sigmoid function, q is the sample query text, and s w To provide a more accurate or efficient inference path for samples in the preferred dataset, s l Inference paths for samples in the preferred dataset that have poor accuracy or efficiency.

[0174] In one example, a dynamically adjusted temperature parameter can be used to control the sensitivity of the contrast loss. For instance, in early training, the contrast loss can be made more sensitive to enhance the initial model's learning of preferences; in later training, the contrast sensitivity can be reduced to make the initial model more robust.

[0175] In one example, in addition to using the DPO algorithm, algorithms such as Kahneman-Tversky Optimisation (KTO) and Identity Preference Optimisation (IPO) based on human psychological cognitive processes can also be used for direct preference optimization.

[0176] During the model training phase, this application employs a novel hybrid training strategy, utilizing a high-quality preference dataset D for optimizing answer accuracy. acc and the preference dataset D used to optimize retrieval efficiency retThe two datasets are merged into a single comprehensive dataset for unified training of the initial model. This approach breaks away from the traditional phased or alternating training patterns common in multi-objective optimization. By simultaneously optimizing accuracy and efficiency objectives during training, it significantly improves the overall performance of the initial model. During training, samples from the two datasets are mixed in a specific ratio according to a particular sampling strategy. Furthermore, to ensure the initial model can simultaneously address both optimization objectives, a dynamic loss balancing mechanism is designed. In this mechanism, the weight parameters in the loss function are dynamically adjusted based on the current training progress during training on the comprehensive dataset. For example, in the early stages, the initial model tends to prioritize accuracy optimization, therefore D... acc The sample proportion and corresponding weights in the loss function will be relatively high. However, as training progresses, optimizing retrieval efficiency gradually becomes the focus, and the weights of the loss function and the sample sampling strategy will shift towards D. ret Tilt; this dynamic adjustment mechanism allows the initial model to gradually achieve a balance between the two objectives during a single training process.

[0177] As shown in Figure 8, the training process includes the following steps S810 to S860:

[0178] S810, Randomly initialize the model parameters of the initial model.

[0179] S820. Determine the accuracy comparison sample D based on the current training progress. acc Efficiency comparison sample D ret The sampling ratio, and based on the sampling ratio, the accuracy of sample D is compared. acc Efficiency comparison sample D ret The preference dataset for training batches is generated by mixing the two.

[0180] For example, in the early stages of training, t≈0, D acc and D ret The sampling ratio was 8:2; as training progressed, the D ratio was gradually reduced. acc The sampling ratio, and increase D ret The sampling ratio.

[0181] S830. Input the preference dataset of the current training batch into the initial model, and calculate the accuracy-comparative loss L based on DPO. acc Compared with efficiency, the loss L ret .

[0182] S840. Based on the current training progress and the initial weights, determine the corresponding loss weights, and then calculate the total model loss L based on the weights for the current training epoch. total .

[0183] For example, in the early stages of training, t≈0, L acc The corresponding initial weight λacc =0.8, L ret The corresponding initial weight λ ret =0.2, calculate the total loss based on the corresponding weights and contrastive loss. As training progresses, gradually reduce λ. acc and increase λ ret The weight.

[0184] S850, Total Loss of Use L total Perform backpropagation to update the model parameters.

[0185] S860. Repeat the above steps until training is complete.

[0186] After each training round, adjust λ based on the training progress or model performance. acc and λ ret The sampling ratio and loss weight are continuously and dynamically adjusted until the training ends.

[0187] After training, a language model was obtained, and its effectiveness was validated on the open-source question-answering datasets HotpotQA, 2WikiMultihopQA, and Incomplete Information Reading Comprehension (IIRC). Table 1 below shows the performance of different models and methods on the training dataset (in-distribution) and the test dataset (out-of-distribution).

[0188] Table 1

[0189] Among the baselines selected for comparison were the Non-Retrieve method (without retrieval), the Single-Retrieve method (using a single retrieval), and the Non-Retrieve (SFT) and Single-Retrieve (SFT) methods, which fine-tune performance based on Soft Prompt Tuning (SFT) data. SFT is a method for adjusting model parameters to adapt to a specific task or dataset. As shown in Table 1, regardless of the test or training dataset, the language model provided in this embodiment outperforms other models or methods in F1 (a metric combining accuracy and recall) and EM.

[0190] The language model trained in this application embodiment can obtain the query text output by the target object and input the query text into the language model. The language model can generate the answer text through accurate and fast retrieval. The text processing method of this application embodiment is described using a translation scenario. The query text is the text to be translated. As shown in part a of FIG9, the text to be translated can be scanned by software that provides translation functions. For example, by operating the software that provides translation functions, a translation interface is displayed, and in response to the "take a picture" control of the translation interface, the text to be translated is scanned and extracted from the image obtained by taking a picture, or by uploading an image and obtaining the text to be translated. As shown in part b of FIG9, the target image is operated in the chat interface to display the target image in full screen. The full-screen display interface displays a symbol or control for indicating translation. In response to the operation of the symbol or control, the text to be translated is extracted from the target image. As shown in part c of FIG9, a variety of translation methods are provided in the translation application or applet. Therefore, the text to be translated can be obtained directly by inputting the target object, or it can be obtained through image translation or dialogue translation.

[0191] After obtaining the text to be translated, it is input into the language model trained in this embodiment. The language model first performs inference translation based on the text to be translated and the existing translation library. If there is content that the language model cannot translate, a retrieval method is used to obtain a reference translation result. Based on the inference translation and the retrieved translation result, a translated text corresponding to the text to be translated is generated. Because the speech model strengthens the learning of the correct inference path through answer accuracy during the training phase and optimizes the retrieval path through retrieval efficiency, it can generate translated text accurately and quickly.

[0192] Based on the above language model training method and text processing method, this application introduces a language model training apparatus and a text processing apparatus. The language model training apparatus can be used to execute the language model training method in the above embodiments of this application, and the text processing apparatus can be used to execute the text processing method in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the language model training method and text processing method described above.

[0193] This application provides a language model training device, as shown in FIG10. The language model training device 1000 includes: a first acquisition module 1010 configured to acquire at least two sample inference paths generated by an initial model for sample query text; a construction module 1020 configured to construct an accuracy comparison sample based on the answer accuracy of each sample inference path; the construction module 1020 is further configured to determine target sample inference paths with similar answer accuracy and construct an efficiency comparison sample based on the retrieval efficiency of each target sample inference path; a sampling module 1030 configured to sample the accuracy comparison sample and the efficiency comparison sample to obtain a preference dataset; and a training module 1040 configured to train the initial model based on the preference dataset to obtain the language model.

[0194] In some embodiments, the construction module 1020 is further configured to: obtain the reasoning answers of each of the sample reasoning paths; determine the answer accuracy of each of the sample reasoning paths based on the reasoning answers and the standard answers corresponding to the sample query text; obtain positive sample reasoning paths whose answer accuracy is greater than a first preset accuracy threshold, and obtain negative sample reasoning paths whose answer accuracy is less than a second preset accuracy threshold; and construct the accuracy comparison sample based on the positive sample reasoning paths and the negative sample reasoning paths.

[0195] In some embodiments, the construction module 1020 is further configured to compare the inference answer with the standard answer of the sample query text for each sample inference path to obtain a comparison result; and determine the accuracy of the answer of the sample inference path based on the comparison result.

[0196] In some embodiments, the construction module 1020 is further configured to: determine the sample inference path whose answer accuracy is greater than a first preset accuracy threshold as a candidate sample inference path; in response to the number of candidate sample inference paths being greater than a preset number, obtain the reference inference path generated by the optimized reference model for the sample query text; determine the similarity between the candidate sample inference path and the reference inference path; and determine the candidate sample inference path whose similarity is greater than a preset similarity threshold as the positive sample inference path.

[0197] In some embodiments, the construction module 1020 is further configured to determine the number of retrievals performed by the initial model in the target sample inference path; determine the retrieval efficiency of each target sample inference path based on the number of retrievals; and construct the efficiency comparison sample based on the positive target sample inference path with retrieval efficiency greater than a first preset efficiency threshold and the negative target sample inference path with retrieval efficiency less than a second preset efficiency threshold.

[0198] In some embodiments, the sampling module 1030 is further configured to acquire the retrieval time for each retrieval; determine the total retrieval time corresponding to the target sample inference path based on the retrieval time; and determine the retrieval efficiency of the target sample inference path based on the number of retrievals and the total retrieval time.

[0199] In some embodiments, the training module 1040 is further configured to, in the current training batch, adjust the sampling ratio of the accuracy comparison sample and the efficiency comparison sample according to the current training progress; sample the accuracy comparison sample and the efficiency comparison sample according to the adjusted sampling ratio to obtain the preference dataset of the current training batch; update the model parameters of the initial model according to the preference dataset of the current training batch to obtain the training model of the current training batch; re-execute the step of adjusting the sampling ratio until the total number of training rounds is reached, and determine the training model under the total number of training rounds as the language model.

[0200] In some embodiments, the training module 1040 is further configured to, in response to the current training batch being equal to 1, update the model parameters of the initial model according to the preference dataset of the first training batch to obtain the training model under the first training batch; wherein, the preference dataset of the first training batch is a dataset obtained by sampling the accuracy comparison samples using a first sampling ratio and sampling the efficiency comparison samples using a second sampling ratio; the first sampling ratio is the sampling ratio for the accuracy comparison samples set at the beginning of training, and the second sampling ratio is the sampling ratio for the efficiency comparison samples set at the beginning of training; in response to the current training batch being equal to n, and n being greater than 1, determine the training model under the (n-1)th training batch as the initial model under the nth training batch; and update the model parameters of the initial model under the nth training batch according to the preference dataset of the nth training batch to obtain the training model under the nth training batch.

[0201] In some embodiments, the training module 1040 is further configured to obtain a first sampling ratio set at the start of training for the accuracy comparison sample and a second sampling ratio for the efficiency comparison sample; the first sampling ratio is greater than the second sampling ratio; a third sampling ratio is obtained by reducing the first sampling ratio according to the total number of training rounds and the current training progress, and a fourth sampling ratio for the efficiency comparison sample is determined according to the third sampling ratio.

[0202] In some embodiments, the training module 1040 is further configured to: input the preference dataset of the current training batch into the initial model; determine the accuracy comparison loss between the positive sample inference path and the negative sample inference path based on the output of the model strategy and the reference strategy of the initial model for the accuracy comparison samples in the preference dataset of the current training batch; determine the efficiency comparison loss between the positive target sample inference path and the negative target sample inference path based on the output of the model strategy and the reference strategy of the initial model for the efficiency comparison samples in the preference dataset of the current training batch; and update the model parameters of the initial model based on the accuracy comparison loss and the efficiency comparison loss.

[0203] In some embodiments, the training module 1040 is further configured to adjust the first loss weight of the accuracy comparison loss and the second loss weight of the efficiency comparison loss according to the current training progress; calculate the accuracy comparison loss and the efficiency comparison loss according to the adjusted first loss weight and the adjusted second loss weight to obtain the total model loss; and update the model parameters of the initial model according to the total model loss.

[0204] In some embodiments, the training module 1040 is further configured to obtain the accuracy index and efficiency index of the initial model under the current training progress; increase the first loss weight in response to the accuracy index being less than the accuracy index threshold; and increase the second loss weight in response to the efficiency index being less than the efficiency index threshold.

[0205] In some embodiments, the sampling module 1030 is further configured to perform noise processing on the sample query text to obtain noisy sample query text; obtain the noisy sample inference path generated by the initial model for the noisy sample query text; construct robustness comparison samples based on the answer accuracy of the noisy sample inference path; and sample the accuracy comparison samples, the efficiency comparison samples, and the robustness comparison samples according to the sampling ratio to obtain the preference dataset.

[0206] In some embodiments, the sampling module 1030 is further configured to perform noise reduction processing on the sample query text through at least one of the following methods: replacing keywords in the sample query text with synonyms to obtain noisy sample query text; generating spelling error content for the sample query text to obtain noisy sample query text; randomly inserting irrelevant words into the sample query text or deleting non-core words from the sample query text to obtain noisy sample query text; deleting some words from the sample query text to obtain noisy sample query text with missing information.

[0207] This application provides a text processing device, as shown in FIG11. The text processing device 1100 includes: a second acquisition module 1110 configured to acquire query text; an input module 1120 configured to input the query text into a language model, wherein the language model is trained by the above-described language model training method; and a generation module 1130 configured to generate an answer text corresponding to the query text through the language model.

[0208] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs its operation has been described in detail in the method embodiments, and will not be repeated here. The apparatus provided in the above embodiments can be located in a terminal or in a server.

[0209] This application also provides an electronic device, including one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device enables the above-described language model training method and text processing method.

[0210] Figure 12 shows a schematic diagram of a computer system suitable for implementing the embodiments of this application. It should be noted that the computer system 1200 of the electronic device shown in Figure 12 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application. As shown in Figure 12, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1202 or a program loaded from storage portion 1208 into random access memory (RAM) 1203, such as executing the methods in the above embodiments. Various programs and data required for system operation are also stored in RAM 1203. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0211] In some embodiments, the following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.

[0212] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, or installed from removable medium 1211. When the computer program is executed by processor (CPU) 1201, it performs various functions defined in the system of embodiments of this application.

[0213] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory, flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the embodiments of this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0214] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0215] The units or modules described in the embodiments of this application can be implemented in software or hardware, and can also be located in a processor. The names of these units or modules do not necessarily limit the specific unit or module itself.

[0216] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the language model training method and text processing method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0217] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. An electronic device's processor reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the language model training method and text processing method described above in the various embodiments.

[0218] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0219] Other embodiments of the present application will readily conceive of by considering the specification and practicing the methods disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0220] The above content is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A method for training a language model, said method being performed by an electronic device, comprising: Obtain at least two sample inference paths generated by the initial model for the sample query text; An accuracy comparison sample is constructed based on the accuracy of the answers from each sample's reasoning path; Identify the reasoning paths of target samples with similar answer accuracy, and construct efficiency comparison samples based on the retrieval efficiency of each target sample reasoning path; The accuracy comparison sample and the efficiency comparison sample are sampled to obtain a preference dataset; The initial model is trained based on the preference dataset to obtain the language model.

2. The method according to claim 1, wherein, The construction of accuracy comparison samples based on the answer accuracy of each sample reasoning path includes: Obtain the reasoning answers for each of the sample reasoning paths; Based on the reasoning answer and the standard answer of the sample query text, determine the accuracy of the answer for each of the sample reasoning paths; Obtain positive sample inference paths where the accuracy of the answer is greater than a first preset accuracy threshold, and obtain negative sample inference paths where the accuracy of the answer is less than a second preset accuracy threshold; The accuracy comparison sample is constructed based on the positive sample inference path and the negative sample inference path.

3. The method according to claim 1 or 2, wherein, Determining the accuracy of each sample reasoning path's answer based on the reasoning answer and the standard answer of the sample query text includes: For each sample reasoning path, the reasoning answer is compared with the standard answer of the sample query text to obtain the comparison result; The accuracy of the sample reasoning path's response is determined based on the comparison results.

4. The method according to any one of claims 1 to 3, wherein, The step of obtaining the positive sample inference path whose answer accuracy is greater than a first preset accuracy threshold includes: The reasoning paths of samples whose answer accuracy is greater than the first preset accuracy threshold are determined as candidate sample reasoning paths; In response to the number of candidate sample inference paths being greater than a preset number, a reference inference path generated by the reference model for the sample query text is obtained; Determine the similarity between the candidate sample reasoning path and the reference reasoning path; The candidate sample reasoning path with a similarity greater than a preset similarity threshold is determined as the positive sample reasoning path.

5. The method according to any one of claims 1 to 4, wherein, The process of constructing efficiency comparison samples based on the retrieval efficiency of each target sample inference path includes: Determine the number of times the initial model performs retrievals in the inference path of the target sample; Based on the number of searches, the retrieval efficiency of each target sample reasoning path is determined; The efficiency comparison sample is constructed based on the reasoning paths of positive target samples with retrieval efficiency greater than a first preset efficiency threshold and the reasoning paths of negative target samples with retrieval efficiency less than a second preset efficiency threshold.

6. The method according to any one of claims 1 to 5, wherein, Determining the retrieval efficiency of each target sample reasoning path based on the number of retrievals includes: Get the retrieval time for each search; The total retrieval time corresponding to the inference path of the target sample is determined based on the retrieval time. The retrieval efficiency of the target sample reasoning path is determined based on the number of retrievals and the total retrieval time.

7. The method according to any one of claims 1 to 6, wherein, The accuracy comparison sample and the efficiency comparison sample are sampled to obtain a preference dataset; The language model is obtained by training the initial model based on the preference dataset, including: In the current training batch, adjust the sampling ratio of the accuracy comparison sample and the efficiency comparison sample according to the current training progress; Based on the adjusted sampling ratio, the accuracy comparison sample and the efficiency comparison sample are sampled to obtain the preference dataset for the current training batch. Based on the preference dataset of the current training batch, the model parameters of the initial model are updated to obtain the training model under the current training batch; Repeat the step of adjusting the sampling ratio until the total number of training rounds is reached, and determine the training model under the total number of training rounds as the language model.

8. The method according to any one of claims 1 to 7, wherein, The step of updating the model parameters of the initial model based on the preference dataset of the current training batch to obtain the training model for the current training batch includes: In response to the current training batch being equal to 1, the model parameters of the initial model are updated according to the preference dataset of the first training batch to obtain the training model under the first training batch; wherein, the preference dataset of the first training batch is a dataset obtained by sampling the accuracy comparison samples using a first sampling ratio and sampling the efficiency comparison samples using a second sampling ratio; the first sampling ratio is the sampling ratio for the accuracy comparison samples set at the beginning of training, and the second sampling ratio is the sampling ratio for the efficiency comparison samples set at the beginning of training; In response to the current training batch being equal to n and n being greater than 1, the training model under the (n-1)th training batch is determined as the initial model under the nth training batch; and the model parameters of the initial model under the nth training batch are updated according to the preference dataset of the nth training batch to obtain the training model under the nth training batch.

9. The method according to any one of claims 1 to 8, wherein, The step of adjusting the sampling ratio of the accuracy comparison samples and the efficiency comparison samples according to the current training progress includes: Obtain the first sampling ratio set at the start of training for the accuracy comparison samples, and the second sampling ratio for the efficiency comparison samples; the first sampling ratio is greater than the second sampling ratio. Based on the total number of training rounds and the current training progress, the first sampling ratio is reduced to obtain a third sampling ratio, and a fourth sampling ratio for the efficiency comparison samples is determined based on the third sampling ratio.

10. The method according to any one of claims 1 to 9, wherein, The step of updating the model parameters of the initial model based on the preference dataset of the current training batch includes: Input the preference dataset of the current training batch into the initial model; For the accuracy comparison samples in the preference dataset of the current training batch, the accuracy comparison loss between the positive sample inference path and the negative sample inference path is determined based on the output of the model strategy and the reference strategy of the initial model. For the efficiency comparison samples in the preference dataset of the current training batch, the efficiency comparison loss between the positive target sample inference path and the negative target sample inference path is determined based on the output of the model strategy and the reference strategy of the initial model. The model parameters of the initial model are updated based on the accuracy comparison loss and the efficiency comparison loss.

11. The method according to any one of claims 1 to 10, wherein, The step of updating the model parameters of the initial model based on the accuracy comparison loss and the efficiency comparison loss includes: Adjust the first loss weight of the accuracy comparison loss and the second loss weight of the efficiency comparison loss according to the current training progress; Based on the adjusted first loss weight and the adjusted second loss weight, the accuracy comparison loss and the efficiency comparison loss are calculated to obtain the total model loss; The model parameters of the initial model are updated based on the total model loss.

12. The method according to any one of claims 1 to 11, wherein, The step of adjusting the first loss weight of the accuracy comparison loss and the second loss weight of the efficiency comparison loss according to the current training progress includes: Obtain the accuracy and efficiency metrics of the initial model at the current training progress. In response to the accuracy metric being less than the accuracy metric threshold, the first loss weight is increased; In response to the efficiency index being less than the efficiency index threshold, the second loss weight is increased.

13. The method according to any one of claims 1 to 12, wherein, The sampling of the accuracy comparison samples and the efficiency comparison samples to obtain the preference dataset includes: The sample query text is noise-enhanced to obtain noisy sample query text; Obtain the noise sample inference path generated by the initial model for the noise sample query text; Robust comparison samples are constructed based on the accuracy of the responses to the inference paths of the noise samples; Based on the sampling ratio, the accuracy comparison samples, the efficiency comparison samples, and the robustness comparison samples are sampled to obtain the preference dataset.

14. The method according to any one of claims 1 to 13, wherein, The step of noise reduction processing of the sample query text to obtain noisy sample query text includes: The sample query text is noise-reduced using at least one of the following methods: Replace the keywords in the sample query text with synonyms to obtain the noisy sample query text; Generating spelling errors in the sample query text yields noisy sample query text. Randomly insert irrelevant words into the sample query text or delete non-core words from the sample query text to obtain noisy sample query text; By deleting some words from the sample query text, a noisy sample query text with missing information is obtained.

15. A text processing method, the method being executed by an electronic device, comprising: Get the query text; The query text is input into a language model, which is trained using the language model training method according to any one of claims 1 to 14. The language model is used to generate the answer text corresponding to the query text.

16. A training device for a language model, comprising: The first acquisition module is configured to acquire at least two sample inference paths generated by the initial model for the sample query text. The module is configured to build accuracy comparison samples based on the accuracy of the answers to the reasoning paths of each sample. The construction module is further configured to determine the reasoning path of target samples with similar answer accuracy, and to construct an efficiency comparison sample based on the retrieval efficiency of each target sample reasoning path; The sampling module is configured to sample the accuracy comparison samples and the efficiency comparison samples to obtain a preference dataset; The training module is configured to train the initial model based on the preference dataset to obtain the language model.

17. A text processing apparatus, comprising: The second acquisition module is configured to acquire query text; An input module is configured to input the query text into a language model, wherein the language model is trained using the language model training method according to any one of claims 1 to 14; The generation module is configured to generate the answer text corresponding to the query text using the language model.

18. An electronic device comprising: One or more processors; A storage device for storing one or more computer programs that, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1 to 15.

19. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the method of any one of claims 1 to 15.

20. A computer program product comprising a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from and executes the computer program to cause the electronic device to perform the method of any one of claims 1 to 15.

Images