Method and apparatus for training large language model, and device and storage medium

Abstract

A method for training a large language model. The method comprises: inputting a query statement into a large language model, so as to obtain a response statement generated by the large language model; on the basis of the response statement, determining a first loss value and a second loss value, wherein the first loss value is used for measuring the degree of safety of the response statement, and the second loss value is used for measuring the degree of lexical richness of the response statement; and on the basis of the first loss value and the second loss value, updating model parameters of the large language model.

Description

A method, apparatus, device, and storage medium for training a large language model

[0001] This application claims priority to Chinese Patent Application No. 202510466477.7, filed on April 14, 2025, entitled “A method, apparatus, device and storage medium for training a large language model”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for training large language models. Background Technology

[0003] With the development of artificial intelligence, large language models have been widely used in many fields such as natural language processing. Furthermore, with the increasing prevalence of large language model applications, the reliability and security of their output have become a focus of attention.

[0004] Therefore, to enhance the credibility of the output of large language models, they are typically trained using training data containing safe response statements. During training, the large language model learns the safe semantic patterns of safe response statements, understanding which expressions conform to ethical norms and safety guidelines. This allows it to filter out potentially harmful, erroneous, or inappropriate outputs in practical applications.

[0005] However, during the training process, large language models may rely too heavily on safe response statements, resulting in overly monotonous content generated by the model, which can negatively impact the user experience. Summary of the Invention

[0006] This application provides a method, apparatus, device, and storage medium for training large language models.

[0007] Firstly, this application provides a method for training a large language model, comprising: inputting a query statement into a large language model to obtain a response statement generated by the large language model; determining a first loss value and a second loss value based on the response statement, wherein the first loss value is used to measure the safety of the response statement and the second loss value is used to measure the lexical richness of the response statement; and updating the model parameters of the large language model based on the first loss value and the second loss value.

[0008] One possible implementation involves updating the model parameters of a large language model based on a first loss value and a second loss value, including: performing a weighted sum of the first and second loss values ​​to obtain a third loss value; and updating the model parameters of the large language model based on the third loss value.

[0009] One possible implementation involves determining a second loss value based on the response statement, including: obtaining the occurrence counts of various word combination combinations in the response statement; each word combination includes at least one word; and each word is a basic word in the response statement. The frequency of each word combination in the response statement is determined based on the ratio of its occurrence count to the total frequency of all word combinations. The second loss value is then determined based on the frequency of each word combination in the response statement.

[0010] One possible implementation, based on the frequency of each word combination in the response statement, determines the second loss value, further including: performing a negative logarithmic transformation on the frequency of each word combination to obtain the negative logarithm of the frequency of each word combination; determining the joint information entropy of multiple word combinations based on the sum of the products of the frequency of each word combination and the negative logarithm of the frequency of each word combination; and determining the second loss value based on the joint information entropy of multiple word combinations.

[0011] One possible implementation involves determining a second loss value based on the joint information entropy of multiple word combinations, including: determining the second loss value based on a first loss value and the joint information entropy of multiple word combinations.

[0012] One possible implementation involves determining the joint information entropy of multiple word combinations based on the sum of the products of the frequency of each word combination and the negative logarithms of the frequencies of each word combination, including:

[0013] The joint information entropy of multiple word combinations is determined using the following formula:

[0014] Where rd(Y) represents the joint information entropy of multiple word combinations, Y represents the response statement, n represents the total number of word combinations, and g i P(g) represents the i-th word combination. i ) represents the frequency of occurrence of the i-th word combination, -logP(g i The negative logarithm of the frequency of the i-th word combination.

[0015] One possible implementation involves determining the second loss value based on the joint information entropy of multiple word combinations, including: determining the second loss value according to the following formula: loss second =loss first +μ·rd(Y)

[0016] Where, loss second This represents the second loss value, loss first Let represent the first loss value, μ represent the richness intensity coefficient, and rd(Y) represent the joint information entropy of multiple word combinations.

[0017] One possible implementation involves determining a first loss value based on the response statement, including: obtaining a word sequence corresponding to the response statement, where the word sequence includes multiple words; sorting the words in the response statement according to their order of appearance; determining the generation probability of each word in the word sequence based on the query statement and the word sequence; and determining the first loss value based on the generation probability of each word.

[0018] One possible implementation involves determining a first loss value based on the generation probability of each word, including:

[0019] The first loss value is determined according to the following formula.

[0020] Where, loss first Let P(y) represent the first loss value. t |y <t Let y(t) represent the generation probability of the t-th word, X represent the query statement, and y(t) represent the probability of generating the t-th word. <t Let m represent the first t-1 lexical units, and m represent the length of the lexical unit sequence.

[0021] One possible implementation involves, for the first lexical unit at any position in the lexical unit sequence, determining the generation probability of each lexical unit in the sequence based on the query statement and the lexical unit sequence. This includes: obtaining the second lexical unit sequence corresponding to the first lexical unit; the second lexical unit is positioned before the first lexical unit. Based on a self-attention mechanism, the attention weighting result of the second lexical unit sequence and the query statement is determined. Based on the attention weighting result and the linear output layer of the large language model, multiple candidate lexical units and multiple matching scores (logits) corresponding to the first lexical unit are obtained from the linear output layer; each candidate lexical unit corresponds one-to-one with its logits; the candidate lexical units are obtained from the vocabulary of the large language model. The logits corresponding to each of the multiple candidate lexical units are normalized to obtain the generation probability of each candidate lexical unit. Based on the generation probabilities of each candidate lexical unit and a preset lexical unit determination rule, the first lexical unit and its corresponding generation probability are determined from the multiple candidate lexical units.

[0022] Secondly, this application provides a large language model training apparatus, which includes various functional modules for the method described in the first aspect above.

[0023] Thirdly, this application provides an electronic device including a processor and a memory; the memory stores processor-executable instructions; when the processor is configured to execute the instructions, the electronic device performs the method described in the first aspect above.

[0024] Fourthly, this application provides a readable storage medium comprising: software instructions; when the software instructions are executed in an electronic device, they cause the electronic device to perform the method described in the first aspect above.

[0025] Fifthly, this application provides a computer program product that, when run in an electronic device, causes the electronic device to execute the related methods described in the first aspect to implement the methods described in the first aspect.

[0026] The beneficial effects of the second to fifth aspects mentioned above can be referred to the first aspect, and will not be repeated here. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 is a flowchart illustrating a large language model training method provided in an embodiment of this application;

[0029] Figure 2 is a flowchart illustrating a second loss value determination method provided in an embodiment of this application;

[0030] Figure 3 is a flowchart illustrating a second loss value determination method based on word combination generation probability provided in an embodiment of this application;

[0031] Figure 4 is a flowchart illustrating a method for determining a first loss value according to an embodiment of this application;

[0032] Figure 5 is a schematic diagram of the composition of a large language model training device provided in an embodiment of this application;

[0033] Figure 6 is a schematic diagram of the composition of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] Hereinafter, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," or "third," etc., may explicitly or implicitly include one or more of that feature.

[0035] In the training of large language models, loss functions are typically used to guide the optimization direction. These loss functions calculate the difference between the model's output and the true labels to guide model parameter updates. However, during the algorithm credibility enhancement training of large language models, the semantic patterns of the secure responses used in the training data are relatively homogeneous. This can lead to overfitting in the pursuit of security, meaning that in practical applications, large language models may excessively favor generating content with high security. This results in a significant decrease in response quality, manifesting as rejecting reasonable requests or outputting generic security responses with limited vocabulary and low information content (such as "Sorry, I cannot answer this question").

[0036] Therefore, the problem we need to solve is how to improve the richness of response statements while ensuring the security of large language models.

[0037] In view of this, this application provides a method for training a large language model. By calculating the first and second loss values ​​of the response sentences of the large language model, a multi-objective optimization framework is constructed to synergistically improve the model's security and richness. Specifically, by updating the model parameters of the large language model based on the first and second loss values, a balance between the security and richness of the response sentences can be achieved, thereby improving the lexical richness of the model's response sentences while ensuring their security.

[0038] This application provides a large language model training method that can be applied to computing devices. The computing device can be a single server or a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. Optionally, the server can also be implemented on a cloud platform, such as a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud, and multi-cloud, or any combination thereof. This application does not limit the specific form of the computing device.

[0039] The following describes a training method for a large language model provided by an embodiment of this application, with reference to the accompanying drawings.

[0040] Figure 1 illustrates a training method for a large language model provided in an embodiment of this application. This method specifically includes the following steps:

[0041] S101. Input the query statement into the large language model to obtain the response statement generated by the large language model.

[0042] The query statement can be a factual query statement, a knowledge query statement, an opinion query statement, an instruction query statement, or a hypothetical query statement, etc.

[0043] One possible implementation is that the large language model generates a response statement based on the input query statement as follows:

[0044] First, the input query is divided into multiple tokens. An embedding layer converts these tokens into vectors and overlays positional encoding. These token vectors are then fed into a multi-layer Transformer encoder for processing, outputting a context vector containing global semantic relationships. Next, an autoregressive approach is used to generate tokens one by one. During the initialization phase, an initial decoding state and a start marker are established based on the encoding results. At each generation step, an attention mechanism extracts the current context vector, calculates the token probability distribution based on the generated content, and selects the most suitable token to add to the sequence and updates the decoding state according to a preset strategy (e.g., top-p sampling). The entire process continuously tracks the input semantics and generated content, iterating until a termination marker is generated or the maximum length is reached. Finally, a response statement that matches the query intent and maintains linguistic coherence is output.

[0045] It should be noted that a lexical unit is the smallest semantic unit that a large language model can process. For example, the lexical unit of "the weather is very good today" consists of four lexical units: "today", "weather", "very", and "good".

[0046] S102. Based on the response statement, determine the first loss value and the second loss value.

[0047] The first loss value is used to measure the security level of the response statement, and the second loss value is used to measure the lexical richness of the response statement.

[0048] It should be noted that the first loss value can be obtained based on the generation probability of each word in the context. It can quantitatively calculate and judge the consistency between the model's response and the safe response in the training data by the generation probability of each word.

[0049] The second loss value can be obtained based on the frequency of each word combination in the response statement. It can quantify the lexical richness of the response statement by comparing the frequency of each word combination in the response statement.

[0050] Specifically, word combination refers to a combination of phrases or short sentences composed of word elements.

[0051] S103. Update the model parameters of the large language model based on the first loss value and the second loss value.

[0052] In some embodiments, an alternating update method can be used to update the model parameters of a large language model by alternating between the first loss value and the second loss value.

[0053] The large language model training method provided in this application, after inputting a query statement into the large language model to obtain a response statement, simultaneously optimizes the model parameters based on a first loss value and a second loss value, achieving a synergistic improvement in model performance. Specifically, quantifying the security level of the response statement through the first loss value enables the large language model to effectively control the probability of generating non-compliant content, while quantifying the lexical richness of the response statement through the second loss value guides the large language model to explore richer word combinations and sentence structures, solving the problem of monotonous response content caused by overemphasizing security in existing technologies. Finally, through this dual-loss mechanism, the lexical richness of the large language model's response statement can be improved while ensuring its security.

[0054] One possible implementation is to update the model parameters of the large language model using a first loss value during odd-numbered training iterations, and to update the model parameters using a second loss value during even-numbered training iterations. This reduces the conflict between the update objectives of the first and second loss values.

[0055] It should be noted that the model parameters updated based on the first loss value can be the vocabulary projection matrix, attention key-value matrix, safe classifier parameters, position-sensitive parameters, etc. Based on the second loss value, the updated model parameters can be the attention value matrix, richness enhancement parameters, repetition penalty parameters, n-gram suppression weights, etc.

[0056] Among them, the vocabulary projection matrix directly determines the probability distribution of word generation; the attention key matrix affects the encoding of dangerous contexts; the safety classifier parameters provide additional gradient signals; and the position-sensitive parameters adjust the safety sensitivity of specific positions (such as the beginning of a sentence). The attention value matrix can change the model's attention to different information, thereby optimizing the content generation logic; the richness enhancement parameters can encourage the model to generate richer and more diverse responses; the repetition penalty parameters can effectively suppress the model from generating repetitive content, improving the fluency and originality of the output; and the n-gram suppression weights can adjust the weights for specific n-gram combinations to prevent inappropriate or unreasonable sentence combinations.

[0057] It should also be noted that the embodiments of this application do not limit the specific number and types of model update parameters.

[0058] The large language model training method provided in this application involves a computing device acquiring a first loss value and a second loss value for the response statement of the large language model. This enables a dual-loss parameter update mechanism for the large language model. The first loss value measures the safety level of the response statement, clarifying the optimization direction for safety. The second loss value measures the lexical richness of the response statement, prompting the large language model to explore low-frequency words with high information content, thus enriching the richness of the response statement. By jointly updating the model parameters using the first and second loss values, the richness of the response statement can be improved while ensuring its safety.

[0059] In some embodiments, during the process of updating model parameters based on the first loss value and the second loss value, a weighted summation method can also be used to combine the first loss value and the second loss value to obtain a third loss value for updating the model parameters. In this case, step S103 may specifically include:

[0060] S201. The first loss value and the second loss value are weighted and summed to obtain the third loss value.

[0061] In some embodiments, the weights in the weighted summation of the first loss value and the second loss value can be preset with fixed weights at the beginning of training.

[0062] For example, the first loss value and the second loss value can be configured as [0.3,0.7], [0.5,0.5], or [0.7,0.3], etc. The specific weight allocation of the first loss value and the second loss value is not limited in the embodiments of this application.

[0063] In other embodiments, the weights of the first and second loss values ​​can also be non-fixed. In this case, the computing device can adjust the weights of the first and second loss values ​​in the weighted summation process according to different stages of model training.

[0064] For example, in the early stages of training, the first and second loss values ​​can be configured as [0.8, 0.2]. In the middle stages of training, the first and second loss values ​​can be configured as [0.5, 0.5], and finally, in the later stages of training, they gradually transition to [0.3, 0.7]. It can be seen that this approach first ensures the safety of the model, and then, based on this, optimizes it towards richness.

[0065] S202. Update the model parameters of the large language model based on the third loss value.

[0066] As can be seen from steps S201-S202, the third loss value is the weighted sum of the first and second loss values, which can effectively integrate the training requirements of the large language model in terms of both security and richness. When the large language model updates its parameters based on the third loss value, it can achieve synergistic optimization of security and richness, thereby improving the training efficiency of the model.

[0067] The calculation process for the second loss value is described below with reference to the accompanying diagram.

[0068] In some embodiments, the process of determining the second loss value based on the response statement in step S102, as shown in Figure 2, may specifically include the following steps:

[0069] S301. Obtain the occurrence count of each of the various word combinations in the response statement.

[0070] The word combination includes at least one word. A word is a basic vocabulary word in the response statement.

[0071] In some embodiments, the computing device can use a sliding window method to obtain multiple word combinations. Specifically, the computing device can preset a fixed-size window and a sliding step size. First, it slides the window from left to right on the response statement according to the sliding step size. After each slide, the words in the window are extracted as a combination.

[0072] In other embodiments, the computing device can obtain multiple word combinations based on syntactic analysis methods. Specifically, the response statement is analyzed by a syntactic analyzer to obtain the sentence's syntactic structure tree, and then different word combinations are extracted from the tree.

[0073] One possible implementation is to classify and count the occurrences of multiple word combinations as follows: Prepare an empty dictionary to store various word combinations and their counts. Iterate through all the obtained word combinations. If a combination is already in the dictionary, increment its corresponding count in the dictionary by 1. If it is a new combination, create a key-value pair in the dictionary with the key being the word combination and the value being 1. After the iteration is complete, the dictionary will record the occurrence count of each word combination.

[0074] S302. Determine the frequency of each word combination in the response statement based on the ratio of the frequency of each word combination to the total frequency of all word combinations.

[0075] For example, the calculation process for determining the frequency of each word combination in the response statement can be represented as follows:

[0076] Among them, P(g) i) represents the frequency of occurrence of the i-th word combination, count(g i ) represents the frequency of occurrence of the i-th word combination. It indicates the number of occurrences of multiple word combinations.

[0077] For example, assuming that statistics show three word combinations, g1 appears 5 times, g2 appears 3 times, and g3 appears 2 times, with a total of 10 occurrences, according to formula (1), the frequency of g1 is 0.5, the frequency of g2 is 0.3, and the frequency of g3 is 0.2.

[0078] S303. Determine the second loss value based on the frequency of occurrence of each word combination in the response statement.

[0079] In some embodiments, the computing device can directly determine the second loss value based on the frequency of occurrence of each word combination. Specifically, the computing device can preset a high-frequency range (e.g., 50%-70%), and then determine the second loss value by counting the number of word combinations whose frequency falls within the frequency range.

[0080] Specifically, the number of word combinations whose frequency falls within the frequency range is used as the second loss value. The fewer word combinations whose frequency falls within the frequency range, that is, the fewer high-frequency word combinations, the more balanced the distribution of word combinations in the response statement, and the higher the lexical richness of the response statement.

[0081] As shown in steps S301-S303, this method obtains a second loss value by acquiring the frequency of different word combinations in the response statement. This measures the fit between the word combination frequency and the ideal distribution, judges the richness and uniformity of word combinations, and guides the lexical model to generate response statements with uniform and rich word combinations, thereby improving the lexical richness of the response statements.

[0082] In some embodiments, after obtaining the frequency of occurrence of word combinations, the computing device can also calculate the negative logarithm of the frequency of occurrence of word combinations. Information theory shows that the negative logarithm of the frequency of occurrence of word combinations can represent the amount of information carried by the word combination. In this case, the computing device can calculate the second loss value using the frequency of occurrence of word combinations and the negative logarithm of the frequency. This process is shown in Figure 3. Step S303 may further include the following steps:

[0083] S401. Perform a negative logarithmic transformation on the frequency of occurrence of each word combination to obtain the negative logarithm of the frequency of occurrence of each word combination.

[0084] It should be noted that in information theory, the lower the probability of an event occurring, the greater the amount of information it carries. For word combinations, a low frequency of occurrence means high rarity in the corpus. Therefore, computing devices obtain the amount of information carried by word combinations in the response statement by performing a negative logarithmic transformation on the frequency of occurrence of each word combination.

[0085] S402. Determine the joint information entropy of multiple word combinations based on the sum of the products of the occurrence frequency of each word combination and the negative logarithms of the occurrence frequency of each word combination.

[0086] Specifically, since multiple word combinations originate from the response statement, the joint information entropy of multiple word combinations can be called the joint information entropy of the response statement with respect to word combinations.

[0087] It should be noted that the product of the frequency of each word combination and the negative logarithm of its frequency can be understood as the information entropy contributed by that word combination in multiple combinations. When these products corresponding to all word combinations are summed, the joint information entropy of multiple word combinations is obtained.

[0088] Joint information entropy reflects the overall uncertainty or information richness of the word combination system in the entire response statement. A higher joint information entropy indicates that the word combination in the response statement is more diverse, containing more different types and varying frequencies of combinations, reflecting stronger information expression ability and semantic richness. Conversely, a lower joint information entropy indicates that the word combination is relatively simple, and there may be more repetitive or common combination patterns.

[0089] For example, the process of determining the joint information entropy of multiple word combinations can be represented as:

[0090] Where rd(Y) represents the joint information entropy of multiple word combinations; Y represents the response statement; n represents the total number of word combinations; g i P(g) represents the i-th word combination; i ) represents the frequency of occurrence of the i-th word combination; -logP(g i The negative logarithm of the frequency of the i-th word combination.

[0091] For example, suppose the response statement has three word combinations: g1 has a frequency of P(g1) = 0.5, and its negative logarithm -logP(g1) ≈ 0.693; g2 has a frequency of P(g2) = 0.3, and its negative logarithm -logP(g2) ≈ 1.204; g3 has a frequency of P(g3) = 0.2, and its negative logarithm -logP(g3) ≈ 1.609. Substituting these values ​​into formula (2), we get a joint information entropy of 1.030.

[0092] S403. Determine the second loss value based on the joint information entropy of multiple word combinations.

[0093] In some embodiments, the joint information entropy of multiple word combinations can be directly regarded as the second loss value. Specifically, when the joint information entropy is used as the second loss value, the information content of each word combination can be taken into consideration, which can more intuitively reflect the richness of the model-generated response statement from the perspective of information theory.

[0094] As can be seen from steps S401-S403, this method, based on the frequency of word combination occurrences, further calculates the negative logarithm of the frequency of word combination occurrences. Applying the principles of information theory, it treats the negative logarithm of the frequency of word combination occurrences as the information content of that word combination. Then, by correlating the frequency of word combination occurrences with the information content, it constructs a joint information entropy to determine the second loss value. This approach can more accurately measure the richness and abundance of information carried by word combinations in the response statement, providing a quantitative basis for model optimization from an informational perspective, and making the control of response quality during model training more scientific.

[0095] In some embodiments, the second loss value measures the lexical richness of the response statement. This process is independent of measuring the security level of the response statement. Therefore, the lexical richness measured by the second loss value may contain the richness of non-secure word combinations. Therefore, the calculation of the first loss value can be incorporated into the calculation of the second loss value. In this case, step S403 may specifically include:

[0096] S501. Determine the second loss value based on the first loss value and the joint information entropy of multiple word combinations.

[0097] For example, the process of determining the second loss value based on the first loss value and the joint information entropy of multiple word combinations can be represented as: loss second =loss first +μ·rd(Y) Formula (3)

[0098] Where, loss second This represents the second loss value; loss first represents the first loss value; μ represents the richness intensity coefficient; rd(Y) represents the joint information entropy of multiple word combinations.

[0099] For example, the richness intensity coefficient μ is used to control the weight of the joint information entropy in the calculation of the second loss value. It can be 0.5, 1, or 1.5. The specific value of the richness intensity coefficient is not limited in the embodiments of this application.

[0100] It should be noted that adding the first loss value to the second loss value is essentially to construct a dual security loss protection mechanism. First, the global first loss value ensures that the overall response does not deviate from the security baseline. Second, the first loss value calculated in the second loss value is specifically designed to precisely block any "security vulnerabilities" that may arise during the richness optimization process.

[0101] This design stems from a key phenomenon observed in practice—when pursuing lexical richness, models may circumvent safety restrictions through three means: first, by using synonym substitution; second, by constructing complex sentence structures to bypass keyword detection; and third, by combining safe words to express dangerous semantics. By incorporating the first loss value into the calculation of the second loss value, the behavior of reducing richness loss by increasing the richness of dangerous combinations can be reduced.

[0102] The calculation process for the first loss value is described below with reference to the accompanying diagram.

[0103] In some embodiments, the process of determining the first loss value based on the response statement in step S102, as shown in Figure 4, may specifically include the following steps:

[0104] S601. Obtain the word sequence corresponding to the reply statement.

[0105] The word sequence consists of multiple words, and the words are sorted according to the order in which they appear in the response statement.

[0106] S602. Based on the query statement and the word sequence, determine the generation probability of each word in the word sequence.

[0107] Specifically, the detailed process of this step can be found in steps S701-S705 below, and will not be repeated here.

[0108] S603. Determine the first loss value based on the generation probability of each word.

[0109] For example, the calculation process for determining the first loss value can be expressed as follows:

[0110] Where, loss first Let P(y) represent the first loss value. t |y <t Let y(t) represent the generation probability of the t-th word, X represent the query statement, and y(t) represent the probability of generating the t-th word. <t Let m represent the first t-1 lexical units, and m represent the length of the lexical unit sequence.

[0111] For example, suppose the query X is "Today's weather" and the response is "Today's weather is sunny". In this response, the word sequence is ["Today", "Weather", "Sunny"], a total of 4 words. It is known that P(y1|y<1,X)=0.8 (the probability of generating "Today"), P(y2|y<2,X)=0.7 (the probability of generating "Weather"), and P(y3|y<3,X)=0.6 (the probability of generating "Sunny"). The loss can be obtained according to formula (4). first =(log0.8+log0.7+log0.6)=(-0.2231-0.3567-0.5108)=0.0906.

[0112] In some embodiments, for the first word at any position in the word sequence in step S601, step S602 calculates the generation probability of each word in the word sequence based on the query statement and the word sequence, which may specifically include the following steps:

[0113] S701. Obtain the second word sequence corresponding to the first word.

[0114] In this context, the second word element is positioned before the first word element.

[0115] Specifically, as can be seen from the process of generating the response statement by the large language model in step S101, the large language model generates the response statement one word at a time. Therefore, the second word sequence can be understood as the word sequence that the large language model has generated before the first word in the process of generating the response statement.

[0116] It should be noted that when the first term is the first term in the response statement, the sequence of the second term corresponding to the first term can be empty.

[0117] S702. Based on the self-attention mechanism, determine the attention weighting result of the second word sequence and the query statement.

[0118] Specifically, the second word sequence and multiple words in the query statement are converted into vector representations, and the attention score of each word is calculated through the self-attention mechanism of the large language model. Based on the self-attention score, multiple words are weighted and summed to obtain the weighted sum result.

[0119] It should be noted that this weighted summation result can also be called a context-aware representation. Specifically, because this weighted summation result integrates information from the second word sequence and each word in the query statement, and uses a self-attention mechanism to weight different words according to their importance, this result fully perceives the semantic relationships and logical connections of the context. It is no longer a simple set of isolated word vectors, but a feature representation that contains the contextual relationships of the entire input sequence.

[0120] S703. Based on the attention weighting result and the linear output layer of the large language model, obtain multiple candidate words and multiple matching scores logits corresponding to the first word word output by the linear output layer.

[0121] Among them, the candidate lexical units correspond one-to-one with the logits, and the candidate lexical units are obtained from the vocabulary of the large language model.

[0122] It should be noted that the core principle of the linear output layer is to map the high-dimensional hidden state to the vocabulary space through a simple matrix transformation, and generate logits for each candidate word.

[0123] It should also be noted that logits essentially reflect the degree of reasonableness of each candidate lexical in the current context. For example, in a medical consultation scenario, a lexical with a high logits value will simultaneously meet the following conditions: it conforms to the standards of medical terminology (such as "blood glucose monitoring" rather than "measuring blood glucose"), and it can naturally connect with the dialogue history (such as prioritizing "suggestion" rather than "requirement").

[0124] One possible implementation involves first performing post-attention processing on the attention-weighted result (including residual connections, layer normalization, and feedforward neural network transformations) to obtain the high-dimensional hidden state corresponding to the attention-weighted result. This high-dimensional hidden state represents the contextual representation after deep semantic fusion and feature enhancement. It retains the key information dynamically extracted by the attention mechanism (such as core entities and logical relationships in the query) while capturing more complex semantic patterns (such as cross-sentence reasoning clues and domain knowledge associations) through nonlinear transformations. The linear output layer multiplies the high-dimensional hidden state with the vocabulary embedding matrix and calculates the matching degree between this hidden state and each lexical in the vocabulary in the semantic space, thereby determining multiple candidate lexicals and their corresponding logits.

[0125] S704. Normalize the logits corresponding to each of the multiple candidate words to obtain the generation probability of each of the multiple candidate words.

[0126] It should be noted that the purpose of normalizing the logits corresponding to each of the multiple candidate words is to transform the matching score into a probability distribution, so as to clarify the probability of each word among the multiple candidate words being generated in the current context.

[0127] One possible implementation is to use the softmax function to calculate the generation probability of each candidate word.

[0128] S705. Based on the generation probability of each candidate word and the preset word determination rules, determine the first word and the generation probability of the first word from multiple candidate words.

[0129] In some embodiments, the preset word element determination rule can be a greedy search rule, that is, directly selecting the candidate word element with the highest generation probability as the first word element. Alternatively, it can be a sampling rule, that is, randomly sampling according to the generation probability of each candidate word element, with the probability of each word element being selected being proportional to its generation probability.

[0130] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the large language model training device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] In an exemplary embodiment, this application also provides a large language model training method, which can be applied to the aforementioned computing device. As shown in FIG5, the device includes an acquisition module 510 and a processing module 520.

[0132] The acquisition module 510 is used to input the query statement into the large language model and obtain the response statement generated by the large language model.

[0133] The processing module 520 is used to determine a first loss value and a second loss value based on the response statement. The first loss value measures the safety level of the response statement, and the second loss value measures the lexical richness of the response statement. Based on the first and second loss values, the model parameters of the large language model are updated.

[0134] One possible implementation involves the processing module 520 specifically used to update the model parameters of the large language model based on the first loss value and the second loss value, including: performing a weighted summation of the first loss value and the second loss value to obtain a third loss value; and updating the model parameters of the large language model based on the third loss value.

[0135] One possible implementation involves the processing module 520 specifically determining a second loss value based on the response statement, including: obtaining the occurrence counts of various word combination combinations in the response statement; each word combination includes at least one word; and each word is a basic word in the response statement. The frequency of each word combination in the response statement is determined based on the ratio of its occurrence count to the total frequency of all word combinations. The second loss value is then determined based on the frequency of each word combination in the response statement.

[0136] One possible implementation involves the processing module 520 specifically determining a second loss value based on the frequency of each word combination in the response statement. This further includes: performing a negative logarithmic transformation on the frequency of each word combination to obtain the negative logarithm of the frequency of each word combination; determining the joint information entropy of multiple word combinations based on the sum of the products of the frequency of each word combination and the negative logarithms of the frequency of each word combination; and determining the second loss value based on the joint information entropy of multiple word combinations.

[0137] One possible implementation is that the processing module 520 is specifically used to determine a second loss value based on the joint information entropy of multiple word combinations, including: determining the second loss value based on the first loss value and the joint information entropy of multiple word combinations.

[0138] One possible implementation is that the processing module 520 is specifically used to determine the joint information entropy of multiple word combinations based on the sum of the products of the occurrence frequency of each word combination and the negative logarithms of the occurrence frequencies of each word combination, including:

[0139] The joint information entropy of multiple word combinations is determined using the following formula:

[0140] Where rd(Y) represents the joint information entropy of multiple word combinations, Y represents the response statement, n represents the total number of word combinations, and g i P(g) represents the i-th word combination. i ) represents the frequency of occurrence of the i-th word combination, -logP(g i The negative logarithm of the frequency of the i-th word combination.

[0141] One possible implementation is that the processing module 520 is specifically used to determine a second loss value based on the joint information entropy of multiple word combinations, including: determining the second loss value according to the following formula: loss second =loss first +μ·rd(Y)

[0142] Where, loss second This represents the second loss value, loss firstLet represent the first loss value, μ represent the richness intensity coefficient, and rd(Y) represent the joint information entropy of multiple word combinations.

[0143] One possible implementation involves the processing module 520 specifically used to determine a first loss value based on the response statement, including: obtaining a word sequence corresponding to the response statement, the word sequence comprising multiple words, the word sequence being sorted according to the order in which the words appear in the response statement; determining the generation probability of each word in the word sequence based on the query statement and the word sequence; and determining the first loss value based on the generation probability of each word.

[0144] One possible implementation is that the processing module 520 is specifically used to determine a first loss value based on the generation probability of each word, including:

[0145] The first loss value is determined according to the following formula.

[0146] Where, loss first Let P(y) represent the first loss value. t |y <t Let y(t) represent the generation probability of the t-th word, X represent the query statement, and y(t) represent the probability of generating the t-th word. <t Let m represent the first t-1 lexical units, and m represent the length of the lexical unit sequence.

[0147] In one possible implementation, the processing module 520 is further configured to, for the first word element at any position in the word element sequence, determine the generation probability of each word element in the word element sequence based on the query statement and the word element sequence, including: obtaining the second word element sequence corresponding to the first word element; the position of the second word element is before the position of the first word element. Based on a self-attention mechanism, determine the attention weighting result of the second word element sequence and the query statement. Based on the attention weighting result and the linear output layer of the large language model, obtain multiple candidate word elements and multiple matching scores (logits) output by the linear output layer corresponding to the first word element; the candidate word elements and logits are in one-to-one correspondence; the candidate word elements are obtained from the vocabulary of the large language model. Normalize the logits corresponding to each of the multiple candidate word elements to obtain the generation probability corresponding to each of the multiple candidate word elements. Based on the generation probabilities corresponding to each candidate word element and the preset word element determination rules, determine the first word element and the generation probability corresponding to the first word element from the multiple candidate word elements.

[0148] It should be noted that the module division in Figure 5 is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. For example, two or more functions can be integrated into a single processing module. The integrated modules described above can be implemented either in hardware or as software functional modules.

[0149] In exemplary embodiments, as described above, the computing device may specifically be an electronic device with computing processing capabilities, such as a computer or service. In this case, this application embodiment also provides an electronic device, and FIG6 is a schematic diagram of the composition of an electronic device provided in this application embodiment. As shown in FIG6, the electronic device includes: a processor 10, a memory 20, a communication line 30, a communication interface 40, and an input / output interface 50.

[0150] The processor 10, memory 20, communication interface 40, and input / output interface 50 can be connected via communication line 30.

[0151] Processor 10 is used to execute instructions stored in memory 20 to implement the large language model training method provided in the above embodiments of this application. Processor 10 can be a CPU, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / microcontroller, a programmable logic device (PLD), or any combination thereof. Processor 10 can also be any other device with processing functions, such as a circuit, device, or software module, which is not limited in this embodiment. In one example, processor 10 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 6. As an optional implementation, the electronic device may include multiple processors, for example, in addition to processor 10, it may also include processor 60 (shown as an example by dashed lines in FIG. 6).

[0152] The memory 20 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions; it may also be a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions; it may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, etc. The embodiments of this application do not limit this.

[0153] It should be noted that the memory 20 can exist independently of the processor 10 or it can be integrated with the processor 10. The memory 20 can be located inside or outside the electronic device, and this application embodiment does not impose any restrictions on this.

[0154] Communication line 30 is used to transmit information between the components included in the electronic device.

[0155] Communication interface 40 is used to communicate with other devices or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Communication interface 40 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0156] Input / output interface 50 is used to enable human-computer interaction between users and electronic devices. For example, it enables action interaction or information exchange between users and electronic devices.

[0157] For example, the input / output interface 50 can be a mouse, keyboard, display screen, or touch screen. Action or information interaction between the user and the electronic device can be achieved through a mouse, keyboard, display screen, or touch screen.

[0158] In an exemplary embodiment, this application also provides a readable storage medium including software instructions that, when run on an electronic device, cause the electronic device to perform any of the methods provided in the above embodiments.

[0159] In an exemplary embodiment, this application also provides a computer program product containing computer execution instructions, which, when run on an electronic device, causes the electronic device to perform any of the methods provided in the above embodiments.

[0160] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer-executable instructions. When these computer-executable instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0161] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0162] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0163] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a large language model, the method comprising: Input the query statement into the large language model to obtain the response statement generated by the large language model; Based on the response statement, a first loss value and a second loss value are determined; wherein, the first loss value is used to measure the security level of the response statement; and the second loss value is used to measure the lexical richness of the response statement. The model parameters of the large language model are updated based on the first loss value and the second loss value.

2. The method according to claim 1, wherein, The step of updating the model parameters of the large language model based on the first loss value and the second loss value includes: The first loss value and the second loss value are weighted and summed to obtain the third loss value; Based on the third loss value, the model parameters of the large language model are updated.

3. The method according to claim 1 or 2, wherein, Based on the response statement, the second loss value is determined, including: The frequency of occurrence of various word combinations in the response statement is obtained; each word combination includes at least one word; each word is a basic word in the response statement. The frequency of each word combination in the response statement is determined based on the ratio of the frequency of each word combination to the total frequency of all word combinations. The second loss value is determined based on the frequency of occurrence of each word combination in the response statement.

4. The method according to claim 3, wherein, The step of determining the second loss value based on the frequency of occurrence of each word combination in the response statement further includes: Perform a negative logarithmic transformation on the frequency of occurrence of each word combination to obtain the negative logarithmic frequency of occurrence of each word combination; The joint information entropy of the various word combinations is determined based on the sum of the products of the occurrence frequency of each word combination and the negative logarithms of the occurrence frequency of each word combination. The second loss value is determined based on the joint information entropy of the various word combinations.

5. The method according to claim 4, wherein, The determination of the second loss value based on the joint information entropy of the multiple word combinations includes: The second loss value is determined based on the first loss value and the joint information entropy of the various word combinations.

6. The method according to claim 4, wherein, The determination of the joint information entropy of the various word combinations based on the sum of the products of the occurrence frequency of each word combination and the negative logarithms of the occurrence frequency of each word combination includes: The joint information entropy of the various word combinations is determined according to the following formula: Where rd(Y) represents the joint information entropy of the various word combinations; Y represents the response statement; n represents the total number of word combinations; g i P(g) represents the i-th word combination; i ) represents the frequency of occurrence of the i-th word combination; -logP(g i The negative logarithm of the frequency of the i-th word combination.

7. The method according to claim 5, wherein, The determination of the second loss value based on the joint information entropy of the multiple word combinations includes: The second loss value is determined according to the following formula: loss second =loss first +μ·rd(Y) Where, loss second This represents the second loss value; loss first denoted as the first loss value; μ represents the richness intensity coefficient; rd(Y) represents the joint information entropy of the various word combinations.

8. The method according to claim 1, wherein, Determining the first loss value based on the response statement includes: Obtain the word sequence corresponding to the reply statement; the word sequence includes multiple words; the word sequence is sorted based on the order in which the words appear in the reply statement; Based on the query statement and the word sequence, determine the generation probability of each word in the word sequence; The first loss value is determined based on the generation probability of each word.

9. The method according to claim 8, wherein, Determining the first loss value based on the generation probability of each word element includes: The first loss value is determined according to the following formula: Where, loss first P(y) represents the first loss value; t |y <t (x, y) represents the generation probability of the t-th word; X represents the query statement; y <t represents the first t-1 lexical units; m represents the length of the lexical unit sequence.

10. The method according to claim 8, wherein, For the first word at any position in the word sequence, determining the generation probability of each word in the word sequence based on the query statement and the word sequence includes: Obtain the second word sequence corresponding to the first word; the position of the second word is before the position of the first word. Based on the self-attention mechanism, the attention weighting result of the second word sequence and the query statement is determined; Based on the attention weighting result and the linear output layer of the large language model, multiple candidate words and multiple matching scores (logits) corresponding to the first word word are obtained from the output of the linear output layer; the candidate words and the logits are in one-to-one correspondence; the candidate words are obtained from the vocabulary of the large language model. Normalize the logits corresponding to each of the multiple candidate words to obtain the generation probability of each of the multiple candidate words. Based on the generation probabilities of each candidate word and the preset word determination rules, the first word and its corresponding generation probability are determined from multiple candidate words.

11. A large language model training device, the device comprising: Acquisition module, processing module; The acquisition module is used to input the query statement into the large language model and obtain the response statement generated by the large language model; The processing module is used to determine a first loss value and a second loss value based on the response statement; wherein the first loss value is used to measure the security level of the response statement; the second loss value is used to measure the lexical richness of the response statement; and the model parameters of the large language model are updated based on the first loss value and the second loss value.

12. An electronic device, comprising: Processor and memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, it causes the electronic device to implement the method as described in any one of claims 1-10.

13. A readable storage medium, comprising: Software instructions; When the software instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-10.

14. A computer program product comprising: Computer instructions; When the computer instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-10.

Images