Language processing model training method and apparatus, electronic device, and readable storage medium
Patent Information
- Application Number
- PCT/CN2025/095832
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-02-21
- Filing Date
- 2025-05-19
- Publication Date
- 2026-08-27
Smart Images
Figure CN2025095832_27082026_PF_FP_ABST
Abstract
Description
Language processing model training method and device, electronic equipment and readable storage medium
[0001] Cross-reference to related applications
[0002] The present application claims priority to the Chinese patent application No. 2025101964074, filed on February 21, 2025, and entitled "Language processing model training method, device, electronic equipment and readable storage medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD
[0003] The present disclosure relates to the technical field of speech synthesis, in particular, to a language processing model training method, device, electronic equipment and computer readable storage medium. BACKGROUND
[0004] With the rapid development of information technology, speech synthesis technology (Text-To-Speech, TTS) as a technology for converting text into natural and fluent speech has become an important part of human-computer interaction, directly affecting the user experience. TTS is widely used in outbound broadcasting, voice navigation, audio books and other fields.
[0005] In recent years, training of large speech synthesis models based on neural networks has become a research hotspot in the field of TTS. However, existing large speech synthesis models perform poorly when processing style information (such as prosody, emotion, etc.). SUMMARY
[0006] The purpose of the present disclosure is to provide a language processing model training method, device, electronic equipment and readable storage medium to solve the problem of poor performance of existing large speech synthesis models when processing style information.
[0007] In a first aspect, the present disclosure provides a language processing model training method, comprising:
[0008] obtaining training text data and real speech label data;
[0009] inputting the training text data and the real speech label data into a prediction module in a language processing model to obtain predicted speech label data output by the prediction module;
[0010] obtaining real style features according to the real speech label data and predicted style features according to the predicted speech label data;
[0011] obtaining a total loss function according to the real speech label data, the predicted speech label data, the real style features and the predicted style features;
[0012] optimizing and training the prediction module according to the total loss function.
[0013] In an optional implementation, the language processing model comprises a style encoding module, and the obtaining of the real style feature according to the real speech tag data and the obtaining of the predicted style feature according to the predicted speech tag data comprise:
[0014] The real style feature is obtained by performing style extraction on the real speech tag data through the style encoding module.
[0015] The predicted style feature is obtained by performing style extraction on the predicted speech tag data through the style encoding module.
[0016] In an optional implementation, the obtaining of the total loss function according to the real speech tag data, the predicted speech tag data, the real style feature and the predicted style feature comprises:
[0017] The prediction loss function is obtained according to the real speech tag data and the predicted speech tag data.
[0018] The style loss function is obtained according to the real style feature and the predicted style feature.
[0019] The total loss function is obtained according to the prediction loss function and the style loss function.
[0020] In an optional implementation, the obtaining of the prediction loss function according to the real speech tag data and the predicted speech tag data comprises:
[0021] The prediction loss function is obtained by calculating the distribution distance between the real speech tag data and the predicted speech tag data.
[0022] In an optional implementation, the obtaining of the style loss function according to the real style feature and the predicted style feature comprises:
[0023] The style loss function is obtained by calculating the similarity between the real style feature and the predicted style feature.
[0024] In an optional implementation, the obtaining of the total loss function according to the prediction loss function and the style loss function comprises:
[0025] The total loss function is obtained by performing weighted calculation on the prediction loss function and the style loss function.
[0026] In a second aspect, the present disclosure provides a language processing model training device, which comprises:
[0027] a predicted speech label data obtaining module configured to obtain training text data and real speech label data; input the training text data and the real speech label data into a prediction module in a language processing model, and obtain predicted speech label data output by the prediction module;
[0028] a style feature obtaining module configured to obtain real style features according to the real speech label data, and obtain predicted style features according to the predicted speech label data;
[0029] a loss function obtaining module configured to obtain a total loss function according to the real speech label data, the predicted speech label data, the real style features and the predicted style features;
[0030] an optimization training module configured to perform optimization training on the prediction module according to the total loss function.
[0031] In an optional implementation, the language processing model comprises a style encoding module, and the style feature obtaining module is further configured to perform style extraction on the real speech label data by the style encoding module to obtain the real style features, and perform style extraction on the predicted speech label data by the style encoding module to obtain the predicted style features.
[0032] In a third aspect, the present disclosure provides an electronic device comprising a processor and a memory, the memory storing a computer program capable of being executed by the processor, and the computer program executable by the processor implements the language processing model training method according to any one of the preceding embodiments.
[0033] In a fourth aspect, the present disclosure provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the language processing model training method according to any one of the preceding embodiments.
[0034] The language processing model training method, device, electronic device and readable storage medium provided by the embodiments of the present disclosure include: obtaining training text data and real speech mark data and inputting the training text data and the real speech mark data into a prediction module in a language processing model, obtaining predicted speech mark data, extracting style features in the real and predicted speech mark data respectively, constructing a total loss function according to the real speech mark data, the predicted speech mark data, real style features and predicted style features, and optimizing and training the language processing model according to the total loss function. The style feature loss is introduced into the total loss function to ensure that the language processing model can learn style information, so that the language processing model can more accurately reproduce the speech style, thereby improving the performance of the language processing model in processing speech style information. And the speech processing model is a key part of the speech synthesis model, and thus the performance of the speech synthesis large model in processing speech style information can be improved. BRIEF DESCRIPTION OF DRAWINGS
[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.
[0036] FIG. 1 shows a flowchart of a language processing model training method according to an embodiment of the present disclosure;
[0037] FIG. 2 shows another flowchart of a language processing model training method according to an embodiment of the present disclosure;
[0038] FIG. 3 shows a framework diagram of a speech synthesis large model according to an embodiment of the present disclosure;
[0039] FIG. 4 shows another flowchart of a language processing model training method according to an embodiment of the present disclosure;
[0040] FIG. 5 shows a module block diagram of a language processing model training device according to an embodiment of the present disclosure;
[0041] FIG. 6 shows a block diagram of an electronic device according to an embodiment of the present disclosure.
[0042] FIG. 6 shows a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0043] To make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings can be arranged and designed in various different configurations.
[0044] Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present disclosure.
[0045] It should be noted that the relational terms such as "first" and "second" and the like are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.
[0046] With the rapid development of information technology, as a technology for converting text into natural and fluent speech, speech synthesis technology (Text-To-Speech, TTS) has become an important part of human-computer interaction, directly affecting the user experience. TTS is widely used in outbound broadcasting, voice navigation, audio books and other fields.
[0047] The existing large speech synthesis model mainly includes three parts: a language processing model (Large Language Model, LLM), an acoustic feature prediction model, and a vocoder model. The predicted speech tag data output by the language processing model in the first stage mainly includes semantic information and part of style information. The loss function of the existing language processing model mainly focuses on semantic information and pays less attention to style information. Therefore, when the large speech synthesis model performs speech synthesis, it cannot well reproduce the speech style, resulting in poor performance of the large speech synthesis model in processing style information (such as prosody, emotion, etc.).
[0048] To solve the above problems, the inventors propose a language processing model training method, device, server and storage medium provided in the embodiments of the present disclosure. By introducing a style feature loss and optimizing and training the language processing model according to a total loss function, the performance of the language processing model in processing speech style information is improved, and the performance of the speech synthesis large model in processing speech style information is further improved. The specific language processing model training method is described in detail in subsequent embodiments.
[0049] Referring to FIG. 1, FIG. 1 shows a flowchart of a language processing model training method provided in the embodiments of the present disclosure. The language processing model training method can be applied to an electronic device. In the following, the specific process of the present embodiment is described taking the electronic device as an example. The language processing model training method shown in FIG. 1 can include the following steps:
[0050] Step 110: Obtain training text data and real speech label data.
[0051] The training text data refers to a text sentence that has been sorted and labeled. The training text data is input into the language processing model, and the language processing model can learn how to generate corresponding predicted speech label data according to the training text data and the real speech label data. The real speech label data is discrete labels extracted from the reference speech. The real speech label data includes real semantic information and part of real style information of the reference speech. The real semantic information is the real semantic information of the reference speech, and the real style information is the real style information of the reference speech. The style information may, for example, include speech speed, rhythm, tone, emotion, etc. The content of the reference speech can be the same as that of the training text data, i.e., the training text data is the text sentence of the reference speech.
[0052] In some embodiments, the electronic device can obtain reference speech, preprocess the reference speech, for example, cleaning, denoising, and unification, etc., then extract features from the preprocessed reference speech, obtain speech features, and label each frame of speech features to obtain real speech label data.
[0053] Step 120: Input the training text data and the real speech label data into a prediction module in the language processing model to obtain predicted speech label data output by the prediction module.
[0054] The prediction module can be an LLM model.
[0055] In some embodiments, the electronic device can obtain, by the following formula, predicted speech token data of the training text data output by the prediction module in the language processing model, the predicted speech token data comprising predicted semantic information of the reference speech and predicted style information, the predicted semantic information being predicted semantic information of the reference speech, and the predicted style information being predicted style information of the reference speech. speech_token predict = LLM(speech_token truth , text)
[0056] Wherein, speech_token predict represents the predicted speech token data, LLM represents the LMM model, speech_token truth represents the real speech token data, and text represents the training text data.
[0057] Step 130: obtaining real style features according to the real speech token data, and obtaining predicted style features according to the predicted speech token data.
[0058] In some embodiments, the electronic device can perform style extraction on the real speech token data to obtain real style features, and can also perform style extraction on the predicted speech token data to obtain predicted style features.
[0059] Step 140: obtaining a total loss function according to the real speech token data, the predicted speech token data, the real style features, and the predicted style features.
[0060] In some embodiments, the electronic device can construct a total loss function according to the real speech token data, the predicted speech token data, the real style features, and the predicted style features.
[0061] Step 150: optimizing and training the prediction module according to the total loss function.
[0062] In some embodiments, the prediction module can gradually adjust the internal parameters according to the feedback of the total loss function, improve the ability to capture the predicted speech token data and style information, and the prediction module can continuously iterate according to the total loss function until the total loss function reaches a threshold value, or the total loss function tends to be stable.
[0063] The language processing model training method provided by the embodiments of the present disclosure includes the following steps: obtaining training text data and real speech token data, and inputting the training text data and the real speech token data into a prediction module in a language processing model to obtain predicted speech token data; extracting style features in the real and predicted speech token data respectively; constructing a total loss function according to the real speech token data, the predicted speech token data, real style features and predicted style features; and optimizing and training the prediction module according to the total loss function, and introducing style feature loss into the total loss function to ensure that the language processing model can learn style information, so that the language processing model can more accurately reproduce the speech style, thereby improving the performance of the language processing model in processing speech style information. Moreover, the speech processing model is a key part of a speech synthesis model, and thus the performance of the speech synthesis large model in processing speech style information can be improved.
[0064] To ensure that the language processing model can learn style information, a style encoding module is introduced into the language processing model, and the language processing model includes the style encoding module. As shown in FIG. 2, step 130 specifically includes the following steps:
[0065] Step 131: extracting style from the real speech token data by the style encoding module to obtain real style features.
[0066] The architecture of the speech synthesis large model in which the style encoding module is introduced is shown in FIG. 3. The training text data (text) and the real speech token data (speech_token truth ) are input into the language processing model (LLM) in the first stage of the speech synthesis large model to obtain predicted speech token data (speech_token predict ) output by the language processing model (LLM). The real speech token data (speech_token truth ) is extracted by the style encoding module (style encoder) to obtain real style features (style truth). The predicted speech token data (speech_token predict ) is extracted by the style encoding module (style encoder) to obtain predicted style features (style predict). The predicted speech token data (speech_token predict ) output by the language processing model is input into the acoustic feature prediction model (Token-to-mel) in the second stage of the speech synthesis large model, and then the data output by the acoustic feature prediction model (Token-to-mel) is input into the vocoder model (Mel-to-wav).
[0067] In some embodiments, the style encoding module can perform style extraction on the speech token using Linear Predictive Coding (LPC) technique to obtain style features.
[0068] It should be noted that the LPC model characterizes the spectral characteristics of the speech token data by extracting linear prediction coefficients (LPC coefficients), which can reflect the formant position and shape of the speech token data, thereby indirectly representing the style features of the speech, such as intonation, speech rate, and timbre, etc.
[0069] In some embodiments, the electronic device can obtain the real style features by the following formula. style_hidden truth =Style_encoder(speech_token truth )
[0070] Wherein, style_hidden truth represents the real style features, Style_encoder represents the style encoding module, and speech_token truth represents the real speech token data.
[0071] Please continue to refer to FIG. 2, step 132: performing style extraction on the predicted speech token data by the style encoding module to obtain predicted style features.
[0072] In some embodiments, the electronic device can obtain the predicted style features by the following formula. style_hidden predict =Style_encoder(speech_token predict )
[0073] Wherein, style_hidden predict represents the predicted style features, Style_encoder represents the style encoding module, and speech_token predict represents the predicted speech token data.
[0074] In some embodiments, the style encoding module included in the language processing model is a trained model, and during the training process of the prediction module, the style encoding module serves as part of the supervised language processing model training.
[0075] In other embodiments, the style encoding module included in the language processing model is untrained, and the parameters in the style encoding module can be adjusted simultaneously during the training process of the prediction module.
[0076] It can be understood that by extracting the styles of the real speech token data and the predicted speech token data respectively through the unified style encoding module, it can be ensured that the real style features and the predicted style features are compared under the same standard, thereby improving the accuracy of the style loss function calculation in the training process of the language processing model, and further improving the performance of the language processing model in processing language styles.
[0077] To improve the performance of the language processing model in processing language styles, a style loss function is introduced, as shown in FIG. 4, and step 140 specifically includes the following steps:
[0078] Step 141: obtaining a prediction loss function according to the real speech token data and the predicted speech token data.
[0079] The prediction loss function is configured to measure the difference between the real speech token data and the predicted speech token data.
[0080] In some embodiments, the electronic device can calculate the distribution distance between the real speech token data and the predicted speech token data to obtain the prediction loss function.
[0081] In some embodiments, the electronic device can calculate the distribution distance between the real speech token data and the predicted speech token data by calculating the cross-entropy between the real speech token data and the predicted speech token data to obtain the prediction loss function. Loss ce = Cross_Entropy(speech_token truth , speech_token predict )
[0082] Wherein, Loss ce represents the prediction loss function, Cross_Entropy(speech_token truth , speech_token predict ) represents the cross-entropy loss between the real speech token data and the predicted speech token data, speech_token truth represents the real speech token data, and speech_token predict represents the predicted speech token data.
[0083] Step 142: obtaining a style loss function according to the real style features and the predicted style features.
[0084] In some embodiments, the similarity between the real style features and the predicted style features is calculated to obtain the style loss function.
[0085] The electronic device can calculate the similarity between the real style feature and the predicted style feature by using cosine similarity, KL divergence, dynamic time warping (DTW), and the like.
[0086] In some embodiments, the electronic device can calculate the cosine similarity between the real style feature and the predicted style feature by using the following formula to obtain the prediction loss function. Loss style = Cosine_Embedding (style_hidden truth , style_hidden predict )
[0087] In the formula, Lossstyle represents the style loss function, Cosine_Embedding (style_hidden truth , style_hidden predict ) represents the cosine similarity between the real style feature and the predicted style feature, style_hidden truth represents the real style feature, and style_hidden predict represents the predicted style feature.
[0088] Step 143: Obtain a total loss function according to the prediction loss function and the style loss function.
[0089] In some embodiments, the prediction loss function and the style loss function are weighted to obtain the total loss function.
[0090] In some embodiments, the electronic device has pre-set and stored a preset weight value, and the electronic device can weight the prediction loss function and the style loss function to obtain the total loss function according to the following formula. Loss llm＝ α × Loss ce + (1-α) × Loss style
[0091] In the formula, Loss llm represents the total loss function, α represents the preset weight value, Loss ce represents the prediction loss function, and Loss style represents the style loss function.
[0092] It should be noted that the prediction loss function optimizes the hard matching between the real speech label data and the predicted speech label data. When the difference between the real speech label data and the predicted speech label data is small, the style information contained in the speech label data may not be accurately predicted, and therefore a trade-off is made in the difference between the semantic information and the style information by using the weighted calculation.
[0093] The language processing model training method provided in the embodiments of the present disclosure ensures that the speech generated by the speech synthesis large model is consistent with the style of the training text data and the real speech mark data (the speech mark data corresponding to the reference speech) in terms of semantics and style, so as to realize more accurate speech style control and further improve the experience of users.
[0094] In order to perform the corresponding steps in the above embodiments and various possible manners, an implementation manner of a language processing model training apparatus is given below. Optionally, referring to FIG. 5, which is a functional module diagram of a language processing model training apparatus provided in an embodiment of the present disclosure. It should be noted that the language processing model training apparatus provided in the present embodiment has the same basic principles and technical effects as the above embodiments, and for brief description, the part of the present embodiment not mentioned can be referred to the corresponding content of the above embodiments. The language processing model training apparatus 200 comprises a predicted speech mark data obtaining module 210, a style feature obtaining module 220, a loss function obtaining module 230, and an optimization training module 240, wherein:
[0095] The predicted speech mark data obtaining module 210 is configured to obtain the training text data and the real speech mark data, and input the training text data and the real speech mark data into a prediction module in the language processing model to obtain predicted speech mark data output by the prediction module.
[0096] The style feature obtaining module 220 is configured to obtain real style features according to the real speech mark data, and obtain predicted style features according to the predicted speech mark data.
[0097] The loss function obtaining module 230 is configured to obtain a total loss function according to the real speech mark data, the predicted speech mark data, the real style features, and the predicted style features.
[0098] The optimization training module 240 is configured to perform optimization training on the language processing model according to the total loss function.
[0099] Optionally, the language processing model comprises a style encoding module, and the style feature obtaining module is further configured to perform style extraction on the real speech mark data through the style encoding module to obtain the real style features, and perform style extraction on the predicted speech mark data through the style encoding module to obtain the predicted style features.
[0100] Optionally, the loss function obtaining module 230 is specifically configured to obtain a prediction loss function according to the real speech label data and the predicted speech label data; obtain a style loss function according to the real style feature and the predicted style feature; and obtain a total loss function according to the prediction loss function and the style loss function.
[0101] Optionally, the loss function obtaining module 230 is specifically configured to calculate a distribution distance between the real speech label data and the predicted speech label data to obtain the prediction loss function.
[0102] Optionally, the loss function obtaining module 230 is specifically configured to calculate a similarity between the real style feature and the predicted style feature to obtain the style loss function.
[0103] Optionally, the loss function obtaining module 230 is specifically configured to perform weighted calculation on the prediction loss function and the style loss function to obtain the total loss function.
[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and module can refer to the corresponding process in the foregoing method embodiments, which will not be described here.
[0105] In several embodiments provided by the present disclosure, the coupling between modules can be electrically, mechanically or in other forms of coupling.
[0106] In addition, each functional module in each embodiment of the present disclosure can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.
[0107] Please refer to FIG. 6, which is a block schematic diagram of an electronic device 100 provided by an embodiment of the present disclosure. The electronic device 100 includes a memory 110, a processor 120 and a communication module 130. The memory 110, the processor 120 and the communication module 130 are directly or indirectly electrically connected with each other to realize data transmission or interaction. For example, these elements can be electrically connected with each other through one or more communication buses or signal lines.
[0108] The memory 110 is configured to store data or a program. The memory 110 can be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electric Erasable Programmable Read-Only Memory (EEPROM), etc.
[0109] The processor 120 is configured to read / write the data or program stored in the memory and perform corresponding functions. For example, when the computer program stored in the memory 110 is executed by the processor 120, the language processing model training method disclosed in the above embodiments can be implemented.
[0110] The communication module 130 is configured to establish a communication connection between the electronic device 100 and a cloud server through a network, and configured to transceive data through the network.
[0111] It should be understood that the structure shown in FIG. 6 is only a schematic structure of the electronic device, and the electronic device can further include more or less components than those shown in FIG. 6. The components shown in FIG. 6 can be implemented in hardware, software, or a combination thereof.
[0112] The embodiments of the present disclosure further provide a computer readable storage medium, and the computer readable medium stores a computer program. The computer program can be executed by a processor to implement the language processing model training method described in the above method embodiments.
[0113] The computer readable storage medium can be an electronic storage such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer readable storage medium includes a non-transitory computer readable medium. The computer readable storage medium has a storage space for program codes to execute any of the above methods. The program codes can be read from or written to one or more computer program products. The program codes can be compressed in an appropriate form, for example.
[0114] In several embodiments provided by the present disclosure, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which includes one or more executable instructions configured to implement the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from those noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0115] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure, essentially or the part that contributes to the related art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the embodiments of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc.
[0116] The above is only an optional embodiment of the present disclosure, and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure. Industrial applicability
[0117] According to the scheme, the training text data and the real speech mark data are obtained and input into a prediction module in the language processing model to obtain predicted speech mark data, style features in the real and predicted speech mark data are extracted respectively, a total loss function is constructed according to the real speech mark data, the predicted speech mark data, the real style features and the predicted style features, and the prediction module is optimized and trained according to the total loss function. The style feature loss is introduced into the total loss function to ensure that the language processing model can learn the style information, so that the language processing model can more accurately reproduce the speech style, thereby improving the performance of the language processing model in processing the speech style information. Moreover, the speech processing model is a key part of the speech synthesis model, and thus the performance of the speech synthesis large model in processing the speech style information can be improved.
Claims
1. A method for training a language processing model, characterized in that, The method includes: Obtain training text data and real speech tag data; The training text data and the real speech tag data are input into the prediction module of the language processing model to obtain the predicted speech tag data output by the prediction module. The real style features are obtained based on the real speech tag data, and the predicted style features are obtained based on the predicted speech tag data; Based on the real speech tag data, the predicted speech tag data, the real style features, and the predicted style features, the total loss function is obtained; The prediction module is optimized and trained based on the total loss function.
2. The method according to claim 1, characterized in that, The language processing model includes a style encoding module. The step of obtaining real style features based on the real speech tag data and obtaining predicted style features based on the predicted speech tag data includes: The style encoding module extracts the style from the real speech mark data to obtain the real style features; The predicted style features are obtained by extracting style from the predicted speech tag data through the style encoding module.
3. The method according to claim 1 or 2, characterized in that, The step of obtaining the total loss function based on the real speech tag data, the predicted speech tag data, the real style features, and the predicted style features includes: Based on the real speech tag data and the predicted speech tag data, a prediction loss function is obtained; Based on the true style features and the predicted style features, a style loss function is obtained; The total loss function is obtained based on the prediction loss function and the style loss function.
4. The method according to claim 3, characterized in that, The step of obtaining the prediction loss function based on the real speech tag data and the predicted speech tag data includes: Calculate the distribution distance between the real speech tag data and the predicted speech tag data to obtain the prediction loss function.
5. The method according to claim 3 or 4, characterized in that, The step of obtaining the style loss function based on the true style features and the predicted style features includes: Calculate the similarity between the true style features and the predicted style features to obtain the style loss function.
6. The method according to any one of claims 3-5, characterized in that, The step of obtaining the total loss function based on the prediction loss function and the style loss function includes: The total loss function is obtained by weighting the prediction loss function and the style loss function.
7. The method according to any one of claims 2-6, characterized in that, The step of extracting style from the real speech tag data through the style encoding module to obtain the real style features includes: The style encoding module extracts the style from the real speech tag data to obtain the linear prediction coefficients of the real speech tag data. The linear prediction coefficients of the real speech tag data characterize the spectral characteristics of the real speech tag data. The real style features are obtained based on the spectral characteristics of the real speech tag data.
8. The method according to any one of claims 2-7, characterized in that, The step of extracting style from the predicted speech tag data through the style encoding module to obtain the predicted style features includes: The style encoding module extracts the style from the predicted speech tag data to obtain the linear prediction coefficients of the predicted speech tag data. The linear prediction coefficients of the predicted speech tag data characterize the spectral characteristics of the predicted speech tag data. The predicted style features are obtained based on the spectral characteristics of the predicted speech tag data.
9. The method according to any one of claims 4-8, characterized in that, The step of calculating the distribution distance between the real speech tag data and the predicted speech tag data to obtain the prediction loss function includes: Calculate the cross-entropy between the real speech tag data and the predicted speech tag data; The prediction loss function is obtained based on the cross-entropy.
10. The method according to any one of claims 2-8, characterized in that, The method further includes: The style coding module is optimized and trained based on the total loss function.
11. A language processing model training device, characterized in that, The device includes: A predicted speech tag data acquisition module is configured to acquire training text data and real speech tag data; input the training text data and the real speech tag data into the prediction module in the language processing model to obtain the predicted speech tag data output by the prediction module; The style feature acquisition module is configured to obtain real style features based on the real speech tag data and predictive style features based on the predicted speech tag data. The loss function acquisition module is configured to obtain the total loss function based on the real speech tag data, the predicted speech tag data, the real style features, and the predicted style features; The training module is optimized and configured to perform optimized training on the prediction module based on the total loss function.
12. The apparatus according to claim 11, characterized in that, The language processing model includes a style encoding module, and the style feature acquisition module is further configured to extract style from the real speech tag data through the style encoding module to obtain the real style features; and to extract style from the predicted speech tag data through the style encoding module to obtain the predicted style features.
13. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program executable by the processor, the processor executing the computer program to implement the language processing model training method according to any one of claims 1-10.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the language processing model training method as described in any one of claims 1-10.