Model training method for speech synthesis model, and speech synthesis method

By combining autoregressive and non-autoregressive encoding and decoding, a speech synthesis model is generated, which solves the problems of excessive tokens and loss of acoustic details in the generation of long-sequence speech, and achieves high-quality speech generation, which is suitable for applications such as virtual anchors and audiobooks.

WO2026237497A1PCT designated stage Publication Date: 2026-11-19ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/082421
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-15
Filing Date
2026-03-10
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing speech generation technologies suffer from problems in long sequence processing, such as excessive tokens leading to distraction and affecting text-speech alignment. Furthermore, ultra-high compression rate speech tokenization technology loses key acoustic details and cannot meet the requirements for high-quality speech generation.

Method used

A method combining autoregressive and non-autoregressive encoding and decoding is adopted to generate predicted speech information by using the feature information of adjacent decoded characters and compressed decoded characters, thus preserving key acoustic details and reducing redundant information.

Benefits of technology

It improves the naturalness and fidelity of speech generation, solves the problem of long sequences, ensures speech quality and stability, and is suitable for demanding application scenarios such as virtual anchors and audiobook production.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026082421_19112026_PF_FP_ABST
    Figure CN2026082421_19112026_PF_FP_ABST
Patent Text Reader

Abstract

A model training method for a speech synthesis model, and a speech synthesis method. The model training method for a speech synthesis model comprises: inputting sample text information and reference speech information into an initial speech synthesis model, so as to obtain predicted speech information output by the initial speech synthesis model, wherein the predicted speech information is generated by means of autoregressive prediction information and non-autoregressive prediction information, and a current decoded character in the autoregressive prediction information is generated by means of an adjacent decoded character feature information set and compressed decoded character feature information corresponding to the sample text information and the reference speech information; on the basis of the predicted speech information and label speech information, calculating a model loss value; and, on the basis of the model loss value, adjusting model parameters of the initial speech synthesis model so as to obtain a target speech synthesis model.
Need to check novelty before this filing date? Find Prior Art

Description

Training methods for speech synthesis models, speech synthesis methods

[0001] This disclosure claims priority to Chinese Patent Application No. 202510629720.2, filed with the China Patent Office on May 15, 2025, entitled “Model Training Method for Speech Synthesis Model, Speech Synthesis Method”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of artificial intelligence, and in particular to a method for training a speech synthesis model and a speech synthesis method. Background Technology

[0003] With the development of artificial intelligence technology, the demand for speech generation technology has exploded in fields such as intelligent voice interaction and audio content creation. Users' requirements for voice quality and interactive experience are constantly rising, expecting generated speech to be natural and fluent, highly relevant to the scene, and responsive. Current speech generation technologies face the challenge of long sequences: in neural codec language models, neural audio codecs encode audio into a large number of voice tokens to achieve high-fidelity audio reconstruction. For example, a 7-8 second speech may generate hundreds or even thousands of tokens. Too many tokens can lead to distraction and affect the alignment of text and speech.

[0004] To address these issues, a high-compression voice token technique has been proposed. While this reduces the number of tokens, it also results in the loss of crucial acoustic details, leading to a lack of emotion and expressiveness in the synthesized speech. Other language modeling methods also fail to fully adjust speech generation. Therefore, how to solve the problem of long speech sequences while preserving key acoustic details has become a pressing issue for researchers. Summary of the Invention

[0005] In view of this, embodiments of this disclosure provide a model training method for a speech synthesis model and a speech synthesis method. One or more embodiments of this disclosure also relate to a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the present disclosure, a method for training a speech synthesis model is provided, comprising:

[0007] Obtain speech synthesis training samples, wherein the speech synthesis training samples include sample text information, reference speech information, and labeled speech information;

[0008] The sample text information and the reference speech information are input into the initial speech synthesis model to obtain the predicted speech information output by the initial speech synthesis model. The predicted speech information is generated by autoregressive prediction information and non-autoregressive prediction information. The current decoded character in the autoregressive prediction information is generated by the set of adjacent decoded character feature information and compressed decoded character feature information corresponding to the sample text information and the reference speech information. The non-autoregressive prediction information is generated by the sample text information and the reference speech information. The set of adjacent decoded character feature information is the set of already decoded character feature information adjacent to the current decoded character. The compressed decoded character feature information is the compressed feature information of already decoded character feature information that is more than a preset threshold away from the current decoded character.

[0009] Calculate the model loss value based on the predicted speech information and the labeled speech information;

[0010] The model parameters of the initial speech synthesis model are adjusted based on the model loss value to obtain the target speech synthesis model.

[0011] According to a second aspect of the present disclosure, a speech synthesis method is provided, comprising:

[0012] Obtain the text information to be processed and the reference speech information;

[0013] The text information to be processed and the reference speech information are input into the target speech synthesis model to obtain the predicted speech information output by the target speech synthesis model, wherein the target speech synthesis model is trained by the above training method.

[0014] According to a third aspect of the present disclosure, a method for training a speech synthesis model is provided, applied to a cloud-based device, comprising:

[0015] The speech synthesis training samples sent by the receiving end device include sample text information, reference speech information, and labeled speech information.

[0016] The sample text information and the reference speech information are input into the initial speech synthesis model to obtain the predicted speech information output by the initial speech synthesis model. The predicted speech information is generated by autoregressive prediction information and non-autoregressive prediction information. The current decoded character in the autoregressive prediction information is generated by the set of adjacent decoded character feature information and compressed decoded character feature information corresponding to the sample text information and the reference speech information. The non-autoregressive prediction information is generated by the sample text information and the reference speech information. The set of adjacent decoded character feature information is the set of already decoded character feature information adjacent to the current decoded character. The compressed decoded character feature information is the compressed feature information of already decoded character feature information that is more than a preset threshold away from the current decoded character.

[0017] The model loss value is calculated based on the predicted speech information and the labeled speech information, and the model parameters of the initial speech synthesis model are adjusted based on the model loss value to obtain the target speech synthesis model.

[0018] Obtain the model parameters of the target speech synthesis model and send the model parameters to the end device.

[0019] According to a fourth aspect of the present disclosure, a speech synthesis method is provided, applied to a cloud-side device, comprising:

[0020] The receiving end sends the text information to be processed and the reference voice information;

[0021] The text information to be processed and the reference speech information are input into the target speech synthesis model to obtain the predicted speech information output by the target speech synthesis model, wherein the target speech synthesis model is trained by the above training method;

[0022] The predicted speech information is sent to the end-side device.

[0023] According to a fifth aspect of the present disclosure, a task platform is provided, including a request interface and a response unit;

[0024] The request interface is used to receive unprocessed text information and reference voice information sent by the end device;

[0025] The response unit is used to input the text information to be processed and the reference speech information into the target speech synthesis model to obtain the predicted speech information output by the target speech synthesis model, wherein the target speech synthesis model is trained by the above-described training method.

[0026] According to a sixth aspect of the present disclosure, a computing device is provided, comprising:

[0027] Memory and processor;

[0028] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0029] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0030] According to an eighth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0031] The speech synthesis model training method provided in this disclosure performs autoregressive and non-autoregressive encoding / decoding on sample text information and reference speech information respectively in the speech synthesis model. Then, it fuses the autoregressive and non-autoregressive prediction information to generate predicted speech information. When generating the autoregressive prediction information, it references a set of adjacent decoded character feature information and compressed decoded character feature information. The set of adjacent decoded character feature information represents local feature information, while the compressed decoded character feature information represents long-distance feature information. This ensures that when generating the final predicted speech information, the set of adjacent decoded character feature information preserves the high-fidelity reconstruction capability of the encoder / decoder, while the compressed decoded character feature information reduces redundant information and avoids the loss of key acoustic details, guaranteeing a high degree of naturalness and fidelity in the output predicted speech information. Attached Figure Description

[0032] Figure 1 is a flowchart of a model training method for a speech synthesis model provided in an embodiment of this disclosure;

[0033] Figure 2 is a schematic diagram of the model architecture of a speech synthesis model provided in an embodiment of this disclosure;

[0034] Figure 3 is a flowchart of the processing procedure of a speech synthesis method provided in an embodiment of this disclosure;

[0035] Figure 4 is a flowchart of the processing procedure of a model training method for a speech synthesis model provided in an embodiment of this disclosure;

[0036] Figure 5 is a flowchart of the processing procedure of a speech synthesis method provided in another embodiment of this disclosure;

[0037] Figure 6 is a schematic diagram of the architecture of a task platform provided in an embodiment of this disclosure;

[0038] Figure 7 is a schematic diagram of the structure of a model training device for a speech synthesis model provided in an embodiment of this disclosure;

[0039] Figure 8 is an architecture diagram of a model training system for a speech synthesis model provided in an embodiment of this disclosure;

[0040] Figure 9 is a structural block diagram of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0041] Numerous specific details are set forth in the following description to provide a full understanding of this disclosure. However, this disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this disclosure. Therefore, this disclosure is not limited to the specific implementations disclosed below.

[0042] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this disclosure, and similarly, second may also be referred to as first. Depending on the context, the word “if” as used herein may be interpreted as “when”, “in response to a determination”, or “when…”.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0045] First, the terms and concepts involved in one or more embodiments of this disclosure will be explained.

[0046] Neural Audio Codec: Also known as an audio segmenter, it is an algorithm used to compress and encode audio signals. It uses neural networks to learn and represent the features of audio signals, and then encodes them into digital signals for transmission and storage in computers.

[0047] Speech token: The codebook number of the audio signal after encoding and quantization.

[0048] codebook: In a neural audio encoder, the codebook is used to store the continuous representation of the encoded audio signal.

[0049] Vector quantization (VQ) maps a continuous input space to a finite, discrete codebook, thereby achieving an approximate representation of the input signal. In VQ, a multidimensional vector is mapped to a vector in the codebook; this process is called quantization.

[0050] With the development of artificial intelligence technology, the demand for speech generation has exploded in fields such as intelligent voice interaction and audio content creation. Users' requirements for voice quality and interactive experience are constantly rising, expecting generated speech to be natural, fluent, highly context-appropriate, and responsive. However, current speech synthesis technology faces many bottlenecks. For example, speech generation faces the challenge of long sequences: in neural codec language models, neural audio codecs encode audio into a large number of voice tokens to achieve high-fidelity audio reconstruction. For example, a 7-8 second speech may generate hundreds or even thousands of tokens, making context modeling extremely difficult. An excessive number of tokens leads to attentional distraction, affecting the accurate alignment of text and speech, while increasing redundant information, interfering with speech prediction, and reducing the accuracy and stability of the model.

[0051] To address these issues, researchers proposed an ultra-high compression rate speech tokenization technique. This approach encodes speech into syllable-like units, significantly compressing the token rate from 50Hz to 4-5Hz. The aim is to reduce the number of tokens while preserving key semantic information, thus alleviating the challenges of long context modeling for language models caused by long speech token sequences. However, this over-compression leads to the loss of crucial acoustic details such as prosody and timbre. These details are essential for high-quality speech reconstruction and generation; their loss results in synthesized speech lacking emotion, expressiveness, and naturalness, severely impacting speech quality and failing to meet the demands of applications requiring high-quality speech, such as virtual broadcasters and audiobook production.

[0052] There are also language modeling schemes based on specific architectures, such as group encoding / decoding modeling, local-global-local modeling, and forced text-speech token alignment. Group encoding / decoding modeling processes speech tokens through specific grouping methods, local-global-local modeling optimizes modeling by starting with contextual information from different ranges, and forced alignment schemes aim to make text and speech tokens correspond better. These schemes alleviate the sparsity problem in long speech token sequences to some extent, allowing language models to focus more on key information when processing context. However, they do not completely address the redundancy problem; redundant information generated by speech signal discretization still exists, which can still interfere with the prediction of the current speech token, reduce the robustness of the model, and affect the accuracy and stability of speech generation.

[0053] The LLMLingua cue compression scheme employs a coarse-to-fine strategy, using a budget controller, an iterative token-level cue compression algorithm, and a distribution alignment method to compress long cues into shorter ones, thereby accelerating model inference and reducing costs. However, it primarily focuses on text cue compression, which differs from the speech token sequence processing scenario in speech generation. When applied to speech generation, it struggles to directly handle long-distance dependencies and redundancy issues in speech token sequences, failing to effectively improve the quality and efficiency of speech generation and lacking sufficient specificity for speech generation tasks.

[0054] Based on this, this disclosure provides a model training method for a speech synthesis model and speech synthesis. This disclosure also relates to a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0055] Referring to Figure 1, Figure 1 shows a flowchart of a model training method for a speech synthesis model according to an embodiment of the present disclosure, which specifically includes the following steps.

[0056] Step 102: Obtain speech synthesis training samples, wherein the speech synthesis training samples include sample text information, reference speech information, and labeled speech information.

[0057] The method provided in this disclosure provides a method for training a speech synthesis model. The method provided in this disclosure adopts a supervised training approach. By inputting sample data into the speech synthesis model, processing it, outputting a prediction result, comparing the prediction result with the sample labels in the training samples, and adjusting the model parameters of the speech synthesis model based on the comparison result, the model training of the speech synthesis model is achieved.

[0058] Specifically, in the model training method provided in this embodiment, speech synthesis training samples are first obtained. These training samples include sample text information, reference speech information, and labeled speech information. Sample text information refers to the text information corresponding to the speech to be generated by the speech synthesis model; reference speech information is the timbre information corresponding to the final output speech of the speech synthesis model; and labeled speech information is the speech information used to read the sample text information based on the reference speech information. For example, the sample text information might be "The weather is nice today, and I'm in a good mood," the reference speech information might be a segment of speech by character A, and the labeled speech information might be the speech information of character A reading "The weather is nice today, and I'm in a good mood."

[0059] The method provided in this embodiment is supervised training, which obtains speech synthesis training samples to provide data support in subsequent model training.

[0060] Step 104: Input the sample text information and the reference speech information into the initial speech synthesis model to obtain the predicted speech information output by the initial speech synthesis model. The predicted speech information is generated by autoregressive prediction information and non-autoregressive prediction information. The current decoded character in the autoregressive prediction information is generated by the set of adjacent decoded character feature information corresponding to the sample text information and the reference speech information and the compressed decoded character feature information. The non-autoregressive prediction information is generated by the sample text information and the reference speech information.

[0061] The initial speech synthesis model can be understood as a speech synthesis model that has not yet been trained. In the method provided in this embodiment, the purpose is to train the initial speech synthesis model with speech synthesis training samples, adjust the model parameters in the initial speech synthesis model, and thus obtain a trained speech synthesis model.

[0062] The adjacent decoded character feature information set is the set of already decoded character feature information adjacent to the current decoded character, and the compressed decoded character feature information is the compressed feature information of already decoded character feature information that is more than a preset threshold away from the current decoded character.

[0063] In this embodiment, sample text information and reference speech information are needed during the speech synthesis process using the initial speech synthesis model. Within the initial speech synthesis model, autoregressive prediction information and non-autoregressive prediction information are generated through the model's network structure, thereby synthesizing predicted speech information based on these two information. The non-autoregressive prediction information is generated from the sample text information and the reference speech information. The current decoded character in the autoregressive prediction information is generated from the set of adjacent decoded character feature information and compressed decoded character feature information corresponding to the sample text information and the reference speech information. The set of adjacent decoded character feature information is the set of already decoded character feature information adjacent to the current decoded character, and the compressed decoded character feature information is the compressed feature information of already decoded character feature information that is more than a preset threshold away from the current decoded character.

[0064] Autoregressive prediction information is generated through an autoregressive encoder-decoder network. In the decoder of this network, the decoder part is autoregressive, meaning that when generating each output token, it relies on previously generated tokens. This implies that the decoder automatically regresses when generating sequences, utilizing previously generated token information each time it predicts the next token.

[0065] In the method provided in this disclosure, the autoregressive prediction information is generated from the set of already predicted adjacent decoded character feature information and compressed decoded character feature information. The set of adjacent decoded character feature information refers to the set of already decoded tokens adjacent to the current decoded character. The compressed decoded character feature information can be understood as the compressed feature information of already decoded tokens that are more than a preset threshold away from the current decoded character. In the method provided in this disclosure, if the current decoded character is located relatively late in the sentence, the redundant information of the already decoded tokens increases, which may interfere with speech prediction. To ensure the prediction accuracy of the current decoded character, the already predicted tokens can be compressed. When generating the current decoded character, the compressed feature information of already decoded tokens that exceed the preset threshold can be referenced, thereby effectively filtering noise and redundancy in long-distance information.

[0066] In one specific embodiment provided in this disclosure, the initial speech synthesis model includes an embedding layer, an autoregressive encoding / decoding layer, a non-autoregressive encoding / decoding layer, and a speech decoding layer;

[0067] In the method provided in this embodiment, the initial speech synthesis model is further structurally defined. The initial speech synthesis model includes an embedding layer, an autoregressive encoding / decoding layer, a non-autoregressive encoding / decoding layer, and a speech decoding layer. In practical applications, sample text information and reference speech information are input into the initial speech synthesis model. After processing by the embedding layer, the sample text information and reference speech information are mapped to corresponding spatial feature information. Then, after processing by the autoregressive encoding layer and the non-autoregressive encoding / decoding layer respectively, predicted speech information is generated.

[0068] The sample text information and the reference speech information are input into the initial speech synthesis model to obtain the predicted speech information output by the initial speech synthesis model, including:

[0069] S1042. Input the sample text information and the reference speech information into the embedding layer to obtain the spliced ​​sample feature information.

[0070] Embedding layers are primarily used to map discrete input data (such as characters, tokens, etc.) into a continuous vector space. In natural language processing tasks, embedding layers are used to convert each word or token in the vocabulary into a fixed-dimensional numerical vector that captures the semantic and contextual information of the word.

[0071] The sample text information and reference speech information are input into the embedding layer for vectorization processing to obtain concatenated sample feature information. This concatenated sample feature information is the feature information output by the embedding layer, which is generated by concatenating the vectorized sample text information and reference speech information.

[0072] In one specific embodiment provided in this disclosure, the embedding layer includes a text embedding unit, a speech embedding unit, and a fusion unit;

[0073] The sample text information and the reference speech information are input into the embedding layer to obtain concatenated sample feature information, including:

[0074] The sample text information is input into the text embedding unit to obtain text feature information;

[0075] The reference speech information is input into the speech embedding unit to obtain speech feature information;

[0076] The text feature information and the speech feature information are input into the fusion unit to generate spliced ​​sample feature information.

[0077] In practical applications, the embedding layer includes a text embedding unit, a speech embedding unit, and a fusion unit. The text embedding unit converts text information into vector feature information, and the speech embedding unit converts speech information into vector feature information. The fusion unit concatenates and fuses the vector feature information output by the text embedding unit and the speech embedding unit.

[0078] Specifically, the sample text information is input into the text embedding unit for embedding processing to obtain text feature information, which is the vectorized representation of the sample text information in vector space. Reference speech information is input into the speech embedding unit for processing to obtain speech feature information, which is the vectorized representation of the reference speech information in vector space. After obtaining the text and speech feature information, they are input into the fusion unit for concatenation and fusion to obtain concatenated sample feature information. This ensures that the generated concatenated sample feature information references both text and speech information, providing data support for subsequent processing.

[0079] S1044. Input the spliced ​​sample feature information into the autoregressive encoder-decoder layer to obtain autoregressive prediction information.

[0080] An autoregressive encoder-decoder layer is a neural network architecture commonly used for sequence generation tasks. Autoregressive decoders are typically used in conjunction with encoder-decoder network architectures, where the encoder encodes the input sequence into a context vector, and the autoregressive decoder generates the output sequence step-by-step based on the context vector.

[0081] In an autoregressive encoder-decoder layer, the decoder is autoregressive, meaning that the generation of each output token depends on previously generated tokens. Typically, at each time step, the decoder receives the symbols generated in the previous time step as input, while using the context vector generated by the encoder to guide the generation process. In this way, the autoregressive decoder can progressively generate the output sequence, maintaining the syntactic and semantic coherence of the sequence.

[0082] In the method provided in this embodiment, the spliced ​​sample feature information is input into an autoregressive encoder-decoder layer for processing to obtain autoregressive prediction information. The autoregressive prediction information can be understood as the information predicted at the current time step.

[0083] In one specific embodiment provided in this disclosure, the autoregressive codec layer includes an encoder and an autoregressive decoder;

[0084] The spliced ​​sample feature information is input into the autoregressive encoder-decoder layer to obtain autoregressive prediction information, including:

[0085] The spliced ​​sample feature information is input into the encoder to obtain autoregressive coding features;

[0086] The autoregressive encoded features are input into the autoregressive decoder to obtain autoregressive prediction information.

[0087] Specifically, the autoregressive encoder-decoder layer includes an encoder and an autoregressive decoder. The concatenated sample feature information is input into the encoder for encoding processing to obtain autoregressive encoded features. These autoregressive encoded features are then input into the autoregressive decoder for preliminary decoding processing to obtain autoregressive prediction information. In the generation process of the autoregressive prediction information provided in this embodiment, the next element is predicted recursively, with each prediction building upon previous predictions. This facilitates the capture of long-distance dependencies and ensures that subsequent outputs contain predictive speech information with high naturalness and fidelity.

[0088] Specifically, the autoregressive encoded features are input into the autoregressive decoder to obtain autoregressive prediction information, including:

[0089] The autoregressive encoded features are input into the autoregressive decoder, and decoded character feature information is output sequentially. The current decoded character feature information is generated based on the previously output decoded character feature information, which includes a set of adjacent decoded character feature information and compressed decoded character feature information.

[0090] In this embodiment, in the autoregressive codec layer, autoregressive encoded features are input to the autoregressive decoder for decoding processing, and decoded character feature information is output sequentially. The current decoded character feature information output at the current time step is generated based on the already generated output decoded character feature information. In the method provided in this embodiment, the output decoded character feature information includes a set of adjacent decoded character feature information and compressed decoded character feature information.

[0091] Specifically, the set of adjacent decoded character feature information refers to the set of character feature information of a preset length adjacent to the current decoded character feature information, which is referred to as the set of local feature information in this embodiment. For example, taking a preset length of 30 characters as an example, when the current decoded character feature information is predicted to be the 60th token, the set of adjacent decoded character feature information is the set of feature information of the 30 tokens counting forward from the 60th token, that is, the set of adjacent decoded character feature information of the 30th to 59th tokens is the set of adjacent decoded character feature information of the 60th token.

[0092] In this embodiment, the compressed decoded character feature information can be referred to as long-distance feature information. Researchers have found that speech synthesis models exhibit a significant short-distance information dependency phenomenon; that is, the concatenated sample feature information and the set of adjacent decoded character feature information already contain the semantic and acoustic features required for prediction, maintaining text-speech alignment and preserving linguistic features. However, distant decoded character feature information may introduce redundant information, interfering with the performance of the speech synthesis model.

[0093] However, although there is information redundancy in the decoded character feature information that is far away, some of the information still has potential value for predicting the current token. Therefore, in the method provided in this disclosure embodiment, the decoded characters and feature information that are far away are divided into multiple non-overlapping groups. Each group includes a preset number of decoded character feature information. Compressed decoded character feature information is added after each group. By setting an attention mask, the compressed decoded character feature information only focuses on the decoded character feature information in the corresponding group. When predicting the current decoded character feature information, the compressed decoded character feature information that is far away is used. The compressed decoded character feature information that is more than a preset character length away from the current decoded character feature information is used for decoding. This can effectively filter noise and redundancy and retain the main semantic information.

[0094] In another specific embodiment provided in this disclosure, the method further includes:

[0095] At least one group of decoded character features is determined based on the output decoded character feature information and the preset compressed character threshold.

[0096] The decoded character feature information in each decoded character feature group is compressed and fused to obtain the compressed decoded character feature information corresponding to each decoded character feature group.

[0097] In this embodiment, the preset compressed character threshold refers to the number of decoded character feature information in a group. For example, when the preset compressed character threshold is 10 tokens, every 10 tokens constitute a decoded character feature group. Once the preset compressed character threshold is determined, the decoded character feature groups can be determined based on the output decoded character features.

[0098] In the method provided in this embodiment, the autoregressive decoder outputs token by token. Therefore, whenever the output decoded character feature information meets the preset compressed character threshold, a decoded character feature group can be determined.

[0099] After determining the decoded character feature group, a self-attention mask is set to compress and fuse the decoded character feature information within the feature group, generating compressed decoded character feature information corresponding to the decoded character feature group. Specifically, a pre-defined mask is set in the self-attention module to control the direction of information flow, and average pooling or other pooling strategies are used to process the decoded character feature information to obtain compressed decoded character feature information. By compressing and fusing the decoded character feature information within the decoded character feature group, the main semantic information of the feature group can be retained, noise and redundancy can be eliminated, and a reference can be provided for predicting decoded character feature information at longer distances.

[0100] In one specific embodiment provided in this disclosure, the current decoded character feature information is generated through the following steps:

[0101] Determine the character position information of the currently decoded character feature information;

[0102] When the character position information is less than or equal to the preset character length information, the current decoded character feature information is generated based on the set of adjacent decoded character feature information.

[0103] If the character position information is greater than the preset character length information, the current decoded character feature information is generated based on the set of adjacent decoded character feature information and the compressed decoded character feature information.

[0104] In the process of generating the current decoded character feature information, it is necessary to consider the character position information of the current decoded character feature information. The character position information can be understood as the position information of the current decoded character feature information in the generated decoding sequence. For example, if the character position information of the current decoded character feature information is 15, then the current decoded character feature information is the 15th token.

[0105] In the method provided in this embodiment, the current decoded character feature information is generated based on the set of adjacent decoded character feature information, and the set of adjacent decoded character feature information is determined based on a preset length. Therefore, when the character position information is less than or equal to the preset character length information, it is only necessary to determine the current decoded character feature information based on the set of adjacent decoded character feature information.

[0106] For example, if the preset character length is 30 tokens, it means that the current decoded character feature information is predicted based on the 30 previously predicted tokens. If the character position information is 20, it means that the currently predicted decoded sequence only has 19 tokens. In this case, the 19 predicted tokens are less than the preset character length, so only the set of local feature information needs to be processed.

[0107] If the character position information is greater than the preset character length information, the current decoded character feature information can be generated based on the set of adjacent decoded character feature information and the compressed decoded character feature information.

[0108] For example, still taking a preset character length of 30 tokens as an example, the output decoded character feature information in the decoded character feature group corresponding to the compressed decoded character feature information is 10 tokens. If the character position information is 70, it means that there are currently 69 tokens in the predicted decoded sequence. The 40th to 69th tokens are selected as the set of adjacent decoded character feature information. The first 40 tokens correspond to 4 compressed decoded character feature information. When the current decoded character feature information is the 70th token, it is necessary to fuse the adjacent decoded character feature information set composed of the 40th to 69th tokens and the first 4 compressed decoded character feature information to generate the current character feature information. Through the method provided in this embodiment, information on decoded character features at close range can be referenced, as well as information on decoded character features at greater distances. This preserves the high-fidelity reconstruction capability of the codec while reducing redundant information and avoiding the loss of key acoustic details.

[0109] The method provided in this disclosure allows for the gradual acquisition of autoregressive prediction information through an autoregressive encoding / decoding layer, i.e., the gradual acquisition of feature information for each decoded character.

[0110] S1046. Input the spliced ​​sample feature information into the non-autoregressive encoding / decoding layer to obtain non-autoregressive prediction information.

[0111] A non-autoregressive codec layer can be understood as a non-autoregressive codec, which is also an encoder-decoder network architecture. In practical applications, the encoder of the non-autoregressive codec layer and the encoder of the autoregressive codec layer can be the same or different. In the method provided in the embodiments of this disclosure, there is no limitation on this.

[0112] Non-autoregressive codecs can improve decoding efficiency through parallel decoding. In a non-autoregressive codec, the decoder can predict all character information at once when decoding coded feature information, thus obtaining non-autoregressive prediction information.

[0113] S1048. Input the autoregressive prediction information and the non-autoregressive prediction information into the speech decoding layer to obtain predicted speech information.

[0114] After obtaining the autoregressive prediction information and the non-autoregressive prediction information, both are input into the speech decoding layer. The autoregressive prediction information and the non-autoregressive prediction information are concatenated and decoded in the speech decoding layer to obtain the predicted speech information.

[0115] In one specific embodiment provided in this disclosure, the autoregressive prediction information and the non-autoregressive prediction information are input into the speech decoding layer to obtain predicted speech information, including:

[0116] Determine the current decoded character feature information corresponding to the autoregressive prediction information;

[0117] Based on the current decoded character feature information, a reference decoded character feature information is determined from the non-autoregressive prediction information;

[0118] The current decoded character feature information and the reference decoded character feature information are input into the speech decoding layer to obtain the predicted character speech information corresponding to the current decoded character feature information;

[0119] Predicted speech information is generated based on the predicted character speech information corresponding to the feature information of each decoded character.

[0120] In the method provided in this embodiment, the autoregressive prediction information is decoded character feature information output one by one, while the non-autoregressive prediction information is decoded character feature information output all at once. To ensure alignment and consistency between the autoregressive and non-autoregressive prediction information, the current decoded character feature information can be determined from the autoregressive prediction information, and a reference decoded character feature information can be determined from the non-autoregressive prediction information based on the position information of the current decoded character feature information. The reference decoded character feature information is a non-autoregressive prediction information character that has a corresponding positional relationship with the current decoded character feature information in the non-autoregressive prediction information.

[0121] After determining the current decoded character feature information and the reference decoded character feature information, these two pieces of information are input into the speech decoding layer for decoding, thereby outputting the predicted character speech information corresponding to the current decoded character feature information. Based on the same method, the final predicted speech information can be generated according to the predicted character speech information corresponding to each decoded character feature information.

[0122] Referring to Figure 2, which shows a schematic diagram of the model architecture of a speech synthesis model provided in an embodiment of the present disclosure, the speech synthesis model includes an embedding layer (including a speech embedding unit and a text embedding unit (not shown)), an autoregressive encoding and decoding layer, a non-autoregressive encoding and decoding layer, and a speech decoding layer.

[0123] In this embodiment, sample text information and reference speech information are input. The sample text information is processed by the text embedding unit to obtain text feature information, and the reference speech information is processed by the speech embedding unit to obtain speech feature information.

[0124] Text and speech feature information are input into the autoregressive encoder-decoder layer for processing to obtain the current decoded character feature information. Simultaneously, during the autoregressive encoder-decoder process, decoded character feature groups are formed at every predetermined number of decoded character features. The decoded character features within these groups are then compressed and fused to obtain compressed decoded character feature information (W0 and W1 in Figure 2). A predetermined number of previously predicted decoded character features are used as the set of adjacent decoded character feature information. When predicting the current decoded character feature information, the adjacent decoded character feature information set and the compressed decoded character feature information are referenced. Furthermore, the current decoded compressed character feature is used as the reference adjacent decoded character feature information for the next decoded compressed character feature.

[0125] Simultaneously, textual and speech feature information are input into a non-autoregressive codec layer for processing to obtain non-autoregressive prediction information. The non-autoregressive codec employs a sliding window approach for decoding, with the window size being the character space corresponding to a preset distance before and after the reference decoded character feature information. Specifically, reference decoded character feature information is determined from the non-autoregressive prediction information based on the current decoded character feature information. This reference decoded character feature information is determined based on a set of reference decoded character feature information, which can consist of adjacent decoded character feature information.

[0126] After determining the features of the current decoded compressed character and the reference decoded character, the two feature information are input into the speech decoding layer for feature concatenation and decoding, thereby outputting the predicted character speech information.

[0127] After the feature information of each decoded character has been processed as described above, the predicted speech information output by the speech synthesis model can be obtained.

[0128] Step 106: Calculate the model loss value based on the predicted speech information and the labeled speech information.

[0129] After obtaining the predicted speech information, the model loss value can be calculated based on the predicted speech information and the labeled speech information. In the method provided in this disclosure, there are many ways to calculate the model loss value, such as cross-entropy loss function, maximum loss function, average loss function, etc. In this disclosure, the specific method of loss function is not limited, and the actual application shall prevail.

[0130] Step 108: Adjust the model parameters of the initial speech synthesis model according to the model loss value to obtain the target speech synthesis model.

[0131] After obtaining the model loss value, the model parameters of the initial speech synthesis model can be adjusted according to the model loss value. Specifically, the model parameters of the initial speech synthesis model can be updated sequentially by backpropagating the model loss value.

[0132] After adjusting the model parameters, the above steps can be repeated to continue training the initial speech synthesis model until the training stopping condition is met. In practical applications, the training stopping condition for the initial speech synthesis model includes:

[0133] The model loss value is less than a preset threshold; and / or

[0134] The training rounds have reached the preset number of training rounds.

[0135] Specifically, during the training of the initial speech synthesis model, the training stopping condition can be set to the model loss value being less than a preset threshold, or the training stopping condition can be set to the number of training rounds as preset, such as 10 rounds. In this disclosure, the preset threshold for the loss value and / or the preset number of training rounds are not specifically limited, and the actual application shall prevail.

[0136] After the training stopping condition is met, the target speech synthesis model is obtained. The model structure of the target speech synthesis model is the same as that of the initial speech synthesis model. The difference between the two lies in the model parameters.

[0137] The speech synthesis model training method provided in this disclosure performs autoregressive and non-autoregressive encoding / decoding on sample text information and reference speech information respectively in the speech synthesis model. Then, it fuses the autoregressive and non-autoregressive prediction information to generate predicted speech information. When generating the autoregressive prediction information, it references a set of adjacent decoded character feature information and compressed decoded character feature information. The set of adjacent decoded character feature information represents local feature information, while the compressed decoded character feature information represents long-distance feature information. This ensures that when generating the final predicted speech information, the set of adjacent decoded character feature information preserves the high-fidelity reconstruction capability of the encoder / decoder, while the compressed decoded character feature information reduces redundant information and avoids the loss of key acoustic details, guaranteeing a high degree of naturalness and fidelity in the output predicted speech information.

[0138] Specifically, this method improves the structure of the speech synthesis model by incorporating autoregressive and non-autoregressive codec layers. Autoregressive prediction information is obtained from the autoregressive codec layer, and non-autoregressive prediction information is obtained from the non-autoregressive codec layer. This ensures that the high-fidelity reconstruction capability of the original codec is preserved when generating the final predicted speech information, reducing redundant information and avoiding the loss of key acoustic details, thus maintaining the naturalness and fidelity of the speech.

[0139] Secondly, the method provided in this embodiment reduces the length of the input sequence and the computational load by using the set of adjacent decoded character feature information. Simultaneously, compressing long-distance feature information to obtain compressed decoded character feature information effectively removes redundant information, rather than relying solely on complex attention mechanisms to capture long-distance dependencies, thus effectively avoiding performance degradation caused by processing redundant information.

[0140] Furthermore, this method does not rely on specific speech data features and task scenarios for preprocessing and feature selection. Instead, it uses a unified processing approach based on the short-range dependency characteristics of speech feature sequences. This makes it applicable to different neural audio codecs and language models, avoiding the need to redesign processing methods due to scenario changes and preventing the deletion of useful information, thus ensuring the integrity of speech generation. This scheme focuses on the speech token sequence itself, without relying on additional multimodal data, avoiding the complex process of multimodal data acquisition, fusion, and information alignment, thereby reducing system cost and complexity.

[0141] Figure 3 shows a flowchart of the processing procedure of a speech synthesis method provided in an embodiment of the present disclosure, which specifically includes the following steps.

[0142] Step 302: Obtain the text information to be processed and the reference speech information.

[0143] The text information to be processed is the text information for which speech reading is to be generated in this embodiment. The reference speech information can be understood as the speech information of the character to be used in this embodiment.

[0144] For example, in one specific embodiment provided in this disclosure, if we want to synthesize a voice message from character B saying "The vehicle ahead is moving slowly, please control your speed," then we obtain the text information to be processed, "The vehicle ahead is moving slowly, please control your speed," and simultaneously obtain any segment of voice information from character B as reference voice information.

[0145] Step 304: Input the text information to be processed and the reference speech information into the target speech synthesis model to obtain the predicted speech information output by the target speech synthesis model, wherein the target speech synthesis model is trained by the above-mentioned speech synthesis model training method.

[0146] In this embodiment, the text information to be processed and the reference speech information are input into the target speech synthesis model for processing to obtain the predicted speech information output by the target speech synthesis model. The predicted speech information is the speech of the voice with the timbre and rhythm of the reference speech information reading the text information to be processed.

[0147] Specifically, the target speech synthesis model is trained using the model training method for the speech synthesis model described in the above embodiments. The model structure of the target speech synthesis model is shown in the model architecture diagram in Figure 2 above. The internal processing methods of the target speech synthesis model are the same as those of the initial speech synthesis model described above, and will not be repeated here.

[0148] The method provided in this disclosure provides two different inference strategies: a normal inference strategy and a fast inference strategy.

[0149] Under the normal inference strategy, the attention mask setting can make the compressed decoded character feature information compactly visible, while the decoded character feature information at a greater distance is not visible. However, the processing speed improvement of the normal inference mode is not significant.

[0150] Under the fast inference strategy, the compressed and decoded character feature information is pre-cached. When needed, the compressed and decoded character feature information is directly retrieved from the cache, thereby reducing computational complexity and speeding up the inference process.

[0151] The method provided in this disclosure, without relying on specific neural audio codecs and language models, can adapt to mainstream encoder-decoder architectures by inserting compressed decoding character feature information and corresponding attention masks. This method can accelerate inference speed and enable the system to respond quickly while ensuring speech generation quality, meeting the needs of application scenarios with high real-time requirements.

[0152] Figure 4 shows a flowchart of a model training method for a speech synthesis model provided in an embodiment of this disclosure. The method is applied to a cloud-side device and specifically includes the following steps.

[0153] Step 402: Receive speech synthesis training samples sent by the receiving end device, wherein the speech synthesis training samples include sample text information, reference speech information, and labeled speech information.

[0154] Step 404: Input the sample text information and the reference speech information into the initial speech synthesis model to obtain the predicted speech information output by the initial speech synthesis model. The predicted speech information is generated by autoregressive prediction information and non-autoregressive prediction information. The current decoded character in the autoregressive prediction information is generated by the set of adjacent decoded character feature information corresponding to the sample text information and the reference speech information and the compressed decoded character feature information. The non-autoregressive prediction information is generated by the sample text information and the reference speech information.

[0155] The adjacent decoded character feature information set is the set of already decoded character feature information adjacent to the current decoded character, and the compressed decoded character feature information is the compressed feature information of already decoded character feature information that is more than a preset threshold away from the current decoded character.

[0156] Step 406: Calculate the model loss value based on the predicted speech information and the labeled speech information, and adjust the model parameters of the initial speech synthesis model based on the model loss value to obtain the target speech synthesis model.

[0157] Step 408: Obtain the model parameters of the target speech synthesis model and send the model parameters to the end-side device.

[0158] In practical applications, training a model requires a large amount of data and significant computing resources, which edge devices may lack. Therefore, the model training process can be implemented on cloud devices. After obtaining the model parameters of the target speech synthesis model, the cloud device can send these parameters to the edge device. The edge device can then build the target speech synthesis model locally based on these parameters and further utilize it for speech synthesis.

[0159] The speech synthesis model training method provided in this disclosure is deployed on cloud-side devices, avoiding the problem of poor training results caused by insufficient performance of edge devices. Furthermore, this method improves the model structure of the speech synthesis model by setting autoregressive and non-autoregressive encoder-decoder layers. During the generation of predicted speech information, predicted speech information is generated by fusing autoregressive and non-autoregressive prediction information. When generating autoregressive prediction information, the set of adjacent decoded character feature information and compressed decoded character feature information are referenced. The set of adjacent decoded character feature information represents local feature information, while the compressed decoded character feature information represents long-distance feature information. This ensures that the high-fidelity reconstruction capability of the original encoder-decoder is preserved when generating the final predicted speech information, reducing redundant information and avoiding the loss of key acoustic details, thus maintaining the naturalness and fidelity of the speech.

[0160] Secondly, the method provided in this embodiment reduces the length of the input sequence and the computational load by using the set of adjacent decoded character feature information. Simultaneously, compressing long-distance feature information to obtain compressed decoded character feature information effectively removes redundant information, rather than relying solely on complex attention mechanisms to capture long-distance dependencies, thus effectively avoiding performance degradation caused by processing redundant information.

[0161] Furthermore, this method does not rely on specific speech data features and task scenarios for preprocessing and feature selection. Instead, it uses a unified processing approach based on the short-range dependency characteristics of speech feature sequences. This makes it applicable to different neural audio codecs and language models, avoiding the need to redesign processing methods due to scenario changes and preventing the deletion of useful information, thus ensuring the integrity of speech generation. This scheme focuses on the speech token sequence itself, without relying on additional multimodal data, avoiding the complex process of multimodal data acquisition, fusion, and information alignment, thereby reducing system cost and complexity.

[0162] Figure 5 shows a flowchart of a speech synthesis method according to another embodiment of this disclosure. The method is applied to a cloud-side device and specifically includes the following steps.

[0163] Step 502: The receiving end sends the text information to be processed and the reference voice information.

[0164] Step 504: Input the text information to be processed and the reference speech information into the target speech synthesis model to obtain the predicted speech information output by the target speech synthesis model, wherein the target speech synthesis model is trained by the above-mentioned speech synthesis model training method.

[0165] Step 506: Send the predicted speech information to the end-side device.

[0166] In this embodiment, a speech synthesis method deployed on a cloud-side device is provided. Users can send the text to be processed and reference speech information to the cloud-side device through the end-side device. The cloud-side device processes the text to be processed and the reference speech information based on the target speech synthesis model to generate predicted speech information, and then sends the predicted speech information to the end-side device.

[0167] With this solution, users do not need to deploy the target speech synthesis model on the edge device. By leveraging the speech synthesis capabilities and powerful computing resources provided by the cloud-side device, speech synthesis tasks can be completed more quickly, avoiding excessive costs for users on the edge device.

[0168] Figure 6 shows a schematic diagram of the architecture of a task platform provided in one embodiment of the present disclosure. The task platform includes a request interface 602 and a response unit 604.

[0169] The request interface 602 is used to receive text information to be processed and reference voice information sent by the end device;

[0170] The response unit 604 is used to input the text information to be processed and the reference speech information into the target speech synthesis model to obtain the predicted speech information output by the target speech synthesis model, wherein the target speech synthesis model is trained by the above-mentioned speech synthesis model training method.

[0171] Corresponding to the above method embodiments, this disclosure also provides an embodiment of a model training device for a speech synthesis model. Figure 7 shows a schematic diagram of the structure of a model training device for a speech synthesis model provided in one embodiment of this disclosure. As shown in Figure 7, the device includes:

[0172] The acquisition module 702 is configured to acquire speech synthesis training samples, wherein the speech synthesis training samples include sample text information, reference speech information and labeled speech information;

[0173] The prediction module 704 is configured to input the sample text information and the reference speech information into an initial speech synthesis model to obtain the predicted speech information output by the initial speech synthesis model. The predicted speech information is generated from autoregressive prediction information and non-autoregressive prediction information. The current decoded character in the autoregressive prediction information is generated from the set of adjacent decoded character feature information and compressed decoded character feature information corresponding to the sample text information and the reference speech information. The non-autoregressive prediction information is generated from the sample text information and the reference speech information. The set of adjacent decoded character feature information is the set of already decoded character feature information adjacent to the current decoded character. The compressed decoded character feature information is the compressed feature information of already decoded character feature information that is more than a preset threshold away from the current decoded character.

[0174] The calculation module 706 is configured to calculate the model loss value based on the predicted speech information and the labeled speech information;

[0175] Training module 708 is configured to adjust the model parameters of the initial speech synthesis model based on the model loss value to obtain the target speech synthesis model.

[0176] Optionally, the initial speech synthesis model includes an embedding layer, an autoregressive encoder-decoder layer, a non-autoregressive encoder-decoder layer, and a speech decoding layer;

[0177] The prediction module 704 is further configured as follows:

[0178] The sample text information and the reference speech information are input into the embedding layer to obtain the spliced ​​sample feature information;

[0179] The spliced ​​sample feature information is input into the autoregressive encoder-decoder layer to obtain autoregressive prediction information;

[0180] The spliced ​​sample feature information is input into the non-autoregressive encoding / decoding layer to obtain non-autoregressive prediction information;

[0181] The autoregressive prediction information and the non-autoregressive prediction information are input into the speech decoding layer to obtain predicted speech information.

[0182] Optionally, the embedding layer includes a text embedding unit, a speech embedding unit, and a fusion unit;

[0183] The prediction module 704 is further configured as follows:

[0184] The sample text information is input into the text embedding unit to obtain text feature information;

[0185] The reference speech information is input into the speech embedding unit to obtain speech feature information;

[0186] The text feature information and the speech feature information are input into the fusion unit to generate spliced ​​sample feature information.

[0187] Optionally, the autoregressive codec layer includes an encoder and an autoregressive decoder;

[0188] The prediction module 704 is further configured as follows:

[0189] The spliced ​​sample feature information is input into the encoder to obtain autoregressive coding features;

[0190] The autoregressive encoded features are input into the autoregressive decoder to obtain autoregressive prediction information.

[0191] The prediction module 704 is further configured as follows:

[0192] The autoregressive encoded features are input into the autoregressive decoder, and decoded character feature information is output sequentially. The current decoded character feature information is generated based on the previously output decoded character feature information, which includes a set of adjacent decoded character feature information and compressed decoded character feature information.

[0193] Optionally, the prediction module 704 is further configured to:

[0194] Determine the character position information of the currently decoded character feature information;

[0195] When the character position information is less than or equal to the preset character length information, the current decoded character feature information is generated based on the set of adjacent decoded character feature information.

[0196] If the character position information is greater than the preset character length information, the current decoded character feature information is generated based on the set of adjacent decoded character feature information and the compressed decoded character feature information.

[0197] Optionally, the prediction module 704 is further configured to:

[0198] At least one group of decoded character features is determined based on the output decoded character feature information and the preset compressed character threshold.

[0199] The decoded character feature information in each decoded character feature group is compressed and fused to obtain the compressed decoded character feature information corresponding to each decoded character feature group.

[0200] Optionally, the prediction module 704 is further configured to:

[0201] Determine the current decoded character feature information corresponding to the autoregressive prediction information;

[0202] Based on the current decoded character feature information, a reference decoded character feature information is determined from the non-autoregressive prediction information;

[0203] The current decoded character feature information and the reference decoded character feature information are input into the speech decoding layer to obtain the predicted character speech information corresponding to the current decoded character feature information;

[0204] Predicted speech information is generated based on the predicted character speech information corresponding to the feature information of each decoded character.

[0205] The speech synthesis model training apparatus provided in this disclosure improves the model structure of the speech synthesis model by setting an autoregressive encoder-decoder layer and a non-autoregressive encoder-decoder layer in the speech synthesis model. During the generation of predicted speech information, predicted speech information is generated by fusing autoregressive and non-autoregressive prediction information. When generating the autoregressive prediction information, the set of adjacent decoded character feature information and the compressed decoded character feature information are referenced. The set of adjacent decoded character feature information represents local feature information, while the compressed decoded character feature information represents long-distance feature information. This ensures that the high-fidelity reconstruction capability of the original encoder-decoder is preserved when generating the final predicted speech information, reducing redundant information and avoiding the loss of key acoustic details, thus maintaining the naturalness and fidelity of the speech.

[0206] Secondly, the apparatus provided in this embodiment reduces the length of the input sequence and the computational load by using the set of adjacent decoded character feature information. Simultaneously, compressing long-distance feature information to obtain compressed decoded character feature information effectively removes redundant information, rather than relying solely on complex attention mechanisms to capture long-distance dependencies, thus effectively avoiding performance degradation caused by processing redundant information.

[0207] Furthermore, this device does not rely on specific speech data features and task scenarios for preprocessing and feature selection. Instead, it performs unified processing based on the short-range dependency characteristics of speech feature sequences, making it applicable to different neural audio codecs and language models. This avoids the need to redesign the processing device due to changes in scenarios and does not delete useful information, ensuring the integrity of speech generation. This solution focuses on the speech token sequence itself, without relying on additional multimodal data, avoiding the complex process of multimodal data acquisition, fusion, and information alignment, thus reducing system cost and complexity.

[0208] The above is an illustrative scheme of a model training device for a speech synthesis model according to this embodiment. It should be noted that the technical solution of this model training device for a speech synthesis model belongs to the same concept as the technical solution of the model training method for a speech synthesis model described above. For details not described in detail in the technical solution of the model training device for a speech synthesis model, please refer to the description of the technical solution of the model training method for a speech synthesis model described above.

[0209] Referring to Figure 8, Figure 8 shows an architecture diagram of a model training system for a speech synthesis model provided in an embodiment of the present disclosure. The model training system for the speech synthesis model may include a client 100 and a server 200.

[0210] Client 100 is used to send speech synthesis training samples to server 200, wherein the speech synthesis training samples include sample text information, reference speech information and labeled speech information;

[0211] Server 200 is used to input the sample text information and the reference speech information into an initial speech synthesis model to obtain predicted speech information output by the initial speech synthesis model. The predicted speech information is generated from autoregressive prediction information and non-autoregressive prediction information. The current decoded character in the autoregressive prediction information is generated from the set of adjacent decoded character feature information and compressed decoded character feature information corresponding to the sample text information and the reference speech information. The non-autoregressive prediction information is generated from the sample text information and the reference speech information. The set of adjacent decoded character feature information is the set of feature information of already decoded characters adjacent to the current decoded character. The compressed decoded character feature information is the compressed feature information of already decoded character feature information that is more than a preset threshold away from the current decoded character. The server calculates a model loss value based on the predicted speech information and the labeled speech information, and adjusts the model parameters of the initial speech synthesis model based on the model loss value to obtain a target speech synthesis model. The server then obtains the model parameters of the target speech synthesis model and sends the model parameters to client 100.

[0212] Client 100 is also used to receive model parameters sent by server 200.

[0213] The speech synthesis model training system can include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and server 200 can be referred to as cloud devices. Multiple clients 100 can establish communication connections through server 200. In the speech synthesis model training scenario, server 200 is used to provide training services for the speech synthesis model among multiple clients 100. Each client 100 can act as a sender or receiver, communicating through server 200.

[0214] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the training scenario of speech synthesis model, users can publish data streams to server 200 through client 100, and server 200 can generate model parameters of speech synthesis model based on the data stream and push the model parameters to other clients that have established communication.

[0215] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0216] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on a computing device and depends on the device or certain apps on the device to run. The computing device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured on the computing device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0217] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0218] It is worth noting that the model training method for the speech synthesis model provided in this disclosure is generally executed by the server. However, in other embodiments of this disclosure, the client may also have similar functions to the server, thereby executing the model training method for the speech synthesis model provided in this disclosure. In other embodiments, the model training method for the speech synthesis model provided in this disclosure may also be jointly executed by the client and the server.

[0219] Figure 9 shows a structural block diagram of a computing device 900 according to an embodiment of the present disclosure. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0220] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0221] In one embodiment of this disclosure, the aforementioned components of the computing device 900, as well as other components not shown in FIG. 9, may also be connected to each other, for example, via a bus. It should be understood that the computing device structural block diagram shown in FIG. 9 is merely for illustrative purposes and is not intended to limit the scope of this disclosure. Those skilled in the art can add or replace other components as needed.

[0222] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0223] The processor 920 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned speech synthesis model training method and speech synthesis method.

[0224] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the model training method of the above-mentioned speech synthesis model and the technical solution of the speech synthesis method. For details not described in detail in the technical solution of the computing device, please refer to the description of the model training method of the above-mentioned speech synthesis model and the technical solution of the speech synthesis method.

[0225] An embodiment of this disclosure also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described speech synthesis model training method and speech synthesis method.

[0226] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are relatively simple in description because they are substantially similar to the model training method and speech synthesis method embodiments of the speech synthesis model. Relevant parts can be referred to the descriptions of the model training method and speech synthesis method embodiments of the speech synthesis model.

[0227] An embodiment of this disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described speech synthesis model training method and speech synthesis method.

[0228] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the above-mentioned speech synthesis model training method and speech synthesis method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the above-mentioned speech synthesis model training method and speech synthesis method.

[0229] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0230] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0231] It should be noted that the above description describes specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this disclosure.

[0232] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0233] The preferred embodiments disclosed above are merely illustrative of this disclosure. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments of this disclosure. These embodiments are selected and specifically described in this disclosure to better explain the principles and practical applications of the embodiments of this disclosure, thereby enabling those skilled in the art to better understand and utilize this disclosure. This disclosure is limited only by the claims and their full scope and equivalents.

Claims

A method for training a speech synthesis model, comprising: Obtain speech synthesis training samples, wherein the speech synthesis training samples include sample text information, reference speech information, and labeled speech information; The sample text information and the reference speech information are input into the initial speech synthesis model to obtain the predicted speech information output by the initial speech synthesis model. The predicted speech information is generated by autoregressive prediction information and non-autoregressive prediction information. The current decoded character in the autoregressive prediction information is generated by the set of adjacent decoded character feature information and compressed decoded character feature information corresponding to the sample text information and the reference speech information. The non-autoregressive prediction information is generated by the sample text information and the reference speech information. The set of adjacent decoded character feature information is the set of already decoded character feature information adjacent to the current decoded character. The compressed decoded character feature information is the compressed feature information of already decoded character feature information that is more than a preset threshold away from the current decoded character. Calculate the model loss value based on the predicted speech information and the labeled speech information; The model parameters of the initial speech synthesis model are adjusted based on the model loss value to obtain the target speech synthesis model. The method as described in claim 1, wherein the reference speech information is baseline speech data used to define the timbre attributes of the target synthesized speech, and the tagged speech information is standard speech data obtained by reading the sample text information using the timbre of the reference speech information. As described in claim 1 or 2, the adjacent decoded character feature information set is a local feature set composed of decoded character features of a preset length adjacent to the position of the current decoded character, and the compressed decoded character feature information is the core feature after compression and fusion of long-distance decoded character features that are more than a preset threshold away from the current decoded character. The method according to any one of claims 1-3, wherein the initial speech synthesis model comprises an embedding layer, an autoregressive encoder-decoder layer, a non-autoregressive encoder-decoder layer, and a speech decoding layer; The sample text information and the reference speech information are input into the initial speech synthesis model to obtain the predicted speech information output by the initial speech synthesis model, including: The sample text information and the reference speech information are input into the embedding layer to obtain the spliced ​​sample feature information; The spliced ​​sample feature information is input into the autoregressive encoder-decoder layer to obtain autoregressive prediction information; The spliced ​​sample feature information is input into the non-autoregressive encoding / decoding layer to obtain non-autoregressive prediction information; The autoregressive prediction information and the non-autoregressive prediction information are input into the speech decoding layer to obtain predicted speech information. The method as described in claim 4, wherein the spliced ​​sample feature information is joint feature data obtained by fusing the text features corresponding to the sample text information and the speech features corresponding to the reference speech information. The method as described in claim 4 or 5, wherein the embedding layer comprises a text embedding unit, a speech embedding unit, and a fusion unit; The sample text information and the reference speech information are input into the embedding layer to obtain concatenated sample feature information, including: The sample text information is input into the text embedding unit to obtain text feature information; The reference speech information is input into the speech embedding unit to obtain speech feature information; The text feature information and the speech feature information are input into the fusion unit to generate spliced ​​sample feature information. The method as described in claim 6, wherein the text embedding unit is a network unit that converts discrete text information into a fixed-dimensional continuous vector and captures text semantic features, and the speech embedding unit is a network unit that converts discrete speech signals into a fixed-dimensional continuous vector and captures speech acoustic features. The method according to any one of claims 4-7, wherein the autoregressive encoder-decoder layer comprises an encoder and an autoregressive decoder; The spliced ​​sample feature information is input into the autoregressive encoder-decoder layer to obtain autoregressive prediction information, including: The spliced ​​sample feature information is input into the encoder to obtain autoregressive coding features; The autoregressive encoded features are input into the autoregressive decoder to obtain autoregressive prediction information. The method of claim 8, wherein the encoder is used to encode the input spliced ​​sample features into encoded features containing global context information. The method as described in claim 8 or 9, wherein the autoregressive encoded features are input into the autoregressive decoder to obtain autoregressive prediction information, includes: The autoregressive encoded features are input into the autoregressive decoder, and decoded character feature information is output sequentially. The current decoded character feature information is generated based on the previously output decoded character feature information, which includes a set of adjacent decoded character feature information and compressed decoded character feature information. The method as described in claim 10, wherein the current decoded character feature information is generated through the following steps: Determine the character position information of the currently decoded character feature information; When the character position information is less than or equal to the preset character length information, the current decoded character feature information is generated based on the set of adjacent decoded character feature information. If the character position information is greater than the preset character length information, the current decoded character feature information is generated based on the set of adjacent decoded character feature information and the compressed decoded character feature information. The method of claim 10 or 11 further includes: At least one group of decoded character features is determined based on the output decoded character feature information and the preset compressed character threshold. The decoded character feature information in each decoded character feature group is compressed and fused to obtain the compressed decoded character feature information corresponding to each decoded character feature group. The method according to any one of claims 4-12, wherein the autoregressive prediction information and the non-autoregressive prediction information are input into the speech decoding layer to obtain predicted speech information, includes: Determine the current decoded character feature information corresponding to the autoregressive prediction information; Based on the current decoded character feature information, a reference decoded character feature information is determined from the non-autoregressive prediction information; The current decoded character feature information and the reference decoded character feature information are input into the speech decoding layer to obtain the predicted character speech information corresponding to the current decoded character feature information; Predicted speech information is generated based on the predicted character speech information corresponding to the feature information of each decoded character. A speech synthesis method, comprising: Obtain the text information to be processed and the reference speech information; The text information to be processed and the reference speech information are input into the target speech synthesis model to obtain the predicted speech information output by the target speech synthesis model, wherein the target speech synthesis model is trained by the training method described in any one of claims 1-13. A method for training a speech synthesis model, applied to cloud-based devices, includes: The speech synthesis training samples sent by the receiving end device include sample text information, reference speech information, and labeled speech information. The sample text information and the reference speech information are input into the initial speech synthesis model to obtain the predicted speech information output by the initial speech synthesis model. The predicted speech information is generated by autoregressive prediction information and non-autoregressive prediction information. The current decoded character in the autoregressive prediction information is generated by the set of adjacent decoded character feature information and compressed decoded character feature information corresponding to the sample text information and the reference speech information. The non-autoregressive prediction information is generated by the sample text information and the reference speech information. The set of adjacent decoded character feature information is the set of already decoded character feature information adjacent to the current decoded character. The compressed decoded character feature information is the compressed feature information of already decoded character feature information that is more than a preset threshold away from the current decoded character. The model loss value is calculated based on the predicted speech information and the labeled speech information, and the model parameters of the initial speech synthesis model are adjusted based on the model loss value to obtain the target speech synthesis model. Obtain the model parameters of the target speech synthesis model and send the model parameters to the end device. A speech synthesis method, applied to cloud-side devices, includes: The receiving end sends the text information to be processed and the reference voice information; The text information to be processed and the reference speech information are input into the target speech synthesis model to obtain the predicted speech information output by the target speech synthesis model, wherein the target speech synthesis model is trained by the training method described in any one of claims 1-13; The predicted speech information is sent to the end-side device. A task platform, comprising a request interface and a response unit; The request interface is used to receive text information to be processed and reference voice information sent by the end device; The response unit is configured to input the text information to be processed and the reference speech information into the target speech synthesis model to obtain the predicted speech information output by the target speech synthesis model, wherein the target speech synthesis model is trained by the training method described in any one of claims 1-13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-16. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-16. A computer program product comprising a computer program / instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1-16.