Speech enhancement method and apparatus, electronic device, and storage medium

Abstract

A speech enhancement method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring a speech signal to be processed, performing feature extraction on the speech signal to be processed to obtain an initial signal feature, and inputting the initial signal feature to a speech enhancement network, wherein the speech enhancement network comprises an encoder and a decoder that are connected to each other, and both the encoder and the decoder are provided with a plurality of processing blocks that are sequentially cascaded; on the basis of a first processing block in the encoder, performing frame hybrid mapping on the initial signal feature to obtain a first mapped feature, performing band hybrid mapping on the first mapped feature to obtain a second mapped feature, performing band compression on the second mapped feature to obtain an output of the first processing block, and inputting the output of the first processing block to a next processing block until an encoding result of the encoder is obtained; decoding the encoding result of the encoder on the basis of the decoder to obtain a target signal feature; and on the basis of the target signal feature, restoring an enhanced speech signal from the speech signal to be processed. The computational complexity of speech enhancement can be reduced, and real-time performance of speech enhancement can be improved.

Description

Speech enhancement methods, devices, electronic devices and storage media

[0001] This application claims priority to Chinese Patent Application No. 202510053357.4, filed on January 14, 2025, entitled “Speech Enhancement Method, Apparatus, Electronic Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of speech processing technology, and in particular to a speech enhancement technology. Background Technology

[0003] Acoustic echoes and environmental noise are common in real-time communication systems, which degrades the voice quality during communication. With the development of artificial intelligence technology, voice enhancement can be achieved through the architecture of deep neural networks. However, the computational complexity of this method is often very high, and the real-time performance of voice enhancement needs to be improved. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail in this application. This overview is not intended to limit the scope of the claims.

[0005] This application provides a speech enhancement method, apparatus, electronic device, and storage medium, which can reduce the computational complexity of speech enhancement and improve the real-time performance of speech enhancement.

[0006] On one hand, embodiments of this application provide a speech enhancement method, which is executed by an electronic device and includes:

[0007] The speech signal to be processed is acquired, and features are extracted from the speech signal to obtain initial signal features. The initial signal features are then input into a speech enhancement network, wherein the speech enhancement network includes an encoder and a decoder connected to each other, and each encoder and decoder is provided with multiple processing blocks cascaded in sequence.

[0008] Based on the first processing block in the encoder, the initial signal features are subjected to frame mixing mapping to obtain a first mapping feature, the first mapping feature is subjected to band mixing mapping to obtain a second mapping feature, the second mapping feature is subjected to band compression to obtain the output of the first processing block, and the output of the first processing block is input to the next processing block until the encoding result of the encoder is obtained;

[0009] The encoder's encoding result is decoded based on the decoder to obtain the target signal features;

[0010] An enhanced speech signal is obtained by restoring the speech signal to be processed from the target signal features.

[0011] On the other hand, embodiments of this application also provide a voice enhancement device, which is deployed on an electronic device and includes:

[0012] An input module is used to acquire a speech signal to be processed, extract features from the speech signal to be processed to obtain initial signal features, and input the initial signal features into a speech enhancement network. The speech enhancement network includes an encoder and a decoder connected to each other, and each encoder and decoder is provided with multiple processing blocks cascaded in sequence.

[0013] The encoding module is configured to perform frame mixing mapping on the initial signal features based on the first processing block in the encoder to obtain a first mapping feature, perform band mixing mapping on the first mapping feature to obtain a second mapping feature, perform band compression on the second mapping feature to obtain the output of the first processing block, and input the output of the first processing block to the next processing block until the encoding result of the encoder is obtained;

[0014] A decoding module is used to decode the encoding result of the encoder based on the decoder to obtain the target signal features;

[0015] The restoration module is used to restore the enhanced speech signal from the speech signal to be processed based on the target signal features.

[0016] On the other hand, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described speech enhancement method.

[0017] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a computer program, which is executed by a processor to implement the above-described speech enhancement method.

[0018] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the above-described speech enhancement method.

[0019] The embodiments of this application include at least the following beneficial effects: by acquiring the speech signal to be processed, performing feature extraction on the speech signal to be processed to obtain initial signal features, inputting the initial signal features into the speech enhancement network, performing frame mixing mapping on the initial signal features based on the first processing block in the encoder of the speech enhancement network to obtain first mapping features, performing band mixing mapping on the first mapping features to obtain second mapping features, performing band compression on the second mapping features to obtain the output of the first processing block, and inputting the output of the first processing block into the next processing block until the encoding result of the encoder is obtained. After performing frame mixing mapping and band mixing mapping, the computational complexity can be reduced by band compression, so that the subsequent decoding of the encoder's encoding result based on the decoder can obtain the target signal features. When the enhanced speech signal is restored from the speech signal to be processed based on the target signal features, the real-time performance of speech enhancement can be improved due to the reduced computational complexity.

[0020] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. Attached Figure Description

[0021] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0022] Figure 1 is a schematic diagram of an optional implementation environment provided in an embodiment of this application;

[0023] Figure 2 is an optional flowchart of the speech enhancement method provided in an embodiment of this application;

[0024] Figure 3 is a schematic diagram of an optional structure of the processing block in the speech enhancement network provided in an embodiment of this application;

[0025] Figure 4 is an optional schematic diagram of the frame mixing module in Figure 3 provided in an embodiment of this application;

[0026] Figure 5 is a schematic diagram of an optional log-variance estimation module provided in an embodiment of this application;

[0027] Figure 6 is a schematic diagram of an optional structure with a hybrid module in Figure 3 provided in an embodiment of this application;

[0028] Figure 7 is a schematic diagram of an optional sub-band compression / decompression module in Figure 3 provided in an embodiment of this application;

[0029] Figure 8 is a schematic diagram of an optional band combining method provided in an embodiment of this application;

[0030] Figure 9 is a schematic diagram of another optional structure of the processing block in the speech enhancement network provided in the embodiment of this application;

[0031] Figure 10 is a schematic diagram of an optional frequency band segmentation method provided in an embodiment of this application;

[0032] Figure 11 is an optional schematic diagram of frequency band segmentation of signal sub-features in Figure 10 provided in an embodiment of this application;

[0033] Figure 12 is a schematic diagram of another optional frequency band segmentation provided in an embodiment of this application;

[0034] Figure 13 is a schematic diagram of an optional overall process of the speech enhancement method provided in an embodiment of this application;

[0035] Figure 14 is a schematic diagram of an optional structure of the speech enhancement model in Figure 13 provided in an embodiment of this application;

[0036] Figure 15 is a schematic diagram of another optional structure of the speech enhancement model in Figure 13 provided in an embodiment of this application;

[0037] Figure 16 is a schematic diagram of the speech enhancement device provided in an embodiment of this application;

[0038] Figure 17 is a partial structural block diagram of the terminal provided in an embodiment of this application;

[0039] Figure 18 is a partial structural block diagram of the server provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] It should be noted that in various specific embodiments of this application, when processing data related to the characteristics of the target object, such as target object attribute information or attribute information sets, is required, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. The target object can be a user. In addition, when embodiments of this application need to obtain target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0042] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0043] To facilitate understanding of the technical solutions provided in the embodiments of this application, some key terms used in the embodiments of this application will be explained below:

[0044] Frequency bin: Used to describe the signal in the frequency domain, it can be understood as the interval or resolution between two adjacent points in the spectrum.

[0045] A Multilayer Perceptron (MLP) is a deep learning model based on a feedforward neural network. It consists of a multi-layered structure with multiple neurons, where each neuron layer is connected to all neurons in the previous layer. The MLP receives external data through its input layer, extracts features through a series of hidden layers, and outputs the results through its output layer.

[0046] Acoustic Echo Cancellation (AEC): A key signal processing technique that uses adaptive filters to simulate echo paths and algorithms to estimate and eliminate echoes. It is mainly used to eliminate echoes generated during voice communication when the speaker sound is picked up by the microphone.

[0047] In real-time communication systems, acoustic echo and environmental noise are two common problems that degrade speech quality. With the development of artificial intelligence, especially advancements in deep learning, speech enhancement typically utilizes deep neural network architectures. Deep neural networks possess powerful non-linear modeling capabilities, enabling them to learn complex speech features and thus preserve useful information from the original speech signal as much as possible while eliminating echoes and suppressing noise. However, due to the complexity of deep neural network structures, the large number of parameters and layers, this approach often results in high computational complexity. This high computational complexity not only increases resource consumption but can also lead to processing delays, thereby affecting the real-time performance of speech enhancement.

[0048] Based on this, embodiments of this application provide a speech enhancement method, apparatus, electronic device, and storage medium, which can reduce the computational complexity of speech enhancement and improve the real-time performance of speech enhancement.

[0049] Referring to Figure 1, which is a schematic diagram of an optional implementation environment provided by an embodiment of this application, the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected through a communication network.

[0050] For example, a speech signal to be processed is acquired in terminal 101 and sent to server 102. In server 102, feature extraction is performed on the speech signal to obtain initial signal features. These initial signal features are then input into a speech enhancement network. The speech enhancement network includes interconnected encoders and decoders, each with multiple cascaded processing blocks. In the first processing block of the encoder, frame mixing mapping is performed on the initial signal features to obtain first mapped features. Band mixing mapping is performed on the first mapped features to obtain second mapped features. Band compression is performed on the second mapped features to obtain the output of the first processing block. The output of the first processing block is input to the next processing block until the encoder's encoding result is obtained. The decoder decodes the encoder's encoding result to obtain target signal features. Based on the target signal features, the enhanced speech signal is restored from the speech signal to be processed and sent to terminal 101, completing the speech enhancement process.

[0051] It is understood that the voice enhancement method provided in this application embodiment can also be executed independently in terminal 101.

[0052] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Additionally, server 102 can also be a node server in a blockchain network.

[0053] Terminal 101 can be a mobile phone, computer, smart voice interaction device, smart wearable device, smart home appliance, vehicle terminal, etc., but is not limited to these. Terminal 101 and server can be directly or indirectly connected through wired or wireless communication, and this application embodiment does not impose any limitations.

[0054] Referring to Figure 2, which is an optional flowchart of a voice enhancement method provided in an embodiment of this application, the voice enhancement method can be executed by an electronic device, which can be a terminal, a server, or both. The voice enhancement method includes, but is not limited to, the following S201 to S204.

[0055] S201: Acquire the speech signal to be processed, extract features from the speech signal to be processed to obtain initial signal features, and input the initial signal features into the speech enhancement network.

[0056] The speech signal to be processed can include a mixed speech signal and a reference speech signal. The mixed speech signal can be a near-end speech signal containing interference such as the target speech signal and echo signals, or a speech signal captured by a sound capture device such as a microphone. The target speech signal can be the main speech signal in the mixed speech signal. In one possible implementation, the target speech signal can be the speech signal emitted by the current speaker captured by the microphone. For example, if speaker A is talking to speaker B, speaker A's speech signal is the target speech signal for speaker A's microphone. The reference speech signal can be a speech signal used as a reference for echo prediction. In one possible implementation, the reference speech signal can be a far-end speech signal. The far-end speech signal can be the speech signal sent by the other end of the call. For example, continuing the above example, the speech signal transmitted by speaker B to speaker A is the far-end speech signal. The echo signal can be a signal captured by the microphone along with the target speech signal due to the echo generated when the speaker plays the reference speech signal, and is obtained based on echo prediction. Error speech signals can be used to indicate the difference between mixed speech signals and echo signals. They are signals obtained based on the mixed speech signals and echo signals, for example, signals obtained by removing echo signals from the mixed speech signals. Initial signal features can be signal features obtained after feature extraction from the speech signal to be processed. The speech enhancement network includes interconnected encoders and decoders. The speech enhancement network can be a recurrent UNet network. Both the encoder and decoder have multiple cascaded processing blocks. The encoder encodes the initial signal features or other features obtained based on the initial signal features, and the decoder decodes the encoded initial signal features or other encoded features obtained based on the initial signal features. The encoder and decoder are symmetrically constructed; for example, the encoder has three cascaded processing blocks, and correspondingly, the decoder also has three cascaded processing blocks.

[0057] Specifically, a mixed speech signal and a reference speech signal corresponding to the target speech signal in the mixed speech signal are acquired from a microphone. The mixed speech signal and the reference speech signal are then input into an echo prediction model for echo prediction. By analyzing the frequency, amplitude, and phase characteristics of the mixed speech signal and the reference speech signal, the echo components in the mixed speech signal are identified, resulting in the predicted echo signal. Next, the echo signal and the mixed speech signal are input into an echo cancellation model. Based on the difference between the mixed speech signal and the echo signal, an error speech signal is obtained. Echo cancellation is then performed based on the mixed speech signal and the reference speech signal, yielding both the echo signal and the error speech signal. This process produces a high-quality speech signal, providing valuable data for subsequent noise suppression.

[0058] Next, time-frequency domain transformations are performed on the mixed speech signal, reference speech signal, error speech signal, and echo signal. These transformations convert the mixed speech signal, reference speech signal, error speech signal, and echo signal from the time domain to the frequency domain; the time-frequency domain transformation can be a short-time Fourier transform. Converting these four signals from the time domain to the frequency domain yields the mixed speech spectrum, the reference speech spectrum, the error speech spectrum, and the echo spectrum. During the time-frequency domain transformation, the mixed speech signal, reference speech signal, error speech signal, and echo signal are represented in complex form in the frequency domain; therefore, the mixed speech spectrum, reference speech spectrum, error speech spectrum, and echo spectrum all have corresponding real and imaginary parts. The real and imaginary parts of the mixed speech spectrum, reference speech spectrum, error speech spectrum, and echo spectrum are concatenated along the channel dimension to obtain the speech signal to be processed. The feature dimension of the speech signal to be processed can be... B represents the batch size, T represents the number of frames, and F represents the number of frequency units. Feature extraction is performed on the speech signal to be processed, extracting the real part and imaginary part features of the complex spectrum from the speech signal to be processed. The real part and imaginary part features of the complex spectrum are used as initial signal features, and the initial signal features are input into the speech enhancement network for speech enhancement.

[0059] Alternatively, the feature dimensions of the speech signal to be processed can also be... C represents the number of channels. In one possible implementation, the number of channels can be 8. During feature extraction of the speech signal to be processed, the speech signal can also be input into a two-dimensional convolution for feature extraction. The obtained features are then normalized and passed through an activation function to obtain initial signal features. The feature dimension of the initial signal features can be R. E×T×FE represents the embedding dimension. The initial signal features are input into the speech enhancement network for speech enhancement. Converting the channel dimension to the embedding dimension through two-dimensional convolution can be understood as a dimensionality reduction process. Subsequent steps, based on the initial signal features, can remove redundant or unimportant features, making subsequent processing steps simpler and reducing computational load and complexity, effectively improving the real-time performance of the enhanced speech signal.

[0060] S202: Based on the first processing block in the encoder, the initial signal features are frame-mixed and mapped to obtain the first mapping feature, the first mapping feature is band-mixed and mapped to obtain the second mapping feature, the second mapping feature is band-compressed to obtain the output of the first processing block, and the output of the first processing block is input to the next processing block until the encoder's encoding result is obtained.

[0061] The encoder's processing block performs frame mixing mapping, band mixing mapping, and subband compression operations on the input features. The decoder's processing block performs frame mixing mapping, band mixing mapping, and subband decompression operations on the input features. The processing block includes a frame mixing module, a band mixing module, and a subband compression / decompression module. Frame mixing mapping can model in the time dimension, capturing the dynamic changes of the speech signal in time by combining the feature information of adjacent frames, thus enhancing the temporal continuity of the features. Band mixing mapping can model in the subband dimension, capturing the local characteristics of the speech signal in the frequency domain by fusing the feature information of different subbands, thus enhancing the frequency band resolution of the features. The first mapped feature can be the feature obtained after the initial signal features are processed by frame mixing mapping, and the second mapped feature can be the feature obtained after the first mapped feature is processed by band mixing mapping. The second mapped feature can be regarded as an interband relationship feature used to indicate interband relationships.

[0062] In one possible implementation, the band-mixing mapping method can be to add the first mapping feature to the initial signal feature to obtain the first fused feature, and then perform band-mixing mapping based on the first fused feature to obtain the second mapping feature. The method of band-compressing the second mapping feature to obtain the output of the first processing block can be to add the second mapping feature to the first fused feature to obtain the second fused feature, and then perform band-compressing on the second fused feature to obtain the output of the first processing block.

[0063] Referring to Figure 3, which is a schematic diagram of an optional structure of a processing block in a speech enhancement network provided in this application embodiment, the input of the first processing block of the encoder of the speech enhancement network is the initial signal feature X. The initial signal feature X is input to the frame mixing module for frame mixing mapping to obtain a first mapping feature F1. The first mapping feature F1 is added to the initial signal feature X to obtain a first fusion feature f1. The first fusion feature f1 is input to the band mixing module for band mixing mapping to obtain a second mapping feature F2. The second mapping feature F2 is added to the first fusion feature f1 to obtain a second fusion feature f2. Finally, the second fusion feature f2 is input to the sub-band compression / decompression module for band compression to obtain the output F of the processing block. o .

[0064] In one possible implementation, the processing block includes a gated loop unit. In the process of obtaining the first mapped feature by performing frame mixing mapping on the initial signal features, the initial signal features may be normalized, and the normalized initial signal features may be subjected to frame mixing mapping by the gated loop unit to obtain the output of the gated loop unit. Then, the output of the gated loop unit is subjected to fully connected processing to obtain the first mapped feature.

[0065] Specifically, the initial signal features are normalized to ensure they have a uniform data distribution. The normalized initial signal features are then input into a gated recurrent unit (GRU) for feature mapping along the time dimension. The GRU extracts sequence information or captures time-dependent features of the speech signal from the initial signal features; these information and features reflect the characteristics of the speech signal changing over time. Since the feature dimension of the output of the GRU may differ from that of the initial signal features, the output of the GRU can be input into a fully connected layer for further feature extraction to learn complex feature combinations. The feature dimension of the mapped features is then adjusted by changing the number of output neurons in the fully connected layer. During feature mapping, the mapped features are mapped to a new feature space whose feature dimension matches that of the initial signal features, thus obtaining a first mapped feature with the same feature dimension as the initial signal features. In this embodiment, the GRU can include various types, such as a skip-gated recurrent unit or a bidirectional gated recurrent unit.

[0066] This application embodiment employs a gated loop unit, which can filter out noise in the initial signal features while capturing the timing information and time dependence in the initial signal features, thereby improving the robustness of the encoder and helping to improve the accuracy of the decoder in predicting speech signals.

[0067] Referring to Figure 4, which is an optional schematic diagram of the frame mixing module in Figure 3 provided in the embodiment of this application, taking the gated loop unit as a skip-gated loop unit as an example, the initial signal feature X is input to the frame mixing module. The initial signal feature X is first normalized by the normalization layer to obtain the normalized initial signal feature. Then, the normalized initial signal feature is input to the skip-gated loop unit for feature mapping to obtain the mapped feature. Finally, the mapped feature is input to the fully connected layer for feature mapping again to obtain the first mapped feature F1.

[0068] In one possible implementation, the speech enhancement network is configured to iterate through multiple time steps. The gated recurrent unit (ROU) is equipped with binary gates, which indicate whether to update the hidden state features of the RRU or leave them unchanged. Specifically, during the frame mixing mapping of the normalized initial signal features through the RRU, for any given time step, the initial state update probability and its change in value in the previous time step of the RRU are obtained. The target state update probability of the RRU in the current time step is determined based on these probabilities. Then, the target state update probability is rounded to obtain the gating parameters corresponding to the binary gates in the current time step. The hidden state features output by the RRU in the current time step are then determined based on these gating parameters. Based on these hidden state features, the normalized initial signal features are frame mixed and mapped through the RRU to obtain the output of the RRU.

[0069] The initial state update probability of the current time step can be the final output state update probability after passing through the binary gate in the previous time step. For example, the final output state update probability of the previous time step is p. t-1 The initial state update probability at the current time step is p. t-1 The initial state update probability change value at the current time step can be the final state update probability change value determined after passing through the binary gate in the previous time step; the target state update probability can be the state update probability that has not passed through the binary gate at the current time step.

[0070] In one possible implementation, for any current time step among multiple time steps, the initial state update probability and its change value are obtained. The probability difference between the initial state update probability and the total probability is calculated, where the total probability can be 1. The smaller value between the initial state update probability change value and the probability difference is selected as the state update probability change value of the previous time step. This change value is added to the initial state update probability to obtain the target state update probability of the gated loop unit in the current time step. The target state update probability of the current time step t is thus obtained. The process can be represented by the following formula.

[0071] Where, p t-1 Let Δp be the state update probability after passing through the binary gate at time step t-1. t-1 Let be the change in state update probability after passing through the binary gate in time step t-1. Min(·) is the minimum value function used to find the minimum value in a set of data. It should also be noted that the target state update probability in the current time step increases by the change in state update probability in the previous time step before passing through the binary gate. It can be obtained from the following formula,

[0072] Where γ represents the proportionality coefficient, W p b represents the weighting coefficient. p This represents the proportional offset, and σ(·) represents the sigmoid function. The weighting coefficient W... p and proportional offset b p These can be parameters from a fully connected layer.

[0073] Next, the target state update probability is rounded to obtain the gating parameter g corresponding to the binary gate in the current time step. t The hidden state features output by the gated loop unit in the current time step are determined based on the gating parameters. Then, based on these hidden state features, the normalized initial signal features are frame-mixed and mapped using the gated loop unit. In one possible implementation, the target state update probability can be rounded according to a rounding rule to obtain the gating parameters corresponding to the binary gates in the current time step. The gating parameters control whether the hidden state features of the current time step include the hidden state features of the previous time step, and the values ​​of the gating parameters can be {0,1}. The hidden state features output by the gated loop unit in the current time step are determined based on the gating parameters, and the normalized initial signal features are frame-mixed and mapped using the gated loop unit based on these hidden state features. Therefore, the gating parameter g for the current time step t can be obtained by the following formula. t ,

[0074] in, Here, represents the target state update probability at the current time step t, and Round(·) is the rounding function used to round the target state update probability. Convert to integers. Also, the hidden state feature s at the current time step t. t It can be obtained from the following formula,

[0075] in, The initial hidden state features are those of time step t before passing through the binary gate. It can be derived from the formula s t-1 Let x be the hidden state feature at time step t-1 after passing through the binary gate. t The initial signal features input to the speech enhancement network at time step t. When g t When = 1, the hidden state feature s t For the gating parameter g t and initial hidden state features The product, i.e., updating the hidden state features at the current time step; when g t When = 0, the hidden state feature s t The hidden state feature s equal to time step t-1 t-1 That is, the hidden state features at the current time step are not updated, and the hidden state features s from the previous time step t-1 are used instead. t-1 .

[0076] By controlling the gating parameters corresponding to binary gates, the hidden state features output by the gating loop unit in the current time step are controlled, so that the hidden state features do not need to be updated in each time step. Especially in time steps where the hidden state features do not change much, the computational complexity in the inference process is effectively reduced, which helps to improve the real-time performance of the speech enhancement process.

[0077] In one possible implementation, after rounding the target state update probability to obtain the gating parameter for the current time step, the initial state update probability change value for the current time step can be determined based on the gating parameter and the target state update probability. Specifically, the final state update probability output after passing through the binary gate for the current time step is determined based on the gating parameter and the target state update probability; then, the final state update probability change value output after passing through the binary gate for the current time step is obtained based on the gating parameter, the initial state update probability change value, and the final state update probability output from the previous time step. The state update probability p at time step t... tThe change in state update probability Δp at time step t t It can be obtained from the following formula,

[0078] Among them, g t The gating parameter in time step t, Update the probability of the target state at time step t. Let Δp be the change in the probability of state updates that do not pass through a binary gate in time step t. t-1 This represents the change in state update probability after passing through the binary gate at time step t-1. Since the binary gate only processes parameters with two states, 1 or 0, the initial state update probability and its change at the current time step are updated through the gating parameters of the binary gate. Even if the initial input signal features are subject to a certain degree of noise interference, the gated recurrent unit can still output relatively stable state update probabilities and their changes, improving the stability of the features processed by the processing block and thus enhancing the robustness of the speech enhancement network to noise and interference.

[0079] In one possible implementation, the speech enhancement network also includes a log-variance estimation module. Before inputting the initial signal features into the speech enhancement network, the network can be trained. Specifically, this can involve acquiring sample speech signals and their corresponding sample label signals, calling the speech enhancement network to enhance the sample speech signals, determining the log-variance of the enhancement result using the log-variance estimation module, obtaining the enhancement result of the sample speech signals, and then determining the target loss based on the difference between the enhancement result and the sample label signals, as well as the log-variance. The speech enhancement network is then trained based on this target loss. The log-variance estimation module can be used to estimate the uncertainty in the enhancement result; the sample speech signals can be training speech signals, including sample mixed speech signals, sample reference speech signals, sample echo signals, and sample error speech signals; the sample label signals serve as a reference speech signal for comparison with the enhancement result of the sample speech signals; and the enhancement result of the sample speech signals includes the sample enhanced real part signal corresponding to the sample real part signal, the sample enhanced imaginary part signal corresponding to the sample imaginary part signal, and the sample enhanced speech signal corresponding to the sample speech signal.

[0080] Specifically, the process involves acquiring sample speech signals and their corresponding label signals, extracting features from the sample speech signals to obtain their real and imaginary parts, and then inputting these signals into a speech enhancement network to enhance the speech signals, resulting in enhanced real, imaginary, and speech signals. A log-variance estimation module determines the log-variance of these enhanced signals. Based on the difference between the enhanced speech signals and the label signals, and the log-variance, a target loss is determined, which guides the training of the speech enhancement network.

[0081] By introducing a log-variance estimation module to estimate the uncertainty of the enhancement result of the sample speech signal for training, the speech enhancement network can focus on the content pointed to by the uncertainty of the enhancement result of the sample speech signal, thereby guiding the training direction of the speech enhancement network and making the training of the speech enhancement network more targeted.

[0082] Referring to Figure 5, which is an optional schematic diagram of the log-variance estimation module provided in an embodiment of this application, the enhanced result F of the sample speech signal is used to... st The input is fed into the log-variance estimation module for uncertainty estimation. This module consists of three cascaded sub-modules. For the first sub-module, the enhanced result F of the sample speech signal is input... st The input is fed into a convolutional layer for feature extraction, yielding intermediate features f. These intermediate features f are then input into a nonlinear activation function to introduce nonlinearity, resulting in the output of the first submodule. This output is then fed into the second submodule, and so on, until the output of the third submodule is obtained. The output of the third submodule is the variance θ corresponding to the enhanced result of the sample speech signal. The nonlinear activation function can be an Exponential Linear Unit (ELU). Finally, the variance θ corresponding to the enhanced result of the sample speech signal is transformed using the natural logarithm to obtain the log-variance v corresponding to the enhanced result of the sample speech signal.

[0083] In one possible implementation, the process of obtaining the second mapping feature by performing a band-mixed mapping on the first mapping feature can specifically involve segmenting the first mapping feature along the channel dimension to obtain a first intra-band relational sub-feature and a second intra-band relational sub-feature, normalizing the first intra-band relational sub-feature, convolving the normalized first intra-band relational sub-feature along the time axis in the sub-band dimension to obtain a first convolutional feature, determining the first product of the first convolutional feature and the second intra-band relational sub-feature, and obtaining the second mapping feature based on the sum of the first product and the second intra-band relational sub-feature.

[0084] Specifically, when modeling inter-band relationships for the first mapping feature, firstly, channel mapping is performed on the first mapping feature. The first mapping feature after channel mapping is then segmented along the channel dimension to obtain a first intra-band relationship sub-feature and a second intra-band relationship sub-feature. The first intra-band relationship sub-feature is used to control the information flow in the first mapping feature, while the second intra-band relationship sub-feature is used to directly transmit the information in the first mapping feature. The first intra-band relationship sub-feature is then normalized. When performing frequency band mapping on the normalized first intra-band relationship sub-feature, intra-band relationship modeling is performed along the time axis, i.e., a convolution operation is performed on the sub-band dimension using a first convolution to obtain a first convolution feature. The first convolution is used for frequency band mapping, and the convolution kernel size can be 1×3, with a stride of 3. Then, the first convolution feature and the second intra-band relationship sub-feature are multiplied element-wise to obtain a first product. The sum of the first product and the second intra-band relationship sub-feature yields the second mapping feature used to indicate intra-band relationships. By dividing the first mapping feature into two sub-features, it is helpful to capture the complex relationships in the first mapping feature. This allows the speech enhancement network to gradually introduce sub-frequency band information during the feature learning process, enabling the speech enhancement network to process input data more flexibly during encoding and decoding, thereby improving the quality of the enhanced speech signal.

[0085] The embodiments of this application perform element-wise multiplication of the first convolutional features and the second intra-band relational features, which can further enhance the common features in the first convolutional features and the second intra-band relational features, ignore or reduce attention to irrelevant features or features with weak correlation, and effectively improve the ability to enhance speech signals.

[0086] In one possible implementation, the process of segmenting the first mapping feature along the channel dimension to obtain the first intra-band relational sub-feature and the second intra-band relational sub-feature can specifically involve convolving the first mapping feature along the time axis along the channel dimension to obtain the second convolutional feature, normalizing the second convolutional feature, and segmenting the normalized second convolutional feature along the channel dimension to obtain the first intra-band relational sub-feature and the second intra-band relational feature.

[0087] Specifically, channel mapping is performed on the first mapped feature along the time axis. Then, a second convolution operation is performed on the first mapped feature in the channel dimension to increase the number of channels, resulting in the second convolutional feature. The second convolution is used for channel mapping, and its kernel size can be 1×1. For example, if the feature dimension of the first mapped feature is... Then the feature dimension of the second convolutional feature can be... Then, the second convolutional features are normalized, and the normalized second convolutional features are segmented along the channel dimension to obtain first-band sub-features and second-band sub-features. For example, the feature dimension of the normalized second convolutional features can be... After segmentation along the channel dimension, the channel dimension of the sub-features in the first band can be The channel dimension of the second-band sub-feature can be Alternatively, the channel dimension of the first-band sub-feature can be The channel dimension of the second-band sub-feature can be Make E1 + E2 = 2E.

[0088] By increasing the channel dimension of the first mapping feature, the channel dimensions of the first intra-band relational sub-feature and the second intra-band relational sub-feature are made consistent with the first mapping feature, which facilitates subsequent feature fusion operations such as addition or multiplication.

[0089] Alternatively, the first mapping feature can be normalized first, and then channel mapping can be performed on the normalized first mapping feature along the time axis. Then, the normalized first mapping feature can be convolved in the channel dimension by the second convolution to increase the number of channels of the normalized first mapping feature, thus obtaining the second convolution feature.

[0090] In one possible implementation, the first product of the first convolutional feature and the second intra-band relational sub-feature is determined. When obtaining the second mapping feature based on the sum of the first product and the second intra-band relational sub-feature, specifically, the first product and the sum of the second intra-band relational sub-feature can be convolved along the time axis in the channel dimension to obtain the inter-band relational feature used to indicate the inter-band relation.

[0091] Specifically, the first product and the second intra-band relational feature of each frame are added element-wise along the time axis to obtain the fused feature. The fused feature is then channel-mapped, and a second convolution operation is performed on the channel dimension to obtain the second mapped feature.

[0092] By adding element-wise the second intra-band relational features segmented along the channel dimension to the first product obtained through element-wise multiplication, the speech signal information in both feature maps (second intra-band relational features and the first product) can be balanced. This ensures that the fused feature includes both the fused information of the first product and the original information of the second intra-band relational features, enabling the speech enhancement network to learn more feature combinations and thus enhancing its generalization ability. Furthermore, the addition operation effectively alleviates the gradient vanishing problem that may occur during training, significantly improving the training stability of the speech enhancement network.

[0093] Referring to Figure 6, which is an optional structural diagram of the hybrid module in Figure 3 provided in an embodiment of this application, the transposed intra-band relation features are shown. The input is fed into a multi-layer perceptron for inter-band modeling. First, a one-dimensional convolution is used to perform channel mapping, transposing the intra-band relation features. The number of channels is increased to twice the original value, and the in-band relationship characteristics of the increased number of channels are affected. Normalization is performed to obtain the intermediate feature Z. a The intermediate feature Z a The input is fed into the gating unit module, which is used to capture the intermediate feature Z. a Information exchange between neutron frequency bands. In the gating unit module, the intermediate feature Z is first segmented along the channel dimension using a segmentation head. a The data is segmented into a first intra-band relational sub-feature Z1 and a second intra-band relational sub-feature Z2. The first intra-band relational sub-feature Z1 is input into a normalization layer for normalization processing. The normalized first intra-band relational sub-feature Z1 is then subjected to frequency band mapping along the time axis to obtain the first convolutional feature. The first convolutional features Element-wise multiplication with the second-band in-band relational feature Z2 yields the first product Z. * Then multiply the first product Z * Adding the feature Z2 to the second-band relational feature Z yields the fused feature Z. b Finally, the features Z are fused using a one-dimensional convolution. b By performing channel mapping and changing the feature dimensions of the fused features, inter-band relationship features are obtained to indicate inter-band relationships. By modeling inter-band relationships across multiple perceptual layers along the sub-band dimension, we can better understand the interactions and influences between different sub-bands. This allows for accurate noise reduction and other processing of each sub-band during speech signal enhancement, effectively improving the quality of the enhanced speech signal.

[0094] In one possible implementation, during the band compression of the second mapping features to obtain the output of the first processing block, the second mapping features can be normalized. Parallel convolutions are then performed on the normalized second mapping features to obtain compressed third and fourth convolutional features. Nonlinear activation is applied to the fourth convolutional features to obtain attention weights. These attention weights are then used to weight the third convolutional features to obtain the output of the first processing block. Specifically, the band compression operation is performed in the sub-band compression / decompression module of the encoder's processing block, while the band decompression operation is performed in the sub-band compression / decompression module of the decoder's processing block.

[0095] Specifically, the second mapping feature is normalized, and then parallel convolutions are performed on the normalized second mapping feature to obtain compressed third and fourth convolutional features. The convolution process involves two convolutional layers: the first can be a standard convolution, and the second can be a transposed convolution. The fourth convolutional feature is input into a nonlinear activation function for nonlinear activation to obtain the attention weights corresponding to the second mapping feature. These attention weights are then weighted with the third convolutional feature to obtain the output of the first processing block. The nonlinear activation function can be the Sigmoid function. Referring to Figure 7, which is an optional schematic diagram of the subband compression / decompression module in Figure 3 provided in this embodiment, the second mapping feature F2 is normalized, and then input into the first and second convolutional layers for parallel convolutions. After passing through the first and second convolutional layers, the fourth convolutional feature f4 is obtained. The fourth convolutional feature f4 is used as the input to the nonlinear activation function to obtain the attention weights W corresponding to the second mapping feature F2. Next, the lower path passes through the first and second convolutional layers to obtain the third convolutional feature f3. This third convolutional feature f3 is then weighted based on the attention weight W, i.e., the attention weight W and the third convolutional feature f3 are multiplied element-wise to obtain the output F of the first processing block. o .

[0096] By performing band compression in the encoder, important spectral information can be preserved while reducing interference from redundant information, enabling the processing block to adaptively emphasize important frequency band features. Furthermore, normalizing the second mapping features before band compression ensures the stability of the feature distribution, thereby improving the stability of the speech signal enhanced by the speech enhancement network.

[0097] S203: Decode the encoder's encoding result based on the decoder to obtain the target signal features.

[0098] In the decoding process, the decoder's processing block is used to perform frame mixing, band mixing, and subband decompression operations; the target signal features are the features corresponding to the target speech signal in the mixed speech signal, and are used for speech enhancement.

[0099] In one possible implementation, the process of decoding the encoded result based on the decoder to obtain the target signal features can specifically involve calling the decoder to decode based on the encoder's encoded result, segmenting the decoded result along the frequency dimension to obtain multiple decoded sub-features, restoring the channel dimension of each decoded sub-feature to be consistent with the channel dimension of the speech signal to be processed, and merging the multiple restored decoded sub-features along the frequency dimension to obtain the target signal features. Here, the decoded sub-features are obtained based on sub-frequency band segmentation, and their number is the same as the number of sub-frequency bands. For example, the feature dimension of the compressed feature is... When Q is the number of sub-bands after frequency compression, the number of decoded sub-features is also Q.

[0100] Specifically, the decoder decodes based on the encoder's encoding result, and segments the decoded result along the frequency dimension. The frequency dimension represents the number of sub-bands after frequency compression. Based on the number of sub-bands, the decoded result is segmented into multiple decoded sub-features. Then, convolutional or fully connected operations are used to restore the channel dimension of each decoded sub-feature to match the channel dimension of the speech signal to be processed. The restored decoded sub-features are then merged along the frequency dimension to obtain the target signal feature. Alternatively, after obtaining multiple decoded sub-features, they can be merged along the frequency dimension, and then the channel dimension of the merged decoded feature can be restored to match the channel dimension of the speech signal to be processed, yielding the target signal feature. By restoring the channel dimension of each decoded sub-feature to match the channel dimension of the speech signal to be processed, subsequent feature fusion processes such as feature addition or feature multiplication are facilitated, ensuring the smooth progress of the feature fusion process.

[0101] Referring to Figure 8, which is an optional schematic diagram of band combining provided in an embodiment of this application, assuming that the feature dimension corresponding to the decoding result U is... The feature dimension corresponding to the speech signal to be processed is The decoding result U is segmented based on the frequency dimension Q to obtain Q decoded sub-features. Where q∈{1,...,Q}. Decode the sub-feature U q The channel dimensions are restored to match the channel dimensions of the speech signal to be processed, thus obtaining the decoded sub-features. All decoding sub-features G q Merging along the frequency dimension yields the target feature signal. The above process can be represented by the following formula: {U1,...,U Q} = SubBandSplit(U), ​​G q =MLP q (Norm(U qG = Merge(G1,...,G) Q ),

[0102] SubBandSplit(·) represents subband segmentation.

[0103] In one possible implementation, the process of restoring the channel dimensions of each decoded sub-feature to be consistent with the initial signal features can specifically involve normalizing each decoded sub-feature and using a multilayer perceptron to restore the channel dimensions of the normalized decoded sub-features to be consistent with the channel dimensions of the speech signal to be processed. Specifically, for any decoded sub-feature, it is normalized, and the channel dimensions of the normalized decoded sub-feature are restored to be consistent with the channel dimensions of the speech signal to be processed by adjusting the number of output neurons in the multilayer perceptron. The multilayer perceptron then predicts the sub-time-frequency mask corresponding to this decoded sub-feature, where the sub-time-frequency mask can be a complex-valued mask applied in the time-frequency domain. Next, the sub-time-frequency masks corresponding to each decoded sub-feature are merged along the time axis to obtain the time-frequency mask, i.e., the target signal feature. Multilayer perceptrons can learn the complex linear relationships between various decoding sub-features, and are highly adaptable to different types of noise and interference. The output sub-time-frequency mask can weight the signal in the time-frequency domain to suppress noise or eliminate some echoes, thereby preserving the enhanced speech signal (i.e., target signal features) and effectively improving the clarity and quality of the enhanced speech signal.

[0104] S204: Obtain the enhanced speech signal from the speech signal to be processed based on the characteristics of the target signal.

[0105] In the above steps, the speech signal to be processed is acquired, and features are extracted from the speech signal to obtain initial signal features. These initial signal features are then segmented into frequency bands to obtain segmentation mapping features. The segmentation mapping features are input into a speech enhancement network for noise removal and other operations. The speech enhancement network includes interconnected encoders and decoders, each with multiple cascaded processing blocks. In the encoder's processing block, frame mixing mapping is performed on the input to obtain the first mapping feature. Band mixing mapping is performed on the first mapping feature to obtain the second mapping feature. Band compression is performed on the second mapping feature to obtain the output of the processing block. The output of this processing block is then input into the next processing block until the encoder's encoding result is obtained. The encoding result is input into the decoder for decoding. Frequency band merging is performed on the decoded result to obtain the target signal features. Based on the target signal features, the enhanced speech signal is restored from the speech signal to be processed. The enhanced speech signal is the target speech signal after removing echo signals and suppressing noise and other interference from the mixed speech signal. The restoration process can be understood as performing an inverse time-frequency domain transformation on the target signal features, converting the target signal features from the frequency domain to the time domain. The inverse time-frequency domain transformation can be a short-time Fourier transform.

[0106] In one possible implementation, the process of restoring the enhanced speech signal from the speech signal to be processed based on the target signal features can specifically involve determining a second product between the target signal features and the speech signal to be processed, and filtering the second product to restore the enhanced speech signal from the speech signal to be processed. Here, the second product is the product obtained by multiplying the target speech signal and the speech signal to be processed element-wise.

[0107] Specifically, after obtaining the target signal features, some residual noise may still exist in the target signal features. To further suppress the noise, the target signal features can be post-processed. First, the target signal features are multiplied element-wise with the speech signal to be processed along the channel dimension to obtain a second product. The above element-wise multiplication process can be expressed by the following formula.

[0108] Where C represents the target signal features and the number of channels of the speech signal to be processed (the number of channels of the target signal features has been adjusted to match the number of channels of the speech signal to be processed in the above steps), i represents the i-th channel, I represents the speech signal to be processed, and G represents the target signal features. This formula means that the target signal features and the speech signal to be processed are multiplied element-wise channel by channel, and the multiplication results of each channel are added together to obtain the second product.

[0109] Then, the second product and the speech signal to be processed are input into multiple gated recurrent units to further learn complex sequence features and long-term dependencies, obtaining intermediate features with long-term dependencies. These intermediate features are then linearly transformed through a set of linear layers to restore the enhanced speech signal from the speech signal to be processed. By multiplying the target signal features element-wise with the speech signal to be processed to obtain the second product, the post-processing process can focus more on features in the speech signal to be processed that are more relevant to the target signal features. Based on this, deep filtering of the second product through gated recurrent units and linear layers can reduce the number of parameters while maintaining the processing power of the post-processing process, which helps to restore the enhanced speech signal from the speech signal to be processed, effectively improving the accuracy of speech separation and thus ensuring the quality and reliability of the enhanced speech signal.

[0110] In one possible implementation, before invoking the encoder to encode based on the initial signal features, the encoder and decoder need to be trained. Specifically, this can involve acquiring sample speech signals, reconstructing enhanced sample speech signals from the sample speech signals using the encoder and decoder, acquiring the corresponding labeled speech signals and speech activity tags, determining a first loss based on the difference between the sample speech signals and the labeled speech signals, adjusting the sample speech signals according to the speech activity tags to obtain a second loss, and then weighting and summing the first and second losses to obtain the target loss. The parameters of the encoder and decoder are then adjusted based on the target loss. Here, the sample speech signals are the speech signals used for training, including sample mixed speech signals, sample reference speech signals, sample echo signals, and sample error speech signals; the labeled speech signals serve as the baseline speech signals, used for comparison with the predicted speech signals; the speech activity tags indicate whether the sample speech signals contain or do not contain the target object's speech. When the speech activity tag is 1, it indicates that the sample semantic signal contains the target object's speech; when the speech activity tag is 0, it indicates that the sample semantic signal does not contain the target object's speech. The target object's speech is the sample mixed speech signal, which can be a near-end signal.

[0111] Specifically, sample speech signals are acquired, and encoders and decoders are trained based on these signals, enabling them to reconstruct the enhanced speech signal from the sample speech signals. During training, the real part, imaginary part, and amplitude of the labeled speech signal are first acquired. Then, the real part, imaginary part, and amplitude of the sample speech signal obtained during training are acquired. A first loss is determined based on the mean absolute error between the real part and the sample speech signal, the mean absolute error between the imaginary part and the sample speech signal, and the mean absolute error between the label amplitude and the sample amplitude. According to the above description, the first loss L... MAEIt can be expressed by the following formula,

[0112] in, Represents the real part of the sample. Indicates the imaginary part of the sample. S represents the sample amplitude. R S represents the real part of the label. I Let |S| represent the imaginary part of the label, and |S| represent the label amplitude. Calculating the loss function using the mean absolute error provides a direct way to measure the difference between the predicted spectrum and the actual label, allowing us to understand the prediction accuracy and error distribution of the encoder and decoder.

[0113] Next, the speech activity labels are obtained. The norm is calculated based on the product of the speech activity labels and the sample amplitude to obtain the first norm. Then, the logarithm of the first norm is taken to obtain the second loss. According to the above description, the second loss L... vad It can be expressed by the following formula,

[0114] in, For voice activity tags, To predict the spectrum, the target speech signal is a noise- and echo-free near-end speech signal. ε is a hyperparameter used to prevent over-suppression of the target speech signal in the mixed speech signal. When the speech activity label... When the value is 0, the calculated first norm value is relatively large, and the penalty intensity increases with the increase of the first norm value. Simultaneously, speech activity tags... A value of 0 indicates that the target object's speech is absent in the sample speech signal, which can be understood as near-end speech being missing. In this case, the second loss L... vad The value will gradually approach 0, and with the second loss L vad The value of ...

[0115] Then, weights are assigned to the second loss, and the first and second losses are summed with weights to obtain the target loss, L, which can be expressed by the following formula: L = L MAE +βL vad ,

[0116] Wherein, β is a preset weight (hyperparameter) used to enhance the ability to preserve near-end speech.

[0117] In addition, to effectively suppress echo signals, a third loss is calculated based on the frame rate, frequency, amplitude, and phase of the sample speech signal. The third loss L echo The echo suppression loss, used to evaluate echo suppression, can be expressed by the following formula.

[0118] Among them, L mag The amplitude loss can be calculated based on the difference between the sample amplitude and the label amplitude; W echo These are echo weighting coefficients, and the weights of different frequency units can be adjusted by the power ratio of the echo signal; L pha The phase loss can be calculated from the difference between the sample phase and the label phase.

[0119] After obtaining the third loss, the target loss can be obtained by weighted summation of the first, second, and third losses. At this point, the target loss L can be expressed by the following formula: L = L MAE +βL vad +0.1L echo ,

[0120] β is set to balance the ability between echo suppression and near-end speech preservation.

[0121] Finally, the parameters of the encoder and decoder are adjusted based on the target loss to enable them to effectively suppress noise and echo, outputting target signal features free of noise and echo. A clear enhanced speech signal is then reconstructed based on these target signal features. The joint loss function allows for evaluation of the encoder and decoder in terms of prediction spectrum, enhanced near-end speech preservation, and echo suppression loss, providing targeted optimization directions and helping them improve their noise suppression capabilities.

[0122] In one possible implementation, a target loss is determined based on the enhancement result of the sample speech signal, the difference between the sample label signal and the logarithmic variance. Specifically, during the training of the speech enhancement network based on the target loss, the target loss for the first training stage can be determined based on the difference between the enhancement result of the sample speech signal and the sample label signal, as well as the logarithmic variance. The target loss for the second training stage can then be determined based on the difference between the enhancement result of the sample speech signal and the sample label signal, as well as the logarithmic variance. The speech enhancement network is then trained sequentially based on the target losses of the first and second training stages. The target loss in the first training stage is used to constrain the accuracy of the logarithmic variance, while the target loss in the second training stage is used to assign corresponding weights to the time-frequency units based on the uncertainty of the time-frequency units when determining the difference between the enhancement result of the sample speech signal and the sample label signal. The sample label signal includes the real part signal, the imaginary part signal, and the clean signal. The real part signal corresponds to the sample label, the imaginary part signal corresponds to the sample label, and the clean signal corresponds to the sample label of the sample speech signal.

[0123] Specifically, in the first training phase, the real and imaginary part signals of the sample augmentation are input into the logarithmic variance estimation module to obtain the real variance, imaginary variance, and amplitude variance. In the second training phase, all parameters in the logarithmic variance estimation module are frozen, and the time-frequency units with high uncertainty are prioritized using the real variance, imaginary variance, and amplitude variance. Then, logarithmic operations are performed on the real variance, imaginary variance, and amplitude variance to obtain the logarithmic variance of the real part of the complex spectrum, the logarithmic variance of the imaginary part of the complex spectrum, and the logarithmic variance of the amplitude spectrum. For example, if the calculated real variance is θ = e, a logarithmic operation can be performed according to the formula v = lnθ to obtain the logarithmic variance v of the real part of the complex spectrum. real =1. Next, based on the difference between the enhanced sample speech signal and the sample label signal, as well as the logarithmic variance, the target loss for the first training stage and the target loss for the second training stage are determined respectively. The speech enhancement network is then trained sequentially based on the target losses of the first and second training stages. By training the speech enhancement network in stages, different training objectives and optimization strategies can be set for different stages of training, allowing for gradual optimization of the speech enhancement network and helping it converge more quickly during training.

[0124] In one possible implementation, the logarithmic variance includes the logarithmic variance of the real part of the complex spectrum, the logarithmic variance of the imaginary part of the complex spectrum, and the logarithmic variance of the amplitude spectrum. In determining the target loss for the first training phase based on the difference between the enhanced sample speech signal and the sample label signal, and the logarithmic variance, specifically, this can involve determining the differences in the real part of the complex spectrum, the imaginary part of the complex spectrum, and the amplitude spectrum between the enhanced sample speech signal and the sample label signal. The differences in the real part of the complex spectrum are weighted according to a first natural exponential function, and the differences in the imaginary part of the complex spectrum are weighted according to a second natural exponential function. The first loss is determined based on the sum of the logarithmic variance of the real part of the complex spectrum, the logarithmic variance of the imaginary part of the complex spectrum, the weighted differences in the real part of the complex spectrum, and the weighted differences in the imaginary part of the complex spectrum. The first natural exponential function uses the logarithmic variance of the real part of the complex spectrum as its exponent, and the second natural exponential function uses the logarithmic variance of the imaginary part of the complex spectrum as its exponent. The first loss is used to predict the uncertainty of the logarithmic variance of the real part of the complex spectrum and the logarithmic variance of the imaginary part of the complex spectrum. The amplitude spectrum differences are weighted using the third natural exponential function. The second loss is determined by summing the logarithmic variance of the amplitude spectrum with the weighted amplitude spectrum differences. The third natural exponential function uses the logarithmic variance of the amplitude spectrum as its exponent. The first loss and the second loss are then weighted and summed to obtain the target loss for the first training phase.

[0125] To determine the first loss, the difference in the real part of the complex spectrum is determined based on the real part signals of the sample enhancement and the sample label, and the difference in the imaginary part of the complex spectrum is determined based on the imaginary part signals of the sample enhancement and the sample label. Norms for both the real and imaginary part differences are calculated. Next, an exponential operation is performed on the logarithmic variance of the real part of the complex spectrum to obtain a first exponential result, and on the logarithmic variance of the imaginary part of the complex spectrum to obtain a second exponential result. The norms of the real part difference are then weighted according to the first exponential result to obtain a first weighted result, and the norms of the imaginary part difference are then weighted according to the second exponential result to obtain a second weighted result. Finally, the first loss is determined by the sum of the logarithmic variance of the real part of the complex spectrum, the logarithmic variance of the imaginary part of the complex spectrum, the first weighted result, and the second weighted result. The first loss L is then determined. com_EU The process can be represented by the following formula.

[0126] Among them, v real v represents the logarithmic variance of the real part of the complex spectrum. imag Represents the logarithm and variance of the imaginary part of the complex spectrum. This indicates the real part of the sample enhancement signal. S represents the sample-enhanced imaginary part of the signal. real S represents the real part of the sample label signal. imag Let represent the imaginary part of the sample label signal, and exp(·) represent the natural exponential function.

[0127] To determine the second loss, the amplitude spectrum difference is determined based on the enhanced speech signal and the clean, labeled signal, and the norm of this difference is calculated. Then, an exponential operation is performed on the log-variance of the amplitude spectrum to obtain a third exponential result. This third exponential result is then used to weight the norm of the amplitude spectrum difference to obtain a third weighted result. The second loss is determined by summing this third weighted result and the log-variance of the amplitude spectrum. The second loss L is then determined. mag_EU The process can be represented by the following formula.

[0128] Among them, v mag This represents the logarithmic variance of the amplitude spectrum. |S| represents the sample-enhanced speech signal, and |S| represents the sample-labeled clean signal.

[0129] When the target loss for the first training phase is obtained by weighted summation of the first loss and the second loss, a first weighting factor is assigned to the second loss. The second loss is then weighted based on this first weighting factor. The target loss for the first training phase is then calculated as the sum of the first loss and the weighted second loss. The target loss L for the first training phase is... stage1 This can be expressed by the following formula, where α is a fixed weight coefficient used to control the contribution of the second loss during training. stage1 =L com_EU +αL mag_EU

[0130] By weighted summing the differences in the real part, imaginary part, and amplitude spectrum of the complex spectrum based on the exponential values ​​of the logarithmic variance of the real part, the logarithmic variance of the imaginary part, and the logarithmic variance of the amplitude spectrum, the speech enhancement network can focus on the variance and characteristic distribution of each sub-band during training. This allows the speech enhancement network to effectively remove noise and other interference while minimizing the initial signal features, thereby improving the network's ability to enhance speech signals.

[0131] In one possible implementation, the processing block includes a gated recurrent unit (GRU) configured with binary gates. Specifically, in the process of weighted summing of a first loss and a second loss to obtain the target loss for the first training phase, the signal-to-noise ratio (SNR) loss can be determined based on the enhancement result of the sample speech signal and the sample label signal. An update rate loss can be determined based on the difference between the update rate of the binary gates and a preset update rate threshold. The first loss and the second loss are weighted summed to obtain a first uncertainty loss. Finally, the first uncertainty loss, the SNR loss, and the update rate loss are weighted summed to obtain the target loss for the first training phase. Here, the SNR loss is used to evaluate the degree of noise influence in the enhanced sample speech signal, and the update rate loss is used to train the GRU that performs the frame mixing mapping process, enabling the GRU to have a higher update rate. The update rate is the frequency at which the hidden state features in the GRU are updated, and the update rate threshold is a preset parameter used to measure and constrain the update frequency of the hidden state features.

[0132] Specifically, the enhancement result of the sample speech signal also includes the sample enhanced speech waveform, and the sample label signal also includes the sample label clean speech waveform. A decentralized sample waveform is obtained based on the sample enhanced speech waveform, and a decentralized sample label waveform is obtained based on the sample label clean speech waveform. Norm operations are performed on the decentralized sample label waveform, and norm operations are also performed on the difference between the decentralized sample waveform and the decentralized sample label waveform. The logarithm of the ratio of the norm value of the decentralized sample label waveform to the norm value of the difference between the decentralized sample waveform and the decentralized sample label waveform is used to obtain the signal-to-noise ratio (SNR) loss. The SNR loss L... SNR It can be expressed by the following formula,

[0133] in, This represents the decentralized sample enhancement waveform, s zm This represents the pure waveform of the decentralized sample, and log(·) represents the logarithmic function. The waveform of the centralized sample is then enhanced. It can be derived from the formula Calculations show that This represents the waveform of the enhanced sample, and the waveform of the purified decentralized sample. zm It can be derived from formula s zm =s-Mean(s) is calculated, where s represents the pure waveform of the sample. Mean(·) is the average value function used to calculate the average value of the enhanced waveform and the pure waveform of the sample.

[0134] Next, the update rate loss is determined based on the difference between the update rate of the binary gate and a preset update rate threshold. The update rate loss L update This can be expressed by the following formula, where g represents the update rate and u represents the update rate threshold. update =||gu||1

[0135] Next, the first loss and the second loss are weighted and summed to obtain the first uncertainty loss, L. EU It can be expressed by the following formula, L EU =L com_EU +αL mag_EU ,

[0136] A second weighting factor is configured for the signal-to-noise ratio loss, and a third weighting factor is configured for the update rate loss. The first uncertainty loss L is then adjusted based on the second and third weighting factors. EU Signal-to-noise ratio loss L SNR and update rate loss L update We perform a weighted summation to obtain the target loss for the first training phase, L. stage1 It can be expressed by the following formula, L stage1 =L EU +βL SNR +γL update ,

[0137] Wherein, β is the second weighting factor, and the value of the second weighting factor β can be 0.02, and γ is the third weighting factor, and the value of the third weighting factor γ can be 0.2.

[0138] The speech enhancement network is trained in the first stage by using the first uncertainty loss, signal-to-noise ratio loss, and update rate loss. This enables the log-variance estimation module to more accurately predict the uncertainty in the target signal features, thereby helping to improve the quality of the enhanced speech signal output by the speech enhancement network.

[0139] In one possible implementation, the process of determining the target loss for the second training stage based on the difference between the enhanced sample speech signal and the sample label signal, as well as the logarithmic variance, can specifically involve determining the difference in the real part of the complex spectrum, the difference in the imaginary part of the complex spectrum, and the difference in the amplitude spectrum between the enhanced sample speech signal and the sample label signal. The difference in the real part of the complex spectrum is then weighted according to the logarithmic variance of the real part of the complex spectrum, and the difference in the imaginary part of the complex spectrum is also weighted according to the logarithmic variance of the imaginary part of the complex spectrum. A third loss is determined based on the sum of the weighted differences in the real and imaginary parts of the complex spectrum. The difference in the amplitude spectrum is then weighted according to the logarithmic variance of the amplitude spectrum, and the weighted difference in the amplitude spectrum is determined as a fourth loss. Finally, the third and fourth losses are weighted and summed to obtain the target loss for the second training stage.

[0140] Specifically, the norm values ​​of the differences in the real part, imaginary part, and amplitude spectrum of the complex spectrum are calculated separately. A fourth weighted result is obtained by weighting the norm values ​​of the differences in the real part of the complex spectrum with the logarithmic variance of the real part. A fifth weighted result is obtained by weighting the norm values ​​of the differences in the imaginary part of the complex spectrum with the logarithmic variance of the imaginary part. A sixth weighted result is obtained by weighting the norm values ​​of the differences in the imaginary part of the amplitude spectrum with the logarithmic variance of the amplitude spectrum. The third loss is determined by summing the fourth and fifth weighted results, and the fourth loss is determined by the sixth weighted result. The third loss L com_UDL Fourth loss L mag_UDL It can be expressed by the following formula.

[0141] Next, a first weighting factor is assigned to the fourth loss, and the third and fourth losses are weighted and summed based on this first weighting factor to obtain the target loss for the second training phase. The target loss L for the second training phase is... stage2 It can be expressed by the following formula. L stage2 =L com_UDL +αL mag_UDL

[0142] By applying the logarithmic variance of the real part of the complex spectrum, the logarithmic variance of the imaginary part of the complex spectrum, and the logarithmic variance of the amplitude spectrum to the differences in the real part, imaginary part, and amplitude spectrum, respectively, greater weights can be assigned to time-frequency units under high uncertainty. This enables the speech enhancement network to better focus on highly uncertain content, thereby paying better attention to subtle features and improving the enhancement quality of speech.

[0143] In one possible implementation, the process of weighted summing of the third and fourth losses to obtain the target loss for the second training stage can specifically involve determining the signal-to-noise ratio (SNR) loss based on the enhancement results of the sample speech signals and the sample label signals, determining the update rate loss based on the difference between the update rate of the binary gate and a preset update rate threshold, weighted summing of the third and fourth losses to obtain the second uncertainty loss, and weighted summing of the second uncertainty loss, the SNR loss, and the update rate loss to obtain the target loss for the second training stage. Specifically, the second uncertainty loss is obtained by weighted summing of the third and fourth losses, where the second uncertainty loss L... UDL It can be expressed by the following formula, L UDL =L com_UDL +αL mag_UDL ,

[0144] A second weighting factor is configured for the signal-to-noise ratio loss, and a third weighting factor is configured for the update rate loss. The second uncertainty loss L is then adjusted based on the second and third weighting factors. UDL Signal-to-noise ratio loss L SNR and update rate loss Lupdate We perform a weighted summation to obtain the target loss for the second training phase, L. stage2 It can be expressed by the following formula. L stage2 =L UDL +βL SNR +γL update

[0145] In addition, the processing blocks of the speech enhancement network can also be used for causal temporal sampling. Causal temporal sampling is performed while maintaining the causality of the data, meaning that the order and temporal relativity of data points remain unchanged during the sampling process. Causal temporal sampling can include causal temporal downsampling and causal temporal upsampling. Different processing blocks have different time compression ratios when performing causal temporal sampling, but the time compression ratio is the same within any processing block. Therefore, after downsampling and upsampling within any processing block, the frame rate of the input and output features can remain unchanged. The target signal features are the features corresponding to the target speech signal in the mixed speech signal, and are used for speech enhancement.

[0146] In one possible implementation, during the encoder's encoding process based on initial signal features, the initial signal features are input into the first processing block of the encoder. The initial signal features are downsampled to obtain downsampled features. These downsampled features are then mapped to obtain intra-band relationship features indicating intra-band relationships. The first mapped features are then mapped to obtain inter-band relationship features indicating inter-band relationships. These inter-band relationship features are then upsampled, and the output of the first processing block is input into the next processing block, until the output of the last processing block is obtained, which serves as the encoder's encoding result. The downsampling operation reduces the frame rate of the initial signal features; this downsampling operation can be a temporal causal downsampling operation. The upsampling operation increases the frame rate of the inter-band relationship features; this upsampling operation can be a causal temporal upsampling operation.

[0147] Specifically, the initial signal features are input into the first processing block of the encoder. In this block, the initial signal features are first fed into a downsampling layer, where they are downsampled. This downsampling layer includes a first causal convolution, which can be composed of one-dimensional causal convolutions and primarily affects the frame rate of the initial signal features. During the downsampling operation, the kernel size and stride of the first causal convolution are the same as the temporal compression ratio. By reducing the frame rate of the initial signal features according to the temporal compression ratio, a low-frame-rate downsampled feature is obtained. For example, the feature dimension of the initial signal features is... Q represents the number of sub-bands after compression, so the feature dimension of the downsampled feature can be... λ represents the time compression ratio. Then, based on the downsampled features, mapping is performed to obtain intra-band relationship features indicating intra-band relationships. These intra-band relationship features are then transposed (using either the `transpose` or `reshape` function) to map the first mapped features, resulting in inter-band relationship features indicating inter-band relationships. The inter-band relationship features are input into an upsampling layer, where they are upsampled. This upsampling layer includes a second causal convolution, which can be a one-dimensional point convolution primarily affecting the frame rate of the inter-band relationship features. During the upsampling operation, the kernel size and stride of the second causal convolution are the same as the time compression ratio. First, interpolation is performed along the time dimension on the inter-band relationship features. Then, the frame rate of the interpolated inter-band relationship features is increased in the second causal convolution according to the time compression ratio, resulting in the output of the first processing block with the same frame rate as the initial signal features. For example, the feature dimension of the initial signal features is... When the feature dimension of the inter-band relationship feature is At that time, the output characteristics of the first output block can be The output of the first processing block is input into the next processing block to obtain the output of the next processing block. This process is repeated in each cascaded processing block until the output of the last processing block is obtained. The output of the last processing block is then used as the encoding result of the encoder.

[0148] It's important to note that the time compression ratio configured for each processing block in the encoder is different. Since the encoder and decoder are symmetrically constructed, the time compression ratio of each processing block in the decoder is the same as the time compression ratio of the processing block symmetrically positioned in the encoder. For example, if the encoder has six sequentially cascaded processing blocks with time compression ratios of 1, 2, 4, 8, 16, and 32 respectively, then the time compression ratios of the six sequentially cascaded processing blocks in the decoder will be 32, 16, 8, 4, 2, and 1. Because the time compression ratio configured for each processing block is different, upsampling and downsampling based on the time compression ratio can effectively control the computational complexity of the processing blocks, thereby improving the training and inference speeds of the encoder and decoder, and ultimately enhancing the real-time performance of speech enhancement.

[0149] Referring to Figure 9, which is a schematic diagram of another optional structure of a processing block in a speech enhancement network provided in an embodiment of this application, assuming that this processing block is the first processing block and the input signal to this processing block is the initial signal feature. The initial signal feature Z i The input signal is downsampled in the downsampling layer, and the initial signal characteristics Z are reduced by the time compression ratio λ. i The frame rate is used to obtain low frame rate downsampling features. Then downsample the feature Z↓ The input is fed into the in-band relation model for feature mapping to obtain the in-band relation features f. inter The in-band relation feature f is reshaped using the Reshape function. inter The frame dimension (T) and frequency dimension (Q) are transformed to achieve feature transpose, resulting in the transposed in-band relation features. Transposed in-band relation features The input is fed into a multilayer perceptron for interband modeling to perform a second feature mapping, resulting in interband relationship features. Finally, the inter-band relationship features The input is fed into the upsampling layer, and the inter-band relationship features are compressed using a time compression ratio λ. The frame rate is restored to the initial signal characteristic Z. i The output of this processing block is obtained when the frame rate is consistent with the frame rate. The above process can be represented by the following formula, Z ↓ =DS(Z) i ),

[0150] Where DS(·) represents downsampling, FC represents a fully connected layer, GRU represents a gated recurrent unit, Norm represents a normalization layer, and MLP represents a normalized layer. inter (·) represents a multi-sensor layer for inter-band modeling, and US(·) represents upsampling.

[0151] In one possible implementation, the process of mapping based on downsampled features to obtain in-band relation features indicating in-band relationships can specifically involve normalizing the downsampled features, mapping the normalized downsampled features through a gated recurrent unit to obtain the (N-1)th mapped feature, performing a fully connected operation on the (N-1)th mapped feature to obtain the Nth mapped feature, and summing the Nth mapped feature with the upsampled features to obtain the in-band relation features indicating in-band relationships. Here, the (N-1)th mapped feature is the feature after mapping by the gated recurrent unit, and the Nth mapped feature is the feature after the fully connected operation.

[0152] Specifically, the downsampled features are normalized to ensure they have the same data distribution. The normalized downsampled features are then input into a gated recurrent unit (ROU) for feature mapping. The ROU extracts sequence information or captures time-dependent features of the speech signal from the downsampled features; these features reflect the time-varying characteristics of the speech signal. The ROU includes reset and update gates. The reset gate discards weakly correlated features from the downsampled features, while the update gate retains important features relevant to the speech signal, resulting in a time-dependent (N-1)th mapped feature. The feature dimension of the (N-1)th mapped feature may differ from that of the downsampled features. This (N-1)th mapped feature is then input into a fully connected layer for further feature extraction to learn complex feature combinations. When the feature dimension of the (N-1)th mapped feature differs from that of the downsampled features, the number of output neurons in the fully connected layer is adjusted to remap the feature dimension of the (N-1)th mapped feature. During feature mapping, the (N-1)th mapped feature is mapped to a new feature space. The feature dimension of the new feature space is consistent with the feature dimension of the downsampled feature, thus obtaining the Nth mapped feature with the same feature dimension as the downsampled feature. Finally, the Nth mapped feature is added element-wise to the upsampled feature to fuse features of different scales and levels, resulting in multi-scale fused in-band relation features. For example, as shown in the in-band relation model in Figure 9, the downsampled feature Z... ↓ The input is fed into a normalization layer for normalization, resulting in normalized downsampled features. These normalized downsampled features are then fed into a gated recurrent unit for feature mapping, yielding the (N-1)th mapped feature. This (N-1)th mapped feature is then fed into a fully connected layer for further feature mapping, resulting in the Nth mapped feature. Finally, the Nth mapped feature is compared with the downsampled feature Z. ↓ The summation yields the in-band relation features.

[0153] By employing gated recurrent units, noise in the downsampled features can be filtered out while capturing temporal information and time dependencies, effectively improving the robustness of the encoder and decoder. Furthermore, combining normalization, gated recurrent units, fully connected layers, and feature fusion to perform feature mapping and feature combination on the downsampled features yields richer feature representations, which helps improve the accuracy of the decoder in predicting speech signals.

[0154] In addition, when the feature dimension of the (N-1)th mapped feature is much higher or lower than the feature dimension of the downsampled feature, feature mapping can be performed through convolution operations to avoid the situation of too many parameters or information loss that may occur when feature mapping is performed only through fully connected layers.

[0155] In one possible implementation, the outputs of each processing block are input to all the remaining processing blocks. The input of the nth processing block is obtained by summing the normalized outputs of the first to (n-1)th processing blocks, where n is a positive integer, n>2. Specifically, the output of each processing block is connected across scales with the inputs following it. That is, starting from the second processing block, the input of the nth processing block is related to the outputs of all the previous (n-1) processing blocks. The outputs of the previous (n-1) processing blocks are normalized and summed to obtain the input of the nth processing block. For example, the encoder contains 3 processing blocks (processing block 1, processing block 2, and processing block 3), and the decoder contains 3 processing blocks (processing block 4, processing block 5, and processing block 6). The initial signal features are input to processing block 1 of the encoder to obtain the first intermediate feature of processing block 1. The first intermediate feature is then input to processing block 2 to obtain the second intermediate feature. The second and first intermediate features are normalized separately. The normalized second and first intermediate features are then added together to obtain the first fused feature. This first fused feature is input into processing block 3 to obtain the third intermediate feature of processing block 3. The third, first, and first fused features are normalized separately and then added together to obtain the second fused feature. The second fused feature is input into processing block 4 to obtain the fourth intermediate feature. The fourth, first, second, and first intermediate features are normalized separately and then added together to obtain the third fused feature. The third fused feature is input into processing block 5 to obtain the fifth intermediate feature. The fifth, first, second, third, and first intermediate features are normalized separately and then added together to obtain the fourth fused feature. The fourth fused feature is input into processing block 6. The output of processing block 6 is the decoder output, and subsequent speech signal enhancement processing is performed based on this output. By connecting multiple processing blocks through cross-scale skip connections, the encoder and decoder can adapt to input data at different scales while calibrating the input data of previous processing blocks. This facilitates the encoder and decoder in capturing and integrating feature information from different scales. Furthermore, cross-scale skip connections can extract richer feature representations without increasing computational cost, improving the robustness and generalization ability of the encoder and decoder, thereby enhancing the effect of speech signal enhancement.

[0156] In one possible implementation, during the encoder's encoding process based on the initial signal features, the initial signal features can be divided into multiple signal sub-features according to multiple frequency ranges. Each signal sub-feature is then frequency-compressed to obtain a compressed sub-feature. These compressed sub-features are then merged along the frequency dimension to obtain a compressed feature. The encoder is then invoked to encode based on this compressed feature. Here, the frequency range is divided according to frequency units; a frequency range can consist of at least one frequency unit, and one frequency unit corresponds to a specific frequency sub-range. The signal sub-feature is obtained by dividing the initial signal features according to multiple frequency ranges, i.e., one frequency range corresponds to one signal sub-feature. The compressed sub-feature is obtained after frequency compression of the signal sub-feature.

[0157] Specifically, firstly, multiple frequency ranges are set, each containing different frequency units. The initial signal features are segmented into multiple signal sub-features along the frequency dimension according to these frequency ranges. Frequency compression is then performed on each signal sub-feature using convolutional groups, compressing *a* frequency units within the corresponding frequency range of each signal sub-feature into *b* sub-bands, where *a* > *b*, resulting in compressed sub-features for each signal sub-feature. Finally, these compressed sub-features are merged along the frequency dimension to obtain compressed features, which are then encoded using an encoder. By compressing multiple frequency units into a smaller number of sub-bands along the frequency dimension, dimensionality reduction is achieved, reducing computational overhead for subsequent operations based on compressed features and improving the efficiency of speech enhancement.

[0158] Referring to Figure 10, which is an optional schematic diagram of frequency band segmentation provided in an embodiment of this application, the initial signal features are divided into P frequency ranges along the frequency dimension, and the signal sub-features R corresponding to each frequency range are... p Frequency band segmentation is performed using a set of two-dimensional convolutions with different kernel sizes and the same output channel. The features processed by different kernel sizes are then concatenated to obtain a compressed three-dimensional representation. Where p∈{1,...,P}. For example, the signal sub-feature R1 is composed of a set of convolutional kernels of size k. 11 ,...,k 1M A two-dimensional convolution with a stride of s1 is used for frequency band segmentation. Features processed by different convolution kernel sizes are then concatenated to obtain the compressed sub-features corresponding to the signal sub-features R1. Then, the compressed sub-features corresponding to each signal sub-feature are merged to obtain the compressed features. The above process can be represented by the following formula: {R1,...,R P} = RegionSplit(R),

[0159] Among them, RegionSplit(·) represents a frequency range-based segmentation operation, Cat(·) represents a splicing operation, and Merge(·) represents a merging operation.

[0160] For example, assuming the initial signal features contain 161 frequency units, then the feature dimension of the initial signal features could be... When P=3, the 161 frequency units are divided into 3 frequency ranges, and the initial signal features are further divided into 3 signal sub-features according to these 3 frequency ranges. The first frequency range contains 20 frequency units, so the feature dimension of the first signal sub-feature corresponding to the first frequency range can be... The second frequency range contains 60 frequency units, so the feature dimension of the second signal sub-feature corresponding to the second frequency range can be... The third frequency range contains 81 frequency units, so the feature dimension of the third signal sub-feature corresponding to the third frequency range can be... Frequency compression is performed on each of the three signal sub-features to obtain the feature dimension of the first compressed sub-feature corresponding to the first signal sub-feature. If Q1 < 20, the feature dimension of the second compressed sub-feature corresponding to the second signal sub-feature can be: If Q2 < 60, the feature dimension of the third compressed sub-feature corresponding to the third signal sub-feature can be: Q3 < 81. Finally, these three compressed sub-features are merged along the frequency dimension to obtain the compressed feature. The feature dimension of the compressed feature can be... Where Q = Q1 + Q2 + Q3, the compressed features are input into the encoder of the speech enhancement network for encoding.

[0161] In one possible implementation, the process of frequency compression for each signal sub-feature to obtain compressed sub-features can specifically involve performing frequency compression on any given signal sub-feature through multiple convolutional layers to obtain multiple fifth convolutional features corresponding to the signal sub-feature. These multiple fifth convolutional features corresponding to the same signal sub-feature are then merged along the channel dimension to obtain the compressed sub-features corresponding to each signal sub-feature. Each of the multiple convolutional layers is configured with different kernel sizes and the same number of output channels. The kernel size is related to the frequency range corresponding to the signal sub-feature, and different kernel sizes correspond to different frequency ranges.

[0162] Specifically, for any given signal sub-feature, multiple convolutional layers are used to compress its frequency. Each convolutional layer has a different kernel size but the same stride. After passing through each convolutional layer, the signal sub-feature is transformed into a fifth convolutional feature corresponding to that layer. These fifth convolutional features are then merged along the channel dimension to obtain the compressed sub-feature. By using a set of convolutions with different kernel sizes to compress the frequency of the same signal sub-feature, features at different scales can be captured, thereby enhancing the understanding and representation capabilities of the signal sub-feature. Furthermore, the merging of these fifth convolutional features promotes the fusion and integration of features at different scales, contributing to a richer and more accurate feature representation and improving the ability of subsequent steps to enhance the speech signal.

[0163] For example, referring to Figure 11, which is an optional schematic diagram of frequency band segmentation of signal sub-features in Figure 10 provided by an embodiment of this application, Figure 11 divides each frequency unit into 3 frequency ranges, and the initial signal features are segmented according to the frequency ranges. Divided into first signal sub-features Second signal sub-characteristics and the third signal sub-feature For the first signal sub-feature R1, frequency compression can be performed on it using three sets of convolutions. The kernel size of convolution A is a1×b1, and the stride is (s1,t1). That is, the a1×b1 convolution kernel is used to perform convolution operations on the frame rate dimension of the first signal sub-feature R1 with a stride of s1 and on the frequency dimension with a stride of t1 to obtain the convolutional feature. The kernel size of convolution B is a1×b2, and the stride is (s1,t1). That is, the a1×b2 kernel is used to perform convolution operation on the frame rate dimension of the first signal sub-feature R1 with a stride of s1 and the frequency dimension with a stride of t1 to obtain the convolutional feature. The kernel size of convolution C is a1×b3, and the stride is (s1, t1). That is, using the a1×b3 kernel, convolution is performed on the first signal sub-feature R1 with a stride of s1 in the frame rate dimension and a stride of t1 in the frequency dimension to obtain the convolutional feature. Then, the convolutional features Convolutional features and convolutional features By concatenating along the channel dimension, the first compressed sub-feature corresponding to the first signal sub-feature R1 is obtained. The frequency compression of the second signal sub-feature R2 and the third signal sub-feature R3 is the same as that of the first signal sub-feature R1, and will not be repeated here. It is worth noting that for signal sub-features within the same frequency range, frequency compression is performed using convolution operations with the same length but different kernel sizes; for signal sub-features within different frequency ranges, frequency compression is performed using convolution operations with different lengths and different kernel sizes.

[0164] In addition to using multiple convolutional layers to compress the frequency of signal sub-features, a single convolution can also be used to compress the frequency of signal sub-features. This can be done by using a one-dimensional convolution with a stride of at least 2 in the frequency dimension, or by using a two-dimensional convolution with a kernel size of a×b (a≠b) with a stride of (s,t), s≠t.

[0165] In one possible implementation, the process of merging multiple compressed sub-features along the frequency dimension to obtain compressed features can specifically involve merging multiple compressed sub-features along the frequency dimension, restoring the channel dimension of the merged result to match the channel dimension of the initial signal features, and then normalizing it to obtain compressed features. Specifically, multiple compressed sub-features are merged along the frequency dimension to obtain merged features. Then, the merged features are input into a convolutional layer, which reduces the channel dimension of the merged features, restoring the channel dimension of the merged features to match the channel dimension of the initial signal features. Finally, the merged features with restored channel dimensions are normalized to obtain compressed features. For example, the feature dimension of the initial signal features is... By merging multiple compressed sub-features along the frequency dimension, the feature dimension of the resulting merged feature can be... M represents the convolution scale. The merged features are input into a two-dimensional convolution, reducing the channel dimension from (M×E) to E, i.e., restoring it to the same channel dimension as the initial signal features. Finally, the merged features with the restored channel dimension are normalized to obtain compressed features. The feature dimension of the compressed features can be... By keeping the channel dimension of the output compressed features consistent with the channel dimension of the input initial signal features, it helps to reduce the computational cost of the frequency compression process, thereby reducing the complexity of the frequency compression process, improving the efficiency of frequency compression, and thus improving the efficiency of enhancing speech signals.

[0166] In one possible implementation, during the encoding process based on the initial signal features, the initial signal features can be segmented into multiple signal sub-features along the frequency dimension. Each signal sub-feature is then normalized, and feature mapping is performed on the normalized signal sub-features to obtain multiple mapped sub-features. These multiple mapped sub-features are then merged along the frequency dimension to obtain segmented mapped features, which are then input into the encoder of the speech enhancement network for encoding. Referring to Figure 12, which is another optional schematic diagram of frequency band segmentation provided in this embodiment, the initial signal feature X is segmented into Q signal sub-features along the frequency dimension. Where q = 1, 2, ..., Q, f q This represents the frequency division unit of the q-th sub-band, satisfying f1 + f2 + ... + f Q =F. Next, the Q signal sub-features X q Perform normalization operations separately, and then normalize the signal sub-features X. q The input is fed into a convolutional layer for feature mapping, resulting in multiple sub-feature maps. H q ={H1,...,H Q This convolutional layer can be a one-dimensional convolutional layer. Finally, multiple mapper features H are combined. q Merging along the frequency dimension yields the segmentation mapping feature H. A convolutional layer is then used to map the normalized signal sub-features, further simplifying the frequency band segmentation process of the initial signal features. This effectively reduces the computational load during frequency band segmentation, thereby improving speech enhancement efficiency and contributing to enhanced real-time performance.

[0167] In one possible implementation, the voice enhancement method provided in this application embodiment can be applied in communication scenarios. It acquires mixed voice signals through terminal devices such as telephones, obtains voice signals to be processed based on the mixed voice signals, and inputs the voice signals to be processed into the voice enhancement model provided in this application embodiment for echo cancellation and noise suppression to reduce background noise during communication, making the call clearer, thereby improving the call experience and ensuring call quality.

[0168] In another possible implementation, the speech enhancement method provided in this application embodiment can be applied to a conference system. Since conference venues are usually large and prone to echoes, by collecting mixed speech signals from the conference venue, echo cancellation and noise suppression are performed based on the speech enhancement method provided in this application embodiment to improve the clarity of the conference speech and the participation of each participant.

[0169] Referring to Figures 13 and 14, Figure 13 is a schematic diagram of an optional overall process of the speech enhancement method provided in this application embodiment, and Figure 14 is a schematic diagram of an optional structure of the speech enhancement model in Figure 13 provided in this application embodiment. The principle of the speech enhancement method in this application embodiment is described in general and complete below:

[0170] The speech enhancement method in this application embodiment can be implemented through a speech enhancement model, which includes a feature extraction layer, a frequency band separation layer, a speech enhancement network, a frequency band merging layer, and a post-processing network.

[0171] First, sample speech signals are acquired, and the encoder and decoder are trained based on the joint loss obtained by weighted summation of at least two loss functions to obtain a trained speech enhancement model.

[0172] Next, the mixed speech signal d(n) and the reference speech signal x(n) are acquired and input into the echo prediction model to obtain the echo signal y(n) and the error speech signal e(n). The mixed speech signal d(n), the reference speech signal x(n), the echo signal y(n), and the error speech signal e(n) are then transformed in the time-frequency domain using short-time Fourier transform to obtain the mixed speech spectrum D, the reference speech spectrum X, the echo spectrum Y, and the error speech spectrum E, respectively, so that these four signals are represented in complex form in the frequency domain. Then, the mixed speech spectrum D, the reference speech spectrum X, the echo spectrum Y, and the error speech spectrum E are concatenated along the channel dimension to obtain the speech signal I to be processed. The speech signal I to be processed is input into the feature extraction layer for feature extraction. Feature extraction is performed through two-dimensional convolution, and the extracted features are normalized to obtain the initial signal features R.

[0173] Next, the initial signal features R are input into the speech enhancement model. The initial signal features R are first input into the frequency band separation layer (as shown in Figure 10), where they are divided into P signal sub-features according to the frequency range. Each signal sub-feature is then frequency-compressed using a set of convolutions with different kernel sizes and the same output channel, resulting in compressed sub-features. These compressed sub-features are then merged along the frequency dimension to obtain compressed features H. The compressed features H are then input into the encoder, where they are encoded through multiple cascaded processing blocks. The encoded result is input into the decoder, which decodes the encoded result through multiple cascaded primary blocks to obtain the decoded result U.

[0174] In each processing block (as shown in Figure 9), for the features of the input processing block, the input features are first downsampled according to the time compression ratio to obtain downsampled features. Intra-band relationship modeling is performed on the downsampled features along the frame rate dimension to obtain intra-band relationship features. Based on the intra-band relationship features, inter-band relationship modeling is performed along the frequency dimension (inter-band relationship modeling is shown in Figure 6) to obtain inter-band relationship features. Finally, the inter-band relationship features are upsampled according to the time compression ratio to obtain the output of the processing block.

[0175] Next, the decoding result U is input into the frequency band merging layer (as shown in Figure 8). The decoding result is segmented into multiple decoded sub-features along the frequency dimension. Each decoded sub-feature is normalized and then input into the prediction sub-time-frequency mask in the multilayer perceptual layer. The sub-time-frequency masks are then merged along the frequency dimension to obtain the target speech signal G. The target speech signal G is then element-wise multiplied with the speech signal I to be processed to obtain the fused features. When residual noise exists in the target speech signal, the features will be fused. The signal is input into a post-processing network, where gated recurrent units and linear transforms are used to further suppress noise and obtain enhanced signal features.

[0176] Finally, the enhanced signal features are obtained by inverse short-time Fourier transform. Perform an inverse time-frequency domain transformation to obtain the enhanced speech signal ξ(n).

[0177] Referring to Figure 15, which is a schematic diagram of another optional structure of the speech enhancement model in Figure 13 provided in an embodiment of this application, the speech enhancement model includes a frequency band separation layer, a speech enhancement network, a frequency band merging layer, and a log-variance estimation module. The principle of the speech enhancement method in the embodiment of this application is described in its entirety based on Figure 15 below:

[0178] First, sample speech signals are acquired. Based on these sample speech signals, a first uncertainty loss, a signal-to-noise ratio loss, and a update rate loss, the speech enhancement network undergoes a first-stage training. Then, based on the sample speech signals, a second uncertainty loss, a signal-to-noise ratio loss, and a update rate loss, the speech enhancement network undergoes a second-stage training, resulting in a well-trained speech enhancement network. The log-variance estimation module is used to predict the uncertainty of the enhancement result of the sample speech signals during training.

[0179] Next, the mixed speech signal, reference speech signal, echo signal, and error speech signal are acquired as the speech signal to be processed, I. Initial signal features X are extracted from the speech signal to be processed, wherein the initial signal features X include the real part feature of the complex spectrum X. r and the imaginary part characteristic X of the complex spectrum i .

[0180] Next, the initial signal feature X is input into the frequency band segmentation layer (as shown in Figure 12). Based on the frequency dimension, the initial signal feature X is segmented into Q sub-frequency bands, each sub-frequency band corresponding to a signal sub-feature. Feature mapping is performed on multiple signal sub-features to obtain the mapped sub-features corresponding to each signal sub-feature. The mapped sub-features are then merged to obtain the segmentation mapping feature H. The segmentation mapping feature H is then input into the encoder of the speech enhancement network. The segmentation mapping feature H is encoded through multiple sequentially cascaded processing blocks to obtain the encoding result. The encoding result is then input into the decoder for decoding to obtain the decoding result U.

[0181] In each processing block (as shown in Figure 3), the features of the input encoder processing block are first processed by the frame mixing module along the time dimension to obtain the first mapped features. The first mapped features are then input into the band mixing module for sub-band mixing along the sub-band dimension to obtain the second mapped features. Finally, the second mapped features are input into the sub-band compression / decompression module for band compression to obtain the output of the processing block. For the features of the input decoder processing block, after obtaining the second mapped features, they are input into the sub-band compression / decompression module for band decompression to obtain the output of the processing block.

[0182] Next, the decoding result U is input into the frequency band merging layer (as shown in Figure 8). The decoding result is segmented into multiple decoded sub-features along the frequency dimension. Each decoded sub-feature is normalized and then input into the prediction sub-time-frequency mask in the multilayer perceptual layer. The sub-time-frequency masks are then merged along the frequency dimension to obtain the target speech signal G. The target speech signal G and the initial signal feature X are then multiplied element-wise to obtain the enhanced complex spectrum real part feature X. r and the enhanced imaginary part feature of the complex spectrum X i .

[0183] Finally, the enhanced real part features X of the complex spectrum are obtained by inverse short-time Fourier transform. r and the enhanced imaginary part feature of the complex spectrum X i Perform an inverse time-frequency domain transformation to obtain the enhanced speech signal ξ(n).

[0184] The speech enhancement method provided in this application segmentes the initial speech signal into frequency bands, compresses frequency units along the frequency dimension to obtain compressed features, and then passes the compressed features through an encoding / decoding network to suppress noise, resulting in a decoding result. Next, the decoding result is segmented into multiple decoded sub-features along the frequency dimension. Each decoded sub-feature is normalized and input into a prediction sub-time-frequency mask in a multi-layer perceptual layer. The sub-time-frequency masks are then merged along the frequency dimension to obtain the target speech signal. A post-processing network further suppresses noise in the target speech signal to obtain enhanced signal features. Based on these enhanced signal features, an enhanced speech signal is obtained. This method effectively reduces the computational complexity of speech enhancement and improves the real-time performance of speech enhancement.

[0185] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0186] Referring to Figure 16, which is a schematic diagram of the structure of a speech enhancement device provided in an embodiment of this application, the speech enhancement device 1600 includes:

[0187] The input module 1601 is used to acquire the speech signal to be processed, extract features from the speech signal to be processed to obtain initial signal features, and input the initial signal features into the speech enhancement network. The speech enhancement network includes an encoder and a decoder connected to each other, and each encoder and decoder is provided with multiple processing blocks cascaded in sequence.

[0188] The encoding module 1602 is used to perform frame mixing mapping on the initial signal features based on the first processing block in the encoder to obtain the first mapping feature, perform band mixing mapping on the first mapping feature to obtain the second mapping feature, perform band compression on the second mapping feature to obtain the output of the first processing block, and input the output of the first processing block to the next processing block until the encoding result of the encoder is obtained.

[0189] The decoding module 1603 is used to decode the encoder's encoding result based on the decoder to obtain the target signal features;

[0190] The restoration module 1604 is used to restore the enhanced speech signal from the speech signal to be processed based on the characteristics of the target signal.

[0191] In one possible implementation, the processing block includes a gated loop unit, and the encoding module 1602 is further used for:

[0192] The initial signal features are normalized, and the normalized initial signal features are then frame-mixed and mapped by a gated loop unit to obtain the output of the gated loop unit.

[0193] The output of the gated loop unit is fully connected to obtain the first mapping feature.

[0194] In one possible implementation, the gated loop unit is configured with binary gates, which are used to indicate whether to update the hidden state characteristics of the gated loop unit or to keep the hidden state characteristics unchanged. The encoding module 1602 is also used for:

[0195] For any current time step among multiple time steps, obtain the initial state update probability and the change value of the initial state update probability of the gated loop unit in the previous time step of the current time step, and determine the target state update probability of the gated loop unit in the current time step based on the initial state update probability and the change value of the initial state update probability.

[0196] The target state update probability is rounded to obtain the gating parameters corresponding to the binary gate in the current time step;

[0197] The hidden state features of the gated loop unit in the current time step are determined based on the gating parameters. Based on the hidden state features, the normalized initial signal features are frame-mixed and mapped by the gated loop unit to obtain the output of the gated loop unit.

[0198] In one possible implementation, the encoding module 1602 is also used for:

[0199] Determine the initial state update probability for the current time step based on the gating parameters corresponding to the current time step;

[0200] Based on the gating parameters and target state update probability corresponding to the current time step, determine the change value of the initial state update probability corresponding to the current time step.

[0201] In one possible implementation, the encoding module 1602 is also used for:

[0202] The sample speech signal and the corresponding sample label signal are obtained. The speech enhancement network is called to enhance the sample speech signal and obtain the enhancement result of the sample speech signal.

[0203] The log-variance estimation module determines the log-variance of the enhancement result of the sample speech signal. Based on the difference between the enhancement result of the sample speech signal and the sample label signal, as well as the log-variance, the target loss is determined, and the speech enhancement network is trained based on the target loss.

[0204] In one possible implementation, the encoding module 1602 is also used for:

[0205] The target loss for the first training phase is determined based on the difference between the enhanced sample speech signal and the sample label signal, as well as the logarithmic variance. The target loss for the first training phase is used to constrain the accuracy of the logarithmic variance.

[0206] The target loss of the second training stage is determined based on the difference between the enhancement result of the sample speech signal and the sample label signal, as well as the logarithmic variance. The target loss of the second training stage is used to assign corresponding weights to the time-frequency unit according to the uncertainty of the time-frequency unit when determining the difference between the enhancement result of the sample speech signal and the sample label signal.

[0207] The speech enhancement network is trained sequentially based on the target loss of the first training phase and the target loss of the second training phase.

[0208] In one possible implementation, the encoding module 1602 is also used for:

[0209] Determine the enhancement results of the sample speech signals and the differences in the real part of the complex spectrum, the imaginary part of the complex spectrum, and the amplitude spectrum between the sample label signals;

[0210] The differences in the real part of the complex spectrum are weighted according to the first natural exponential function, and the differences in the imaginary part of the complex spectrum are weighted according to the second natural exponential function. The first loss is determined by the sum of the logarithmic variance of the real part of the complex spectrum, the logarithmic variance of the imaginary part of the complex spectrum, the weighted differences in the real part of the complex spectrum, and the weighted differences in the imaginary part of the complex spectrum. The first natural exponential function uses the logarithmic variance of the real part of the complex spectrum as the exponent, and the second natural exponential function uses the logarithmic variance of the imaginary part of the complex spectrum as the exponent.

[0211] The amplitude spectrum difference is weighted according to the third natural exponential function, and the second loss is determined by the sum of the logarithmic variance of the amplitude spectrum and the weighted amplitude spectrum difference. The third natural exponential function uses the logarithmic variance of the amplitude spectrum as the exponent.

[0212] The target loss for the first training phase is obtained by weighted summation of the first loss and the second loss.

[0213] In one possible implementation, the encoding module 1602 is also used for:

[0214] The signal-to-noise ratio loss is determined based on the enhancement results of the sample speech signal and the sample label signal, and the update rate loss is determined based on the difference between the update rate of the binary gate and the preset update rate threshold.

[0215] The first uncertainty loss is obtained by weighted summing of the first loss and the second loss. The target loss of the first training phase is obtained by weighted summing of the first uncertainty loss, the signal-to-noise ratio loss, and the update rate loss.

[0216] In one possible implementation, the encoding module 1602 is also used for:

[0217] Determine the enhancement results of the sample speech signals and the differences in the real part of the complex spectrum, the imaginary part of the complex spectrum, and the amplitude spectrum between the sample label signals;

[0218] The difference in the real part of the complex spectrum is weighted according to the logarithmic variance of the real part of the complex spectrum, and the difference in the imaginary part of the complex spectrum is weighted according to the logarithmic variance of the imaginary part of the complex spectrum. The third loss is determined by the sum of the weighted difference in the real part of the complex spectrum and the weighted difference in the imaginary part of the complex spectrum.

[0219] The amplitude spectrum difference is weighted according to the logarithmic variance of the amplitude spectrum, and the weighted amplitude spectrum difference is determined as the fourth loss.

[0220] The target loss for the second training phase is obtained by weighted summation of the third and fourth losses.

[0221] In one possible implementation, the processing block includes a gated loop unit configured with binary gates, and the encoding module 1602 is further used for:

[0222] The signal-to-noise ratio loss is determined based on the enhancement results of the sample speech signal and the sample label signal, and the update rate loss is determined based on the difference between the update rate of the binary gate and the preset update rate threshold.

[0223] The second uncertainty loss is obtained by weighted summing of the third and fourth losses. The target loss for the second training phase is obtained by weighted summing of the second uncertainty loss, the signal-to-noise ratio loss, and the update rate loss.

[0224] In one possible implementation, the encoding module 1602 is also used for:

[0225] The first mapping feature is segmented along the channel dimension to obtain the first intra-band relation sub-feature and the second intra-band relation sub-feature;

[0226] The first intra-band relational sub-features are normalized, and then convolved along the time axis in the sub-band dimension to obtain the first convolutional feature.

[0227] Determine the first product of the first convolutional feature and the second intra-band relational feature, and obtain the second mapping feature based on the sum of the first product and the second intra-band relational feature.

[0228] In one possible implementation, the encoding module 1602 is also used for:

[0229] The first mapping feature is convolved along the time axis in the channel dimension to obtain the second convolutional feature;

[0230] The second convolutional feature is normalized, and the normalized second convolutional feature is segmented along the channel dimension to obtain the first intra-band relational sub-feature and the second intra-band relational sub-feature.

[0231] In one possible implementation, the encoding module 1602 is also used for:

[0232] Convolve the sum of the first product and the second intra-band relational feature along the time axis in the channel dimension to obtain the inter-band relational feature used to indicate the inter-band relation.

[0233] In one possible implementation, the encoding module 1602 is also used for:

[0234] The second mapping feature is normalized, and the normalized second mapping feature is convolved in parallel to obtain the third and fourth convolution features with compression.

[0235] The fourth convolutional feature is non-linearly activated to obtain the attention weights;

[0236] The third convolutional features are weighted based on attention weights to obtain the output of the first processing block.

[0237] In one possible implementation, the decoding module 1603 is used for:

[0238] The decoder is invoked to decode based on the encoder's encoding result, and the decoding result is segmented along the frequency dimension to obtain multiple decoded sub-features;

[0239] The channel dimensions of each decoded sub-feature are restored to be consistent with the channel dimensions of the speech signal to be processed, and the multiple restored decoded sub-features are merged along the frequency dimension to obtain the target signal feature. Alternatively, multiple decoded sub-features are merged along the frequency dimension, and the channel dimensions of the merged decoded feature are restored to be consistent with the channel dimensions of the speech signal to be processed to obtain the target signal feature.

[0240] In one possible implementation, the restore module 1604 is used for:

[0241] Determine the second product between the target signal features and the speech signal to be processed;

[0242] The second product is filtered to restore the enhanced speech signal from the speech signal to be processed.

[0243] The electronic device provided in this application embodiment for performing the above-described voice enhancement method can be a terminal. Referring to FIG17, FIG17 is a partial structural block diagram of the terminal provided in this application embodiment. The terminal includes: a camera assembly 1710, a first memory 1720, an input unit 1730, a display unit 1740, a sensor 1750, an audio circuit 1760, a wireless fidelity (WiFi) module 1770, a first processor 1780, and a first power supply 1790, etc. Those skilled in the art will understand that the terminal structure shown in FIG17 does not constitute a limitation on the terminal, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0244] The camera assembly 1710 can be used to capture images or videos. Optionally, the camera assembly 1710 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions.

[0245] The first memory 1720 can be used to store software programs and modules. The first processor 1780 executes various terminal functions and data processing by running the software programs and modules stored in the first memory 1720.

[0246] The input unit 1730 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 1730 may include a touch panel 1731 and other input devices 1732.

[0247] The display unit 1740 can be used to display input or provided information, as well as various menus of the terminal. The display unit 1740 may include a display panel 1741.

[0248] Audio circuitry 1760, speaker 1761, and microphone 1762 provide an audio interface.

[0249] The first power source 1790 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0250] The number of sensors 1750 can be one or more, and these sensors 1750 include, but are not limited to: accelerometers, gyroscopes, pressure sensors, optical sensors, etc.

[0251] An accelerometer can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by the terminal. For example, an accelerometer can be used to detect the components of gravitational acceleration along the three coordinate axes. The first processor 1780 can control the display unit 1740 to display the user interface in either a horizontal or vertical view based on the gravitational acceleration signal acquired by the accelerometer. The accelerometer can also be used for games or for acquiring user motion data.

[0252] The gyroscope sensor can detect the terminal's orientation and rotation angle. It can work in conjunction with an accelerometer to collect 3D user movements on the terminal. Based on the data collected by the gyroscope sensor, the first processor 1780 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0253] The pressure sensor can be installed on the side bezel of the terminal and / or below the display unit 1740. When the pressure sensor is installed on the side bezel of the terminal, it can detect the user's grip signal on the terminal, and the first processor 1780 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor. When the pressure sensor is installed below the display unit 1740, the first processor 1780 can control the operable controls on the UI interface based on the user's pressure operation on the display unit 1740. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0254] An optical sensor is used to collect ambient light intensity. In one embodiment, the first processor 1780 can control the display brightness of the display unit 1740 based on the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1740 is increased; when the ambient light intensity is low, the display brightness of the display unit 1740 is decreased. In another embodiment, the first processor 1780 can also dynamically adjust the shooting parameters of the camera assembly 1710 based on the ambient light intensity collected by the optical sensor.

[0255] In this embodiment, the first processor 1780 included in the terminal can execute the voice enhancement method of the previous embodiment.

[0256] The electronic device for executing the above-described voice enhancement method provided in this application embodiment can also be a server. Referring to Figure 18, which is a partial structural block diagram of the server provided in this application embodiment, the server can vary considerably due to different configurations or performance. It may include one or more second processors 1810 and second memories 1830, and one or more storage media 1840 (e.g., one or more mass storage devices) for storing application programs 1843 or data 1842. The second memory 1830 and storage media 1840 can be temporary or persistent storage. The program stored in the storage media 1840 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the second processor 1810 may be configured to communicate with the storage media 1840 and execute a series of instruction operations in the storage media 1840 on the server.

[0257] The server may also include one or more secondary power supplies 1820, one or more wired or wireless network interfaces 1850, one or more input / output interfaces 1860, and / or one or more operating systems 1841, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0258] The second processor 1810 in the server can be used to execute speech enhancement methods.

[0259] This application also provides a computer-readable storage medium for storing a computer program for executing the speech enhancement methods of the foregoing embodiments.

[0260] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the above-described speech enhancement method.

[0261] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate to describe embodiments of this application, for example, those that can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0262] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0263] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0264] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0265] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0266] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0267] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0268] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0269] The above provides a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A speech enhancement method, the method being performed by an electronic device, comprising: The speech signal to be processed is acquired, and features are extracted from the speech signal to obtain initial signal features. The initial signal features are then input into a speech enhancement network, wherein the speech enhancement network includes an encoder and a decoder connected to each other, and each encoder and decoder is provided with multiple processing blocks cascaded in sequence. Based on the first processing block in the encoder, the initial signal features are subjected to frame mixing mapping to obtain a first mapping feature, the first mapping feature is subjected to band mixing mapping to obtain a second mapping feature, the second mapping feature is subjected to band compression to obtain the output of the first processing block, and the output of the first processing block is input to the next processing block until the encoding result of the encoder is obtained; The encoder's encoding result is decoded based on the decoder to obtain the target signal features; An enhanced speech signal is obtained by restoring the speech signal to be processed from the target signal features.

2. The speech enhancement method according to claim 1, wherein the processing block includes a gated loop unit, and the step of performing frame mixing mapping on the initial signal features to obtain the first mapping features includes: The initial signal features are normalized, and the normalized initial signal features are then subjected to frame mixing mapping by the gated loop unit to obtain the output of the gated loop unit. The output of the gated loop unit is fully connected to obtain the first mapping feature.

3. The speech enhancement method according to claim 2, wherein the speech enhancement network is configured to iterate through multiple time steps, the gated recurrent unit is configured with binary gates, the binary gates being used to indicate whether to update the hidden state features of the gated recurrent unit or to keep the hidden state features unchanged, and the step of performing frame mixing mapping on the normalized initial signal features through the gated recurrent unit to obtain the output of the gated recurrent unit includes: For any current time step among the multiple time steps, obtain the initial state update probability and the change value of the initial state update probability of the gated loop unit in the previous time step of the current time step, and determine the target state update probability of the gated loop unit in the current time step based on the initial state update probability and the change value of the initial state update probability. The target state update probability is rounded down to obtain the gating parameter corresponding to the binary gate in the current time step; The hidden state features output by the gated loop unit in the current time step are determined according to the gated parameters. Based on the hidden state features, the normalized initial signal features are subjected to frame mixing mapping by the gated loop unit to obtain the output of the gated loop unit.

4. The speech enhancement method according to claim 3, after rounding the target state update probability to obtain the gating parameter corresponding to the binary gate in the current time step, the speech enhancement method further includes: The initial state update probability corresponding to the current time step is determined based on the gating parameters corresponding to the current time step. Based on the gating parameters corresponding to the current time step and the target state update probability, determine the change value of the initial state update probability corresponding to the current time step.

5. The speech enhancement method according to any one of claims 1-4, wherein the speech enhancement network further comprises a logarithmic variance estimation module, and before inputting the initial signal features into the speech enhancement network, the speech enhancement method further comprises: Obtain the sample speech signal and the sample label signal corresponding to the sample speech signal, call the speech enhancement network to perform speech enhancement on the sample speech signal, and obtain the enhancement result of the sample speech signal; The log-variance estimation module determines the log-variance of the enhancement result of the sample speech signal, and determines the target loss based on the difference between the enhancement result of the sample speech signal and the sample label signal, as well as the log-variance. The speech enhancement network is then trained based on the target loss.

6. The speech enhancement method according to claim 5, wherein determining the target loss based on the difference between the enhancement result of the sample speech signal and the sample label signal, and the logarithmic variance, and training the speech enhancement network based on the target loss, comprises: The target loss for the first training stage is determined based on the difference between the enhancement result of the sample speech signal and the sample label signal, and the logarithmic variance, wherein the target loss for the first training stage is used to constrain the accuracy of the logarithmic variance. The target loss of the second training stage is determined based on the difference between the enhancement result of the sample speech signal and the sample label signal, and the logarithmic variance. The target loss of the second training stage is used to assign corresponding weights to the time-frequency unit based on the uncertainty of the time-frequency unit when determining the difference between the enhancement result of the sample speech signal and the sample label signal. The speech enhancement network is trained sequentially based on the target loss of the first training phase and the target loss of the second training phase.

7. The speech enhancement method according to claim 6, wherein the logarithmic variance includes the logarithmic variance of the real part of the complex spectrum, the logarithmic variance of the imaginary part of the complex spectrum, and the logarithmic variance of the amplitude spectrum, and the step of determining the target loss of the first training stage based on the difference between the enhancement result of the sample speech signal and the sample label signal, and the logarithmic variance, includes: Determine the enhancement result of the sample speech signal and the differences in the real part of the complex spectrum, the imaginary part of the complex spectrum, and the amplitude spectrum between the sample label signals; The differences in the real part of the complex spectrum are weighted according to a first natural exponential function, and the differences in the imaginary part of the complex spectrum are weighted according to a second natural exponential function. A first loss is determined based on the sum of the logarithmic variance of the real part of the complex spectrum, the logarithmic variance of the imaginary part of the complex spectrum, the weighted differences in the real part of the complex spectrum, and the weighted differences in the imaginary part of the complex spectrum. The natural exponential function uses the logarithmic variance of the real part of the complex spectrum as its exponent, and the second natural exponential function uses the logarithmic variance of the imaginary part of the complex spectrum as its exponent. The amplitude spectrum difference is weighted according to the third natural exponential function, and the second loss is determined according to the sum of the logarithmic variance of the amplitude spectrum and the weighted amplitude spectrum difference, wherein the third natural exponential function uses the logarithmic variance of the amplitude spectrum as the exponent; The target loss for the first training phase is obtained by weighted summation of the first loss and the second loss.

8. The speech enhancement method according to claim 7, wherein the processing block includes a gated recurrent unit, the gated recurrent unit is configured with binary gates, and the step of weighted summing of the first loss and the second loss to obtain the target loss of the first training stage includes: The signal-to-noise ratio loss is determined based on the enhancement result of the sample speech signal and the sample label signal, and the update rate loss is determined based on the difference between the update rate of the binary gate and the preset update rate threshold. The first uncertainty loss is obtained by weighted summing of the first loss and the second loss. The target loss of the first training phase is obtained by weighted summing of the first uncertainty loss, the signal-to-noise ratio loss, and the update rate loss.

9. The speech enhancement method according to claim 6, wherein the logarithmic variance includes the logarithmic variance of the real part of the complex spectrum, the logarithmic variance of the imaginary part of the complex spectrum, and the logarithmic variance of the amplitude spectrum, and the step of determining the target loss of the second training stage based on the difference between the enhancement result of the sample speech signal and the sample label signal, and the logarithmic variance, includes: Determine the enhancement result of the sample speech signal and the differences in the real part of the complex spectrum, the imaginary part of the complex spectrum, and the amplitude spectrum between the sample label signals; The differences in the real part of the complex spectrum are weighted according to the logarithmic variance of the real part of the complex spectrum, and the differences in the imaginary part of the complex spectrum are weighted according to the logarithmic variance of the imaginary part of the complex spectrum. The third loss is determined by the sum of the weighted differences in the real part of the complex spectrum and the weighted differences in the imaginary part of the complex spectrum. The amplitude spectrum difference is weighted according to the logarithmic variance of the amplitude spectrum, and the weighted amplitude spectrum difference is determined as the fourth loss. The target loss for the second training phase is obtained by weighted summation of the third loss and the fourth loss.

10. The speech enhancement method according to claim 9, wherein the processing block includes a gated recurrent unit, the gated recurrent unit is configured with binary gates, and the step of weighted summing of the third loss and the fourth loss to obtain the target loss of the second training stage includes: The signal-to-noise ratio loss is determined based on the enhancement result of the sample speech signal and the sample label signal, and the update rate loss is determined based on the difference between the update rate of the binary gate and the preset update rate threshold. The second uncertainty loss is obtained by weighted summing of the third loss and the fourth loss. The target loss of the second training phase is obtained by weighted summing of the second uncertainty loss, the signal-to-noise ratio loss, and the update rate loss.

11. The speech enhancement method according to any one of claims 1-10, wherein performing band mixing mapping on the first mapping feature to obtain the second mapping feature includes: The first mapping feature is segmented along the channel dimension to obtain the first intra-band relation sub-feature and the second intra-band relation sub-feature; The first intra-band relation sub-feature is normalized, and the normalized first intra-band relation sub-feature is convolved along the time axis in the sub-band dimension to obtain the first convolutional feature; Determine the first product of the first convolutional feature and the second intra-band relational feature, and obtain the second mapping feature based on the sum of the first product and the second intra-band relational feature.

12. The speech enhancement method according to claim 11, wherein segmenting the first mapping feature along the channel dimension to obtain a first intra-band relational sub-feature and a second intra-band relational sub-feature includes: The first mapping feature is convolved along the time axis in the channel dimension to obtain the second convolutional feature; The second convolutional feature is normalized, and the normalized second convolutional feature is segmented along the channel dimension to obtain the first intra-band relational sub-feature and the second intra-band relational sub-feature.

13. The speech enhancement method according to claim 11 or 12, wherein obtaining the second mapping feature based on the sum of the first product and the second in-band relational feature comprises: The second mapping feature is obtained by convolving the first product and the sum of the second intraband relational features along the time axis in the channel dimension.

14. The speech enhancement method according to any one of claims 1-13, wherein the step of band compression of the second mapping feature to obtain the output of the first processing block comprises: The second mapping feature is normalized, and the normalized second mapping feature is convolved in parallel to obtain the third and fourth convolutional features with compression. The fourth convolutional feature is non-linearly activated to obtain attention weights; The third convolutional features are weighted based on the attention weights to obtain the output of the first processing block.

15. The speech enhancement method according to any one of claims 1-14, wherein decoding the encoding result of the encoder based on the decoder to obtain target signal features includes: The decoder is invoked to decode based on the encoder's encoding result, and the decoding result is segmented along the frequency dimension to obtain multiple decoded sub-features; The channel dimensions of each of the decoded sub-features are restored to be consistent with the channel dimensions of the speech signal to be processed, and the multiple restored decoded sub-features are merged along the frequency dimension to obtain the target signal feature. Alternatively, the multiple decoded sub-features are merged along the frequency dimension, and the channel dimensions of the merged decoded feature are restored to be consistent with the channel dimensions of the speech signal to be processed to obtain the target signal feature.

16. The speech enhancement method according to any one of claims 1-15, wherein the step of restoring the enhanced speech signal from the speech signal to be processed based on the target signal features comprises: Determine the second product between the target signal features and the speech signal to be processed; The second product is filtered to recover the enhanced speech signal from the speech signal to be processed.

17. A voice enhancement device, the device being deployed on an electronic device, comprising: An input module is used to acquire a speech signal to be processed, extract features from the speech signal to be processed to obtain initial signal features, and input the initial signal features into a speech enhancement network. The speech enhancement network includes an encoder and a decoder connected to each other, and each encoder and decoder is provided with multiple processing blocks cascaded in sequence. The encoding module is configured to perform frame mixing mapping on the initial signal features based on the first processing block in the encoder to obtain a first mapping feature, perform band mixing mapping on the first mapping feature to obtain a second mapping feature, perform band compression on the second mapping feature to obtain the output of the first processing block, and input the output of the first processing block to the next processing block until the encoding result of the encoder is obtained; A decoding module is used to decode the encoding result of the encoder based on the decoder to obtain the target signal features; The restoration module is used to restore the enhanced speech signal from the speech signal to be processed based on the target signal features.

18. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the speech enhancement method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech enhancement method according to any one of claims 1 to 16.

20. A computer program product comprising a computer program that, when executed by a processor, implements the speech enhancement method according to any one of claims 1 to 16.

Images