Neural network training method, speech denoising method, device and storage medium

Abstract

A neural network training method, a speech denoising method, a computer program product, a device and a storage medium. During the training of a neural network, bit-width truncation processing can be performed on network parameters of the neural network; and the accuracy of denoising parameters determined using the network parameters subjected to bit-width truncation processing, and the impact of the bit-width truncation processing on the network parameters themselves are used as constraints in the neural network training process, such that the impact of the bit-width truncation processing on data precision can be minimized, thereby obtaining, by means of training, a lightweight speech denoising neural network having good performance, thus allowing the lightweight speech denoising neural network to be deployed on a speech acquisition device having low performance.

Description

Neural network training methods, speech denoising methods, equipment and storage media Technical Field

[0001] This application relates to the field of audio processing technology, and more specifically, to a neural network training method, a speech noise reduction method, a computer program product, a device, and a storage medium. Background Technology

[0002] In some voice acquisition scenarios, the voice data collected by voice acquisition devices may contain a lot of environmental noise, resulting in very poor sound reception. Therefore, noise reduction processing is necessary. Traditional voice noise reduction techniques either achieve noise reduction through hardware circuit improvements or through signal processing. However, hardware circuit improvements can only suppress noise within a specific frequency range, while signal processing can only suppress steady-state noise and cannot suppress transient or sudden noise (such as car horns).

[0003] With the development of AI technology, this technology has been gradually applied to the field of speech denoising. Currently, speech denoising using pre-trained neural networks can achieve very good denoising results. This method can suppress various types of noise (such as steady-state, transient, and speech-related noise) and preserve the speech information to a great extent. However, due to the complexity of the neural network structure and the large number of network parameters, and the higher requirements for audio streams (typically 48kHz sampling rate and 24-bit bit width) in fields such as recording where high speech quality is required, the amount of data involved in denoising speech data using neural networks is very large, requiring a large amount of memory and computation, and placing high demands on the hardware performance of the processing equipment. This makes it difficult to apply neural network-based speech denoising methods to some low-performance speech acquisition devices.

[0004] Summary of the Invention

[0005] In view of this, this application provides a neural network training method, a speech denoising method, a computer program product, an apparatus, and a storage medium.

[0006] According to a first aspect of this application, a method for training a speech denoising neural network is provided, the method comprising:

[0007] Noisy speech data is acquired, and features are extracted from the noisy speech data to obtain a first target speech feature, wherein the noisy speech data is obtained by mixing clean speech data with noise;

[0008] The current network parameters of the preset neural network are truncated to a bit width, and the predicted noise reduction parameters for the noisy speech data are determined based on the truncated network parameters and the first target speech features.

[0009] The first loss is determined based on the difference between the predicted noise reduction parameters and the actual noise reduction parameters; wherein the actual noise reduction parameters are determined based on the noisy speech data and the clean speech data;

[0010] The second loss is determined based on the difference between the network parameters after bit-width truncation and the current network parameters;

[0011] A target loss is determined based on the first loss and the second loss, and the current network parameters are adjusted based on the target loss;

[0012] Repeat the above steps until the target loss meets the preset conditions, and perform the bit width truncation process on the current network parameters obtained in the current round of adjustment, which are used as the network parameters of the trained speech denoising neural network.

[0013] According to a second aspect of this application, a speech noise reduction method is provided, the method comprising:

[0014] Acquire the speech data to be denoised;

[0015] The speech data to be denoised is subjected to feature extraction to obtain the second target speech features;

[0016] The second target speech feature is input into a speech denoising neural network trained using the method mentioned in the first aspect above, so as to predict the denoising parameters for the speech data to be denoised through the speech denoising neural network.

[0017] The noise reduction parameters are used to perform noise reduction processing on the speech data to be denoised.

[0018] According to a third aspect of this application, an electronic device is provided, the electronic device including a processor, a memory, and a computer program stored in the memory that is executable by the processor, wherein the processor executes the computer program to implement the methods mentioned in the first and / or second aspects above.

[0019] According to a fourth aspect of this application, a computer program product is provided, the computer program product comprising a computer program that, when executed, implements the methods mentioned in the first and / or second aspects above.

[0020] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed, implements the methods mentioned in the first and / or second aspects above.

[0021] By applying the solution provided in this application, in order to deploy the speech denoising neural network in low-performance speech acquisition devices, the network parameters of the neural network can be truncated during training. The accuracy of the denoising parameters determined by the truncated network parameters, as well as the impact of the truncated parameters on the network parameters themselves, can be used as constraints in the neural network training process. This minimizes the impact of the truncated parameters on data accuracy, resulting in a lightweight and high-performance speech denoising neural network that can be deployed in low-performance speech acquisition devices.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 is a schematic diagram of a neural network training method according to an embodiment of this application.

[0025] Figure 2 is a flowchart of a neural network training method according to an embodiment of this application.

[0026] Figure 3 is a schematic diagram of extracting a first target speech feature from noisy speech data according to an embodiment of this application.

[0027] Figure 4 is a schematic diagram of the structure of a speech denoising neural network according to an embodiment of this application.

[0028] Figure 5 is a schematic diagram of the logical structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In some voice acquisition scenarios, the voice data collected by voice acquisition devices may contain a lot of environmental noise, resulting in very poor sound reception. Therefore, noise reduction processing is necessary. For example, take a wireless lavalier microphone. Because it uses an omnidirectional microphone, it offers virtually no additional attenuation of sound from any direction. Consequently, in noisy environments, the user's voice is easily disrupted by ambient noise, requiring voice noise reduction processing before it can be used.

[0031] Traditional speech noise reduction techniques either achieve noise reduction by improving hardware circuits or by using signal processing methods. However, noise reduction through hardware circuits can only suppress noise within a specific frequency range and cannot cover all types of noise. On the other hand, noise reduction through signal processing can only suppress steady-state noise and cannot suppress some transient or sudden noises (such as car horns).

[0032] With the development of AI technology, this technology has been gradually applied to the field of speech denoising. Currently, speech denoising using pre-trained neural networks can achieve very good denoising results. This method can suppress various types of noise (such as steady-state, transient, and speech-related noise) and can preserve the information of the speech itself to a great extent with minimal distortion. However, due to the complexity of the neural network structure, the large number of network parameters, and the large amount of data, and because fields such as recording require high speech quality, in addition to high intelligibility, high fidelity (high fidelity, no stuttering, etc.) is also required, the requirements for the audio stream are also higher (typically a sampling rate of 48kHz and a bit width of 24bit). As a result, the amount of data involved in the process of using neural networks to denoise speech data is very large, requiring a large amount of memory and computation, and placing high demands on the hardware performance of the processing equipment. This makes it difficult to apply neural network-based speech denoising methods to some low-performance speech acquisition devices (such as wireless lavalier microphones).

[0033] Given the complexity of neural networks and the large number of network parameters, these parameters often require significant memory and computation. To enable this neural network-based speech denoising method to be applied to embedded speech acquisition devices with lower performance, this application proposes truncation of the network parameters to reduce the data volume, thereby saving memory and reducing computation. However, truncation inevitably sacrifices data precision. To minimize the loss of precision during truncation, this application uses the impact of truncation on the network parameters themselves, as well as the accuracy of the denoising parameters determined using the truncation-processed network parameters, as constraints during neural network training. This ensures that the effective data in the trained network parameters is concentrated in the first few bits, thus minimizing the loss of precision after truncation. For example, with 32-bit network parameters, the network parameters can be continuously adjusted during training to concentrate the effective data in the first 16 bits, minimizing the impact on the network parameters even after truncating the last 16 bits. This reduces the memory usage and computational load of neural network parameters, allowing neural networks to be deployed on devices with lower performance while ensuring the accuracy of speech denoising results.

[0034] Based on the above inventive concept, this application provides a training method for a speech denoising neural network, which can be implemented by various devices such as mobile phones, computers, cloud servers, and server clusters.

[0035] The training method is described below with reference to Figures 1 and 2. Figure 1 is a flowchart of the training method, and Figure 2 is a schematic diagram of the training method. As shown in Figure 1, the training method includes the following steps:

[0036] S102, acquire noisy speech data, and extract features from the noisy speech data to obtain the first target speech features; wherein, the noisy speech data is obtained by mixing clean speech data with noise;

[0037] Before training the speech denoising neural network, a large amount of sample data can be collected for training. Each set of sample data includes clean speech data and noisy speech data obtained by mixing the clean speech data with environmental noise. Since the clean speech data corresponding to the noisy speech data is known, appropriate denoising parameters can be determined based on both, hereinafter referred to as the actual denoising parameters, which serve as labels for training the neural network.

[0038] In step S102, noisy speech data can be acquired first, and features can be extracted from the noisy speech data according to some pre-set feature extraction strategies to obtain speech features, hereinafter referred to as the first target speech features. The extracted speech features can be various types of speech features such as logarithmic power spectrum, Mel-frequency cepstral coefficients (MFCC), and Gammatone coefficients; this embodiment does not impose any limitations. The method of feature extraction can also include various approaches. For example, the noisy speech data can be divided into multiple sub-bands, and then features can be extracted from the sub-bands; or features can be extracted directly from the noisy speech data. This embodiment also does not impose any limitations.

[0039] The first target speech features can include multiple types of features, or multiple features, which can be represented by vectors or matrices.

[0040] S104, perform bit-width truncation processing on the current network parameters of the preset neural network, and determine the predicted noise reduction parameters for denoising the noisy speech data based on the bit-width truncation network parameters and the first target speech features.

[0041] In step S104, to reduce the amount of data, the current network parameters of the preset neural network can be truncated. Truncation refers to removing the last few bits of the network parameters. For example, assuming the original network parameters are 32 bits, the last 16 bits can be removed, retaining only the first 16 bits, thus significantly reducing the amount of data. Of course, the number of bits truncated can be flexibly set based on the required data precision; for example, it can be 1, 2, 4, 8, 16, or 24 bits, etc., and this embodiment does not impose any limitations.

[0042] Then, based on the network parameters after bit-width truncation and the first target speech features, denoising parameters for denoising the noisy speech data can be determined, hereinafter referred to as predicted denoising parameters. These denoising parameters can be filter coefficients, based on which a filter suitable for denoising the speech data to be denoised can be designed. Alternatively, denoising parameters can be gains, which can be used to increase the intensity of the effective speech signal, making it more prominent in background noise. The type of denoising parameter can be flexibly set based on actual needs, and this application embodiment does not impose any limitations.

[0043] S106, determine a first loss based on the difference between the predicted noise reduction parameters and the actual noise reduction parameters; wherein, the actual noise reduction parameters are determined based on the noisy speech data and the clean speech data;

[0044] In step S106, after determining the predicted denoising parameters, the actual denoising parameters corresponding to the noisy speech data can be obtained. These actual denoising parameters can be determined based on the noisy speech data and its corresponding clean speech data. Then, a first loss can be determined based on the difference between the predicted and actual denoising parameters. The difference between the predicted and actual denoising parameters can be characterized by various parameters that measure this difference. For example, the mean square error between the predicted and actual denoising parameters can be used as the first loss. Alternatively, the absolute error between the two can be directly used as the first loss. The first loss is used to measure the accuracy of the denoising parameters determined based on the network parameters after bit-width truncation. Using the first loss as a constraint to train the neural network ensures the accuracy of the neural network's prediction results.

[0045] S108, determine the second loss based on the difference between the network parameters after bit width truncation and the current network parameters;

[0046] In step S108, to minimize the impact of bit-width truncation on the neural network prediction results, a second loss can be determined based on the difference between the network parameters before and after bit-width truncation. The second loss measures the difference in network parameters before and after bit-width truncation. Using the second loss as another constraint in the neural network training process minimizes the impact of bit-width truncation on network parameters, meaning that the effective data of the adjusted network parameters is concentrated in the first few bits, i.e., the truncated bit width is as close to 0 as possible.

[0047] S110, determine the target loss based on the first loss and the second loss, and adjust the current network parameters based on the target loss;

[0048] In step S110, after determining the first loss and the second loss, a target loss can be determined based on both. For example, the sum of the two losses can be used as the target loss, or each loss can be assigned a certain weight and then added together to obtain the target loss. Then, the current network parameters of the neural network can be adjusted based on the target loss to obtain new current network parameters. For example, the original 32-bit network parameters of the neural network can be adjusted to obtain new current network parameters for the next round of training.

[0049] S112, repeat the above steps until the target loss meets the preset conditions, and perform the bit width truncation process on the current network parameters obtained in the current round of adjustment, as the network parameters of the trained speech denoising neural network.

[0050] Steps S102-S110 can be repeated until the target loss meets certain conditions, such as convergence or a preset threshold. At this point, the network parameters obtained from the current adjustment can be truncated to become the network parameters of the trained speech denoising model. For example, after the target loss converges, 32-bit network parameters can be obtained. These 32-bit parameters can then be truncated to 16-bit parameters and used as the network parameters of the trained speech denoising neural network, thus obtaining a lightweight speech denoising neural network.

[0051] In this embodiment of the application, when training the neural network, the network parameters of the neural network can be truncated. The impact of the truncation on the network parameters themselves, as well as the accuracy of the denoising parameters determined by the truncation network parameters, are used as constraints in the neural network training process. This minimizes the impact of the truncation on data accuracy and trains a lightweight and high-performance speech denoising neural network.

[0052] When extracting speech features from speech data, the data is typically divided into subbands, creating multiple frequency ranges. The speech features of different frequency ranges (i.e., different subbands) are then analyzed. In previous techniques, fixed subband division methods based on acoustic experiments were commonly used. This fixed method results in relatively limited subband types, failing to comprehensively reflect speech features and lacking flexibility. Furthermore, the optimal subband division method differs for different types of speech data. Therefore, this application does not employ a fixed subband division method. Instead, it sets multiple feasible subband division strategies and selects the optimal strategy from these to divide the speech data, resulting in subbands that more accurately reflect speech features. Specifically, a subband division strategy pool can be pre-set, containing multiple strategies. To ensure the effectiveness of these strategies, they must all meet certain conditions. Specifically, the multiple subbands obtained by dividing the speech data according to these strategies must simultaneously satisfy the following conditions:

[0053] (1) The low-frequency part of any one of the multiple sub-bands obtained by division is denser than the high-frequency part.

[0054] (2) If adjacent sub-bands overlap in the multiple sub-bands obtained by division, the overlapping frequency ranges are the same in proportion to the respective frequency ranges of the two adjacent sub-bands.

[0055] In other words, when dividing subbands, adjacent subbands may or may not overlap. If adjacent subbands overlap, in order to ensure that the overlapping part has the same impact on the two subbands, the proportion of the overlapping frequency range in the respective frequency ranges of the two adjacent subbands can be kept consistent.

[0056] (3) The total frequency range obtained by superimposing the frequency ranges of the multiple sub-bands covers the frequency range of the noisy speech data. That is, these sub-bands can cover the entire frequency range of the noisy speech data, avoiding the omission of some frequency ranges.

[0057] After setting up various subband partitioning strategies in the subband partitioning strategy pool, during neural network training, one strategy can be selected to partition the noisy speech data into multiple subbands. Then, feature extraction can be performed on each subband to obtain its speech features. Finally, a first target speech feature is derived based on these multiple subband speech features. For example, the first target speech feature can be a feature matrix composed of the speech features from these multiple subbands, or it can be speech features obtained by concatenating, merging, or otherwise processing the speech features from these multiple subbands.

[0058] In some embodiments, to select the optimal subband partitioning strategy from the subband partitioning strategy pool for use in the neural network inference phase, during the neural network training phase, each subband partitioning strategy in the pool can be selected sequentially to partition the noisy speech data into subbands to extract first target speech features for training the neural network. Furthermore, for each subband partitioning strategy in the pool, the convergence loss when using that strategy to partition the noisy speech data for training the speech denoising neural network can be recorded. The convergence loss refers to the loss when the neural network converges. Then, the subband partitioning strategy with the minimum convergence loss can be used as the optimal partitioning strategy for the inference phase of the speech denoising neural network.

[0059] Considering the continuity of speech, neighboring sub-bands often have some correlation. To obtain more comprehensive speech features, in some embodiments, as shown in Figure 3, when extracting features from noisy speech data to obtain the first target speech feature, the noisy speech data can be divided into sub-bands to obtain multiple sub-bands (sub-band 1 to sub-band n in the figure). Then, feature extraction is performed on each of these sub-bands to obtain the speech features of each sub-band (speech feature 1 to speech feature n in the figure). For each sub-band, the speech features of that sub-band can be concatenated with the speech features of its associated sub-bands to obtain the concatenated features as the first target speech feature. Here, the associated sub-bands are the neighboring sub-bands of that sub-band. By concatenating the features of neighboring sub-bands with certain correlations, speech features reflecting the relationship between neighboring sub-bands can be reconstructed, making the finally extracted speech features more diverse.

[0060] In some embodiments, when determining the second loss based on the difference between the network parameters after bit width truncation and the network parameters before bit width truncation, the mean square error between the network parameters after bit width truncation and the network parameters before bit width truncation can be determined, and the second loss can be determined based on the mean square error.

[0061] Considering that models trained with sample data that better reflects real-world application scenarios perform better during the inference phase of neural network training, it's important to understand that in the past, when collecting sample data for training speech denoising neural networks, fixed speech acquisition devices were typically used to collect noisy speech data. This sample data fails to reflect the characteristics of speech data collected by the actual speech acquisition device requiring denoising. For example, taking a wireless lavalier microphone as the speech acquisition device requiring denoising, factors such as the microphone's hardware structure, materials, and the distance between the microphone and the mouth all affect the collected user speech and environmental noise. If other devices are used to collect sample data, such as noisy speech data including car horn noise, there will still be differences compared to the noisy speech data collected by the wireless microphone in its actual working environment. To improve the performance of the trained neural network during the inference phase, if the speech denoising neural network is used to denoise the speech data collected by the target speech acquisition device, the noise in the noisy speech data collected for training the neural network can be obtained by simulating the actual working environment of the target speech acquisition device. For example, taking a wireless lavalier microphone as an example, it is possible to simulate the actual working process of a wireless lavalier microphone and collect various types of noise that may exist during this working process, so as to simulate the real working situation of a wireless lavalier microphone.

[0062] In some embodiments, as shown in Figure 4, the speech denoising neural network includes a noise estimation subnetwork and a denoising parameter determination subnetwork. After performing a series of preprocessing steps on the noisy speech data to obtain a first target speech feature, the noise estimation subnetwork can determine the noise features in the noisy speech data based on the first target speech feature. The denoising parameter determination subnetwork can determine the denoising parameters based on the noise features. Both the noise estimation subnetwork and the denoising parameter determination subnetwork can employ deep neural networks (DNNs), i.e., noise estimation and denoising parameter determination are performed separately using a two-stage DNN network, thereby improving accuracy.

[0063] Furthermore, this application also provides a speech denoising method, which can use a speech denoising neural network trained by the neural network training method provided in any of the above embodiments to perform denoising processing on the speech data to be denoised. Since the network parameters of the speech denoising neural network trained by the neural network training method provided in any of the above embodiments occupy a small bit width, i.e., the data volume is small, it can reduce the memory usage of the device and the computational load during the speech data denoising process, allowing the speech denoising neural network to be deployed in some low-performance embedded speech acquisition devices (such as wireless lavalier microphones). Moreover, since the network parameters are optimized during the training process of the speech denoising neural network, the data accuracy loss caused by bit width truncation of the network parameters is minimized, so that the prediction results of the trained lightweight speech denoising neural network still have high accuracy.

[0064] The speech denoising method of this application embodiment can be adapted to various devices that have deployed the above-mentioned speech denoising neural network. For example, the speech denoising neural network can be deployed in some low-performance speech acquisition devices, and the method can be executed by these speech acquisition devices. Alternatively, the speech denoising neural network can also be deployed in servers, server clusters, or other high-performance devices, and the method can be executed by these high-performance devices.

[0065] Specifically, the process begins by acquiring the speech data to be denoised. This speech data can be collected by a speech acquisition device. Then, feature extraction is performed on the speech data to obtain speech features, hereinafter referred to as the second target speech features. These second target speech features can then be input into the speech denoising neural network trained using the training method provided in any of the above embodiments. The speech denoising neural network predicts denoising parameters for processing the speech data to be denoised, and these denoising parameters can then be used to perform denoising processing on the speech data. For example, the denoising parameters can be filter coefficients, and based on these filter coefficients, a filter suitable for denoising the speech data to be denoised can be designed.

[0066] In some embodiments, the speech denoising neural network can be deployed in an embedded platform. To improve the processing speed during speech denoising, an embedded platform including multiple computing units (e.g., a Tensilica HiFi DSP) can be selected. This allows multiple computing units to perform parallel calculations on the speech data to be denoised and the network parameters during the prediction of denoising parameters, thereby obtaining the denoising parameters. Furthermore, to achieve parallel computing, the speech data to be denoised and the network parameters of the speech denoising neural network can be stored according to a preset storage format that meets the parallel computing requirements of the embedded platform.

[0067] For example, it can be stored in the form of a high-dimensional tensor, so that data that needs to be used at the same time can be stored continuously. This makes it convenient for the embedded platform to load multiple data that need to be used in the calculation process in parallel when using the speech data to be denoised and network parameters for calculation, so that the multiple computing units can process these multiple data in parallel.

[0068] Furthermore, considering that accessing and storing computational data (including the speech data to be denoised and the network parameters) is very time-consuming during the prediction of denoising parameters using a speech denoising neural network, in order to achieve fast data storage and improve the processing efficiency of the neural network, this embodiment can store the speech data to be denoised and the network parameters of the speech denoising neural network in the form of a high-dimensional tensor. Then, a four-dimensional array for fast data retrieval from memory can be pre-designed based on the data storage structure. During the calculation of the speech data to be denoised and the network parameters, the data required for the calculation process can be retrieved from memory according to this four-dimensional array. The format of the four-dimensional array is as follows:

[0069] [Pos in Pos out Dim in Dim out ]

[0070] Among them, Pos in Pos represents the storage address of the data to be read from memory. out Dim represents the storage address of the data to be stored in memory. in Dim represents the dimension of the data to be read from memory. out The dimensions of the data to be stored in memory.

[0071] For example, in some embodiments, the embedded platform may include an input pointer orientation module, a logic operation module, and an output pointer orientation module. The input pointer orientation module can be based on Pos in a four-dimensional array. in and Dim inThe system determines the multiple data items to be read from memory at any given time. Then, based on the parallel bit width of the embedded platform (e.g., 128 bits for a Tensilica HiFi DSP), it loads this data from memory and sends it to the logic processing module. Since the data is stored contiguously, the root Pos... in and Dim in It also allows us to know the address of the next piece of data to be read from memory, thus enabling continuous data retrieval from memory. For example, assuming the dimension of the data to be read is 8, Pos... in Offset by 8 addresses gives the address of the next data to be read. Similarly, offset by 8 bits from the address of the next data to be read gives the address of the data to be read after that. If this happens repeatedly, data can be read continuously.

[0072] The logic operation module can perform the same operations on multiple loaded data in parallel to obtain the calculation results, which are the data to be stored in memory. The output pointer orientation module can be based on Pos. out 〖and Dim out Determine the memory address of the data to be stored in memory (i.e., the calculation result), and store it in memory. Based on Dim... out The dimensions of the data to be stored in memory can be determined based on Pos. out and Dim out The storage address of the next data to be stored in memory can be determined, thus allowing for continuous data storage into memory. For example, assuming the data to be stored has 8 dimensions, Pos... out Offset by 8 addresses gives the address of the next data to be stored. Similarly, offset by 8 bits from the address of the next data to be stored gives the address of the data to be stored after that. If this happens repeatedly, data can be stored continuously.

[0073] In this embodiment, the speech data to be denoised and the network parameters are stored in a high-dimensional tensor format, which facilitates the embedded platform to load data in parallel during the calculation process, thereby achieving parallel computing and improving processing speed. Simultaneously, by designing a four-dimensional array for storing and retrieving data based on the data storage structure, the data access efficiency during the calculation process can be accelerated, greatly improving the processing efficiency of the speech denoising neural network when denoising the speech data to be denoised.

[0074] In some embodiments, when extracting features from the speech data to be denoised to obtain the second target speech features, the speech data to be denoised can be divided into multiple sub-bands based on a pre-determined target sub-band partitioning strategy. Then, features are extracted from each of these sub-bands, and the second target speech features are obtained based on the speech features extracted from each sub-band. The target sub-band partitioning strategy can be determined during the training phase of the speech denoising neural network. To extract speech features that more accurately and comprehensively reflect the characteristics of the speech data to be denoised, an optimal sub-band partitioning strategy can be determined during the training phase of the speech denoising neural network to partition the sub-bands and determine the aforementioned second target speech features. Specifically, during the training process of the speech denoising neural network, the noisy speech data used to train the speech denoising neural network can be partitioned into sub-bands according to a variety of pre-defined sub-band partitioning strategies. Speech features are extracted from the partitioned sub-bands for training the speech denoising neural network. Simultaneously, the convergence loss of the speech denoising neural network corresponding to each sub-band partitioning strategy can be recorded, and the sub-band partitioning strategy with the smallest convergence loss is used as the target sub-band partitioning strategy for the inference process.

[0075] Considering the continuity of speech, adjacent sub-bands often store certain correlations. To obtain more comprehensive speech features, in some embodiments, when extracting features from the speech data to be denoised to obtain the second target speech features, the speech data to be denoised can be divided into sub-bands to obtain multiple sub-bands. Then, feature extraction is performed on each of these multiple sub-bands to obtain the speech features of each sub-band. For each sub-band, the speech features of that sub-band can be concatenated with the speech features of its associated sub-bands to obtain the second target speech features, where the associated sub-bands are the neighboring sub-bands of that sub-band.

[0076] The solutions in the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space limitations, they will not be listed one by one here.

[0077] Furthermore, this application also provides a computer program product, which includes a computer program that, when executed, implements the methods described in any of the above embodiments.

[0078] Furthermore, this application embodiment also provides an electronic device. As shown in FIG5, which is a hardware structure diagram of an electronic device 50 according to an embodiment of this application, in addition to the processor 51 and memory 52 shown in FIG5, the device may also include other hardware, such as a forwarding chip responsible for processing messages; from a hardware structure perspective, the device may also be a distributed device, possibly including multiple interface cards, so as to expand message processing at the hardware level. The memory 52 stores computer instructions, and when the processor 51 executes the computer instructions, it implements the image processing method mentioned in any of the above embodiments.

[0079] Accordingly, this application also provides a computer storage medium storing a program that, when executed by a processor, implements the method in any of the above embodiments.

[0080] The embodiments of this application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0081] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0082] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0083] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0084] The methods and apparatus provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this application should not be construed as a limitation of this application.

Claims

1. A method for training a speech denoising neural network, characterized in that, The method includes: Noisy speech data is acquired, and features are extracted from the noisy speech data to obtain a first target speech feature, wherein the noisy speech data is obtained by mixing clean speech data with noise; The current network parameters of the preset neural network are truncated in bit width, and the predicted noise reduction parameters for the noisy speech data are determined based on the network parameters after bit width truncation and the first target speech features. The first loss is determined based on the difference between the predicted noise reduction parameters and the actual noise reduction parameters; wherein the actual noise reduction parameters are determined based on the noisy speech data and the clean speech data; The second loss is determined based on the difference between the network parameters after bit-width truncation and the current network parameters; A target loss is determined based on the first loss and the second loss, and the current network parameters are adjusted based on the target loss; Repeat the above steps until the target loss meets the preset conditions, and perform the bit width truncation process on the current network parameters obtained in the current round of adjustment, which are used as the network parameters of the trained speech denoising neural network.

2. The method of claim 1, wherein, Feature extraction is performed on the noisy speech data to obtain the first target speech features, including: Select a sub-band partitioning strategy from the sub-band partitioning strategy pool to perform sub-band partitioning processing on the noisy speech data to obtain multiple sub-bands; Feature extraction is performed on each of the plurality of sub-bands, and the first target speech feature is obtained based on the speech features extracted from each sub-band; Among them, the multiple subbands obtained by partitioning according to any partitioning strategy in the subband partitioning strategy pool satisfy the following conditions: The low-frequency portion of any one of the plurality of subbands is denser than the high-frequency portion; If adjacent sub-bands overlap, the overlapping frequency range accounts for the same proportion of the frequency range of each of the two adjacent sub-bands. The total frequency range obtained by superimposing the frequency ranges of the multiple sub-bands covers the frequency range of the noisy speech data.

3. The method of claim 2, wherein, The method further includes: For each subband partitioning strategy in the subband partitioning strategy pool, record the convergence loss when using the subband partitioning strategy to perform subband partitioning processing on the noisy speech data to train the speech denoising neural network. The subband partitioning strategy that minimizes the convergence loss is used as the optimal partitioning strategy in the inference phase of the speech denoising neural network.

4. The method of claim 1, wherein, Feature extraction is performed on the noisy speech data to obtain the first target speech features, including: The noisy speech data is divided into sub-bands to obtain multiple sub-bands; Feature extraction is performed on each of the multiple sub-bands to obtain the speech features of each sub-band; For each sub-band, the speech features of the sub-band are concatenated with the speech features of the associated sub-band, and the concatenated speech features are used as the first target speech features, wherein the associated sub-band is the neighboring sub-band of the sub-band.

5. The method of claim 1, wherein, The second loss is determined based on the difference between the network parameters after bit-width truncation and the current network parameters, including: Determine the mean square error of the network parameters after bit-width truncation and the current network parameters, based on... The mean square error determines the second loss.

6. The method of claim 1, wherein, The speech denoising neural network is used to denoise the speech data collected by the target speech acquisition device. The noise in the noisy speech data is obtained by simulating the actual working scenario of the target speech acquisition device.

7. The method of claim 1, wherein, The speech denoising neural network includes a noise estimation subnetwork and a denoising parameter determination subnetwork. The noise estimation subnetwork is used to determine the noise features in the noisy speech data based on the first target speech features; The noise reduction coefficient determination subnetwork is used to determine the noise reduction parameters based on the characteristics of the noise.

8. A voice noise reduction method, characterized by, The method includes: Acquire the speech data to be denoised; Feature extraction is performed on the speech data to be denoised to obtain the second target speech features; The second target speech feature is input into a speech denoising neural network trained using the method described in any one of claims 1-7, so as to predict the denoising parameters for denoising the speech data to be denoised through the speech denoising neural network. The noise reduction process is performed on the speech data to be denoised based on the noise reduction parameters.

9. The method of claim 8, wherein, The speech denoising neural network is deployed in an embedded platform, which includes multiple computing units. The speech data to be denoised and the network parameters of the speech denoising neural network are stored in a preset storage format, wherein the preset storage format is a format that meets the parallel computing requirements of the embedded platform. The embedded platform predicts denoising parameters for denoising the speech data to be denoised using the speech denoising neural network, including: Obtain the speech data to be denoised and the network parameters of the speech denoising neural network; The multiple computing units are used in parallel to process the speech data to be denoised and the network parameters. The noise reduction parameters are obtained through calculation.

10. The method of claim 9, wherein, The speech data to be denoised and the network parameters of the speech denoising neural network are stored in the form of high-dimensional tensors. When the embedded platform uses the multiple computing units to perform parallel calculations on the speech data to be denoised and the network parameters, it includes: During the calculation of the speech data to be denoised and the network parameters, the data required for the calculation process is retrieved from the memory of the embedded platform according to a preset four-dimensional array. The format of the four-dimensional array is as follows: [Pos in ,Pos out ,Dim in ,Dim out ] wherein Pos in represents a storage address of data to be read from the memory, Pos out represents a storage address of data to be stored into the memory, Dim in represents a dimension of data to be read from the memory, Dim out represents a dimension of data to be stored into the memory.

11. The method of claim 10, wherein, The embedded platform includes an input pointer orientation module, a logic operation module, and an output pointer orientation module; The input pointer orientation module is used for determining a plurality of data to be read from the memory of the embedded platform according to the Pos in and Dim in in the four-dimensional array, reading the plurality of data based on the parallel bit width of the embedded platform, and sending to the logical operation module. The logic operation module is used to perform parallel calculations on the multiple data, obtain the calculation results, and send them to the output pointer orientation module; The output pointer orientation module is configured to determine a storage address of the calculation result in a memory of the embedded platform according to Pos out and Dim out , and store the calculation result into the memory based on the storage address.

12. The method of claim 8, wherein, Feature extraction is performed on the speech data to be denoised to obtain the second target speech features, including: The speech data to be denoised is divided into multiple sub-bands based on a predetermined target sub-band division strategy. Feature extraction is performed on each of the multiple sub-bands, and the second target speech feature is obtained based on the speech features extracted from each sub-band; The target sub-band partitioning strategy is determined based on the following method: During the training process of the speech denoising neural network, the noisy speech data is divided into sub-bands one by one according to a variety of preset sub-band partitioning strategies in order to train the speech denoising neural network. The convergence loss of the speech denoising neural network corresponding to each sub-band partitioning strategy is recorded, and the sub-band partitioning strategy with the smallest convergence loss is taken as the target sub-band partitioning strategy.

13. A computer program product, characterised in that, The computer program product includes a computer program that, when executed, implements the method according to any one of claims 1-7 or 8-12.

14. An electronic device, comprising: The electronic device includes a processor, a memory, and computer instructions stored in the memory, wherein the processor executes the computer instructions to implement the method according to any one of claims 1-7 or 8-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1-7 or 8-12.

Images