Multi-source direction-of-arrival estimation method and apparatus based on frequency-focusing spatial spectrum

Abstract

Provided in the present application are a multi-source direction-of-arrival (DOA) estimation method and apparatus based on a frequency-focusing spatial spectrum. The method comprises: firstly, using a preset simulation environment and a microphone array signal simulator to generate microphone array data and construct a training dataset; secondly, constructing a mask estimation network model by means of source masking training; then, performing sub-bandwidth division on an enhanced multi-source signal by means of band overlapping to acquire a weighted frequency-focused covariance matrix; consequently, constructing wideband frequency-focusing spatial spectrum features on the basis of a sub-bandwidth frequency-focused covariance matrix acquired by means of beamforming, and on the basis of the features, constructing a convolutional neural network; and finally, on the basis of a mask estimation network model and a multi-source DOA spatial spectrum estimation network model, performing decoding to obtain a multi-source DOA spatial spectrum, so as to complete multi-source DOA estimation. By means of the present application, the influence of nonlinear distortion of a neural network on DOA estimation is reduced on the basis of an intermediate representation of a wideband frequency-focusing spatial spectrum, and the stability of algorithm performance and the fault tolerance of DOA estimation are improved.

Description

A Method and Device for Estimating the Direction of Arrival of Multiple Sound Sources Based on Frequency Focusing Spatial Spectrum

[0001] This application claims priority to Chinese Patent Application No. 2024119769955, filed on December 30, 2024, entitled "Method and Apparatus for Estimating the Direction of Arrival of Multiple Sound Sources Based on Frequency Focusing Spatial Spectrum", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the fields of acoustic signal processing and sound source localization technology, and in particular to a method and apparatus for estimating the direction of arrival of multiple sound sources based on frequency focusing spatial spectrum. Background Technology

[0003] In recent years, microphone array-based online audiovisual conferencing scenarios have become a hot topic in acoustic research, especially in the field of multi-source DOA (Direction of Arrival) estimation. This technology, by acquiring the spatial information of speakers, supports subsequent tasks such as speech enhancement, multi-speaker speech separation, spatial audio coding, and sound field reconstruction, thereby improving the robustness of speech communication, recognition, and conference content transcription. However, in indoor conferencing environments, factors such as noise and reverberation severely impact the performance of DOA estimation, especially in multi-source and complex acoustic scenarios, where the random distribution of sound sources and signal overlap make DOA estimation even more difficult. Although some signal modeling-based methods, such as SRP, MUSIC, and PHAT, exist, these methods can stably perform DOA estimation under ideal conditions, but their performance deteriorates sharply under noise and reverberation interference. Therefore, improving the accuracy and robustness of DOA estimation in complex environments has become a current research challenge.

[0004] To address the impact of reverberation on DOA estimation performance, neural networks have been introduced into this field in recent years, leading to several supervised learning-based DOA estimation methods. These methods mainly fall into three categories: first, using neural networks to correct pre-extracted features, such as enhancing inter-channel phase features to improve estimation accuracy; second, classifying extracted feature labels using neural networks to improve robustness in reverberant environments, such as the CNN-based SPR-PHAT and MUSIC algorithms; and third, leveraging the sparsity of speaker time-frequency units to perform signal separation using neural networks, thereby reducing the impact of noise and reverberation. Although these neural network-based DOA estimation methods have demonstrated superior performance, they still face some challenges. For example, the network models have poor generalization ability in indoor environments, masking enhancement methods may introduce nonlinear distortion, and robustness in multi-source scenarios is poor.

[0005] Therefore, one or more methods are needed to solve the above problems.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] This application discloses a method and apparatus for estimating the direction of arrival of multiple sound sources based on frequency focusing spatial spectrum.

[0008] In a first aspect, this application discloses a method for estimating the direction of arrival of multiple sound sources based on frequency focusing spatial spectrum, the method comprising:

[0009] Using a preset simulation environment and a microphone array signal simulator, a microphone array dataset is generated from a clean speech corpus of a single sound source, and a training dataset is constructed based on the microphone array dataset.

[0010] A mask estimation network model is generated by training the training dataset on sound source masking; the loss function of the mask estimation network model is the mean square error of complex ideal scale masking of sound sources;

[0011] Based on the mask estimation network model, the enhanced multi-source signal is obtained, and the enhanced multi-source signal is divided into sub-bandwidths by frequency band overlay to obtain the weighted focusing covariance matrix of each sub-bandwidth.

[0012] Based on the sub-bandwidth weighted focusing covariance matrix, a broadband frequency focusing spatial spectrum feature is constructed from the sub-bandwidth frequency focusing covariance matrix obtained by beamforming, and a convolutional neural network model is constructed based on the broadband frequency focusing spatial spectrum feature. The convolutional neural network model is then used to estimate the DOA spatial spectrum of multiple sound sources.

[0013] Using the mask estimation network model and the multi-source DOA spatial spectrum estimation network model, the multi-source DOA spatial spectrum is decoded, and significant peak values ​​are extracted to complete the estimation of multi-source DOA.

[0014] In embodiments of this application, the microphone array dataset includes: clean microphone array speech signals, noisy and reverberant microphone array signals, and sound source DOA tags.

[0015] In embodiments of this application, constructing a training dataset based on the microphone array dataset includes:

[0016] Using the simulation environment, masking training data is constructed by performing short-time Fourier transform on the clean microphone array speech signal and the noisy and reverberant microphone array signal.

[0017] Based on the short-time Fourier transform structure of the noisy and reverberant microphone array signal, the active region of the clean microphone array speech is detected and labeled using the endpoint detection algorithm to construct sound source DOA spatial spectrum training data;

[0018] A training dataset is generated by integrating the masking training data and the sound source DOA spatial spectrum training data.

[0019] In embodiments of this application, the enhanced multi-source signal is divided into sub-bandwidths using a frequency band overlay method, including:

[0020] The broadband spatial spectrum is obtained by focusing the sub-bandwidth covariance matrix as an intermediate representation of the spatial orientation information of the sound source.

[0021] The signal bandwidth is divided into multiple sub-bandwidths by stacking sub-bandwidths, and the speech presence probability of each sub-bandwidth is obtained based on the speech presence probability algorithm. The signals of each sub-bandwidth are weighted to obtain the sub-bandwidth weighted focusing covariance matrix.

[0022] In embodiments of this application, the broadband frequency focusing spatial spectrum features are constructed from the sub-bandwidth frequency focusing covariance matrix obtained by beamforming, including:

[0023] The spatial spectrum of the sub-bandwidth is modeled based on the minimum variance distortionless response beamforming method, and the frequency focusing spatial spectrum characteristics of the sub-bandwidth based on beamforming are obtained.

[0024] In embodiments of this application, the convolutional neural network includes:

[0025] The convolutional neural network includes convolutional layers for extracting discriminative features and fully connected layers for mapping the DOA spatial spectrum of the sound source wave.

[0026] In the embodiments of this application, the multi-source DOA spatial spectrum is decoded using the mask estimation network model and the multi-source DOA spatial spectrum estimation network model, including:

[0027] Based on the decoding process of the mask estimation network model and the sound source DOA estimation network model, the multi-source wave DOA prediction results are generated by calculating the significant peak values ​​of the multi-source DOA spatial spectrum.

[0028] Based on the predicted DOA of the sound source waves, the DOA of multiple sound sources is estimated.

[0029] In embodiments of this application, constructing the sound source DOA spatial spectrum training data includes:

[0030] For the active region of the speech from the clean microphone array, a pseudo-spectral density (DOA) is generated according to the following formula as training labels for the convolutional neural network model to estimate the DOA spatial spectrum of multiple sound sources:

[0031] Where, φ SPS (θ i ) represents the constructed DOA spatial pseudospectrum, θ i θ is the ergodic angle of the spatial spectrum. q Let θ be the true DOA angle of the q-th sound source, and σ be a parameter controlling the pseudo-spectral peak width.

[0032] In the embodiments of this application, the mask estimation network model is a feedforward neural network including three hidden layers, wherein:

[0033] The input signal features of the network model are a complete set of features consisting of the signals from the current frame, the two past frames, and the two future frames; and

[0034] In the three hidden layers, each hidden layer has 512 neurons.

[0035] Secondly, this application discloses a multi-source direction-of-arrival estimation device based on frequency focusing spatial spectrum, the device comprising:

[0036] The data construction module is used to generate a microphone array dataset from a single-source clean speech corpus using a preset simulation environment and a microphone array signal simulator, and to construct a training dataset based on the microphone array dataset.

[0037] The model building module is used to generate a mask estimation network model by training the training dataset with sound source masking; the loss function of the mask estimation network model is the mean square error of the complex ideal scale masking of the sound source;

[0038] The covariance matrix acquisition module is used to acquire the enhanced multi-source signal based on the mask estimation network model, and to divide the enhanced multi-source signal into sub-bandwidths by frequency band overlay to acquire the weighted focusing covariance matrix of each sub-bandwidth.

[0039] The spatial spectrum training module is used to construct broadband frequency focusing spatial spectrum features based on the sub-bandwidth weighted focusing covariance matrix of each sub-bandwidth, and to construct a convolutional neural network model based on the broadband frequency focusing spatial spectrum features, and to estimate the DOA spatial spectrum of multiple sound sources using the convolutional neural network model.

[0040] The spatial spectrum inference module is used to predict the spatial spectrum of sound source DOA based on the decoding process of the mask estimation network model and the sound source DOA estimation network model, and to complete the estimation of DOA for multiple sound sources.

[0041] Thirdly, this application discloses an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to perform the method as described in any of the preceding aspects.

[0042] Fourthly, this application discloses a non-transitory computer-readable storage medium in which, when the instructions in the storage medium are executed by a processor of an electronic device, enable the electronic device to perform the methods described in any of the preceding aspects.

[0043] Fifthly, this application discloses a computer program product in which, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in any of the preceding aspects.

[0044] This application presents a method for estimating the direction of arrival (DOA) of multiple sound sources based on frequency-focused spatial spectrum. The method includes: first, generating a microphone array dataset and a training dataset using a pre-defined simulation environment and a microphone array signal simulator; second, constructing a mask estimation network model through sound source masking training to estimate sound mask information; third, dividing the multi-source signals into sub-bandwidths using a frequency band overlay method to obtain a weighted focusing covariance matrix; fourth, obtaining bandwidth-focused spatial spectrum features based on beamforming and the sub-bandwidth focusing covariance matrix, and training a convolutional neural network to estimate the DOA of the sound sources based on these features; and finally, predicting the spatial spectrum of the DOA of the sound sources based on the decoding process of the network model to complete the estimation of the DOA of multiple sound sources. This application reduces the impact of neural network nonlinear distortion on DOA estimation performance by using an intermediate representation of the broadband frequency-focused spatial spectrum; it improves the stability of the algorithm performance and the fault tolerance of DOA estimation by using an optimized broadband frequency-focused spatial spectrum to map the DOA spatial spectrum; and the proposed algorithm exhibits strong robustness in noisy and reverberant multi-source environments. Attached Figure Description

[0045] Figure 1 is a flowchart of the steps of a multi-source arrival direction estimation method based on frequency focusing spatial spectrum according to this application.

[0046] Figure 2 is a flowchart of the model construction method, training method, testing and inference method of a multi-source arrival direction estimation method based on frequency focusing spatial spectrum according to this application.

[0047] Figure 3 is a schematic diagram of the simulation environment setup for a microphone array signal based on a multi-source arrival direction estimation method using frequency focusing spatial spectrum according to this application.

[0048] Figure 4 shows the network structure for mask training and the network structure for DOA spatial spectrum training of sound source waves, which are based on a multi-source arrival direction estimation method using frequency focusing spatial spectrum according to this application.

[0049] Figure 5 is a structural block diagram of a multi-source arrival direction estimation device based on frequency focusing spatial spectrum according to this application.

[0050] Figure 6 is a block diagram of an electronic device according to this application.

[0051] Figure 7 is a block diagram of a computer-readable storage medium according to this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] In acoustics, masking refers to the process of processing signals using an acoustic mask. Acoustic masks are often represented as the ratio of the target signal to the observed signal, and are therefore mainly used in signal enhancement, signal separation, and target signal extraction.

[0054] Frequency focusing is a processing method that uses a focusing matrix in the frequency domain to transfer signals from different frequency bands to the same frequency band. It has wide applications in signal processing fields such as radar, sonar, antennas, and speech.

[0055] Convolutional Neural Networks (CNNs) are a type of feedforward neural network that includes convolutional computation and has a deep structure. They can perform translation-invariant classification of input information according to a hierarchical structure and are one of the most representative neural networks, with wide applications in the field of image processing.

[0056] Referring to Figure 1, a flowchart of a multi-source direction-of-arrival estimation method based on frequency focusing spatial spectrum according to this application is shown. This method can be applied to electronic devices, and specifically includes the following steps:

[0057] Step S110: Using a preset simulation environment and a microphone array signal simulator, a microphone array dataset is generated from a clean speech corpus of a single sound source, and a training dataset is constructed based on the microphone array dataset.

[0058] Step S120: By training the training dataset on sound source masking, a mask estimation network model is generated; the loss function of the mask estimation network model is the mean square error of complex ideal scale masking of the sound source.

[0059] Step S130: Based on the mask estimation network model, obtain the enhanced multi-source signal, and divide the enhanced multi-source signal into sub-bandwidths by frequency band overlay to obtain the weighted focusing covariance matrix of each sub-bandwidth.

[0060] Step S140: Based on the weighted focusing covariance matrix of each sub-bandwidth, a broadband frequency focusing spatial spectrum feature is constructed from the sub-bandwidth focusing covariance matrix obtained by beamforming, and a convolutional neural network model is constructed based on the broadband frequency focusing spatial spectrum feature. The convolutional neural network model is then used to estimate the DOA spatial spectrum of multiple sound sources.

[0061] Step S150: Using the mask estimation network model and the multi-source DOA spatial spectrum estimation network model, the multi-source DOA spatial spectrum is decoded, and significant peak values ​​are extracted to complete the estimation of multi-source DOA.

[0062] The multi-source direction of arrival (DOA) estimation method based on frequency-focused spatial spectrum provided in this application reduces the impact of neural network nonlinear distortion on DOA estimation performance by using the intermediate representation of broadband frequency-focused spatial spectrum; it improves the stability of algorithm performance and the fault tolerance of DOA estimation by using the optimized broadband frequency-focused spatial spectrum to map the DOA spatial spectrum; the proposed algorithm has strong robustness in noisy and reverberant multi-source environments.

[0063] In this example embodiment, to reduce the sensitivity of DOA estimation to nonlinear distortion caused by masking enhancement, this application uses the broadband frequency-focused spatial spectrum features of masked enhanced speech as an intermediate representation of the spatial orientation information of the sound source, instead of directly using masked enhanced speech to estimate the DOA information of the sound source. To improve the representation power of the broadband frequency-focused spatial spectrum for the spatial orientation information of the sound source, a series of optimization processes are performed, such as improving the frequency resolution of the broadband spatial spectrum features by dividing the sub-bandwidth, using the mean of the snapshot mask to weight the focusing covariance matrix to reduce the influence of the noise-dominant frequency band, and using beamforming to construct the spatial spectrum to avoid the problem of sound source number estimation. To improve the spatial resolution and fault tolerance of the DOA estimation algorithm, clean speech and endpoint detection results are used to construct DOA spatial spectrum labels at a specified resolution, and the broadband spatial spectrum features are mapped to the narrowband DOA spatial spectrum labels through a convolutional neural network to achieve robust DOA estimation performance in complex environments.

[0064] In this example embodiment, as shown in FIG2, the application includes the following steps:

[0065] Step S110: Data generation based on Pyroomacoustics and creation of training dataset.

[0066] 1) The simulation environment for microphone array signal generation is shown in Figure 3. The room dimensions are as follows: length is randomly selected between 5m and 12m; width is randomly selected between 3m and 8m, and width ≥ length / 2; height is randomly selected between 2.8m and 5m. The circular microphone array has 6 elements and a radius of 0.036m. The center of the array is randomly positioned within a smaller dashed box in the center of the room, aligned with the center of the room. The length and width of the dashed box are both 0.4m. The larger dark gray dashed box represents the area for the conference table. The length of the conference table is between 2.5m and 8m, constrained to be greater than or equal to half the length of the room. The width of the conference table is between 1m and 4m, constrained to be greater than or equal to one-third the width of the room. The height of the conference table is randomly selected between 1m and 1.5m. The light gray area represents the speaker activity area. This area is more than 0.25m away from the wall, and the number of speakers is between 1 and 4. The speaker positions are randomly generated within the gray area, with a minimum angle of 5° between speakers and the center of the microphone array.

[0067] 2) Generate microphone array dataset. Using the clean speech dataset and microphone array signal generator, generate corresponding clean microphone array speech, noisy reverberant microphone array signals, and source DOA tags; where the signal-to-noise ratio (SNR) is randomly generated between 0dB and 20dB, the reverberation duration RT60 is randomly generated between 0ms and 1200ms, and the inter-source signal-to-interference ratio (SIR) is randomly generated between -5dB and 5dB.

[0068] 3) Construction of masking training data. Short-time Fourier transform (STFT) is performed on the generated clean speech and noisy, reverberant speech in each channel to generate the corresponding input complex spectrum and the corresponding source complex ideal scale mask (cIRM). The definition of cIRM and the network structure used for training are given in step S120.

[0069] 4) Construction of DOA spatial spectrum training data. First, based on the STFT structure of noisy and reverberant speech in each channel, broadband frequency focusing spatial spectrum features are calculated and used as input for DOA spatial spectrum estimation. For detailed calculation steps, see steps S130 to S150. Second, the endpoint detection (VAD) algorithm is used to detect the active regions of clean speech from each sound source, and DOA spatial pseudo-spectrum (SPS) labels are generated for the active speech regions by equation (1). These labels are used as ideal values ​​for DOA spatial spectrum estimation in order to calculate the loss value and optimize the network parameters. The degree traversal interval is used to obtain pseudospectral data in DOA space at different resolutions; angle θi The traversal interval is generally set to 5°, meaning that when the DOA is correctly resolved, the maximum error in DOA estimation is 2.5°; q and Q are the indices and number of sound sources emitting sound simultaneously, respectively, and θ q Let be the true DOA angle of the q-th sound source.

[0070] Step S120: Construct a network model for sound source masking training.

[0071] Recent studies have shown that separating individual sound sources one by one through masking can make the multi-source DOA estimation problem easier. However, the number of sound sources is often unknown, and training multiple masks makes network training more difficult. Therefore, this application only trains on multi-source clean speech signals (i.e., direct sound signals), as shown in equations (2) to (3) generated in step S110:

[0072] Where, the subscript m represents the microphone array channel index; the subscript q = 1, 2, ..., Q, where q is the sound source index and Q is the total number of sound sources; k = 1, 2, ..., K, l = 1, 2, ..., L, where k and l are the frequency and frame indices respectively, and K and L are the frequency number and frame number respectively; X m (k,l) represents the mixed signal of Q direct sound sources acquired by the m-th channel, S q (k,l) represents the original signal from the q-th sound source, H m,q (0) is the direct sound transfer function from the q-th sound source to the m-th microphone. The symbols “Ι” and “Ρ” represent the operations of taking the imaginary part and taking the real part, respectively, and i represents the imaginary unit. As can be seen from equation (2), the constructed cIRM is the ratio of the mixed signal of the direct sound from the Q sound sources to the observed signal, that is, its purpose is to remove noise and reverberation and retain the direct sound signals of the Q sounds.

[0073] Based on the mask in equation (2), the constructed mask estimation network model is shown in Figure 4. In Figure 4, the input signal is the complete set of features of the past two frames, the current frame, and the future two frames, with a dimension of (246*5)×1; three hidden layers are used to map complex proportional masking, with 512 neurons in each hidden layer; the output layer has a dimension of 257×2, that is, the real part of the mask is 257×1 and the imaginary part is 257×1. Based on equation (2), the masking enhancement signal Z of each channel can be obtained from equation (4). m (k,l): Z m (k,l)=λ m (k,l)Y m (k,l) (4)

[0074] The mean square error of cIRM is used as the loss function Λ, as shown in equation (5).

[0075] in, and To estimate the real and imaginary parts of the complex scaling mask, λ r (k,l) and λ I (k,l) represents the real and imaginary parts of the cIRM.

[0076] Step S130: Obtain the weighted focusing covariance matrix.

[0077] Based on equation (4), a multi-channel enhanced signal with denoising and reverberation can be obtained. To mitigate the performance degradation of DOA estimation caused by masking distortion in Z(k,l), a broadband spatial spectrum is used as an intermediate representation of the spatial orientation information of the sound source. The covariance matrix of the direct sound from the sound source can be estimated as follows:

[0078] Where j = 0, 1, ..., J is the snapshot index, and J is the snapshot number.

[0079] Frequency focusing avoids the high complexity and incomplete rank of the covariance matrix caused by multi-band computation. Simultaneously, weighting the sub-bandwidth focusing covariance matrix using the speech presence probability reduces the influence of noise-dominated bands. Furthermore, dividing the frequency band into sub-bands improves the frequency dimension resolution of the spatial spectrum while reducing focusing errors across the entire bandwidth. Therefore, this patent application employs a sub-band weighted frequency focusing method to construct a broadband spatial spectrum. By uniformly dividing K frequency bands into W sub-bandwidths, frequency focusing can be performed within the W sub-bandwidths to obtain the weighted focusing covariance matrix, as shown in equations (7) to (8).

[0080] in, Let f be the weighted focusing covariance matrix of the w-th sub-bandwidth, where w = 1, 2, ..., W are the sub-bandwidth indices. w Let k be the focusing reference frequency for the w-th sub-bandwidth. w and K w These are the frequency band index and frequency band number within the w-th sub-bandwidth, respectively, λ(k w C(k) represents the estimated probability of the existence of direct speech. w ) is the kth w Focusing matrix for each frequency band.

[0081] Since the number of snapshots in practical applications is often limited (ideally as small as possible to accommodate streaming algorithms and the processing of moving sound sources), statistical methods are used... The spatial characteristics of the sound sources covered are not robust, which affects the subsequent estimation of the sub-bandwidth focused spatial spectrum characteristics. This causes the spatial information of some weaker sound sources to be obscured by residual noise, residual reverberation, and characteristics of other sound source signals. To improve this problem, this application employs sub-bandwidth stacking to obtain the sub-bandwidth focused spatial spectrum. The stacking of sub-bandwidths offers two advantages: first, it increases the number of sub-bandwidths, thereby further enhancing the frequency dimension of the broadband focused spatial spectrum; second, it increases the bandwidth of the sub-bandwidths without reducing frequency resolution, allowing for the inclusion of richer frequency dimension features. middle.

[0082] Step S140: Obtain the broadband focusing spatial spectrum features based on beamforming and construct a convolutional neural network model.

[0083] Based on equation (7), some traditional methods can be used to estimate DOA information, such as MUSIC. However, since the number of sound sources is unknown, and masking and weighted focusing cannot directly eliminate the effects of residual noise and phase distortion, high-resolution subspace methods like MUSIC are very sensitive to the estimated noise subspace. Therefore, this patent application uses minimum variance distortionless response (MVDR) beamforming to model the subbandwidth focusing spatial spectrum, as shown in equations (9) to (10):

[0084] Among them, Ω SRP (f w ,θ i ) represents the spatial response power (SRP) of the w-th sub-bandwidth, θ i ∈[0°~360°] represents the angle of spatial spectral ergodicity, A(f w ,θ i () is the reference frequency f w At θ i The directional guidance vector, h(f) w ,θ i ) represents the filter weights of the MVDR beamformer.

[0085] Based on the W sub-bandwidth focused spatial spectrum obtained by equation (9), effective broadband focused spatial spectrum features can be obtained without estimating the number of sound sources.

[0086] To improve the resolution and fault tolerance of the DOA estimation algorithm, the datasets from equations (1) and (9) are combined, and the DOA spatial spectrum is mapped from the broadband frequency-focused spatial spectrum using a CNN. The structure of the convolutional neural network model is shown in Figure 4, which mainly consists of two parts: the first part is a 4-layer convolutional layer for extracting discriminative features, and the second part is a 4-layer fully connected layer for mapping the DOA spatial spectrum. The input of the CNN is a broadband focused spatial spectrum of dimension Y×W. Each convolutional layer consists of 128 filters with a kernel of 2×1. The dimensions of the four fully connected layers are 512×1, 256×1, 128×1 and Y×1, respectively. The output of the last fully connected layer is the estimated DOA spatial spectrum, where Y is the number of DOA spatial spectra. The mean square error (MSE) shown in equation (11) is used as the loss function for optimizing the weights of the DOA estimation CNN.

[0087] Among them, the DOA space pseudo-spectrum of equation (1) is used as the real DOA space spectrum φ real , i.e. φ real =[φ SPS (θ1),φ SPS (θ2),...,φ SPS (θ γ )).

[0088] Step S150: DOA spatial spectrum test inference based on the mask estimation network model and the sound source DOA estimation network model decoding process.

[0089] Through steps S110 to S150, the trained DNN masking estimator and CNN spatial spectrum mapper can be obtained. The test inference process is shown in Figure 2. By finding the significant peaks in the output DOA spatial spectrum, the corresponding DOA estimation results can be obtained.

[0090] In this example embodiment, the DOA estimation method of this application based on broadband frequency-focused spatial spectrum and convolutional neural network is as follows:

[0091] Joint masking enhances the direct sound signals of each sound source in multi-source scenarios, and utilizes the broadband frequency focusing spatial spectrum of multi-sub-bandwidth superposition as an intermediate representation of the DOA estimation results, thereby reducing the impact of nonlinear distortion caused by masking processing on the DOA estimation algorithm.

[0092] In the process of solving the broadband frequency focusing spatial spectrum, a series of optimizations were performed: First, the mask mean within the snapshot was used as a weight to weight the focusing covariance matrix, reducing the influence of the noise-dominant frequency band; Second, MVDR beamforming was used to solve the spatial response power spectrum of the sub-bandwidth as the sub-bandwidth frequency focusing spatial spectrum, avoiding the problems of sound source number estimation and sub-space decomposition errors; Third, the sub-bandwidth stacking method was used to improve the frequency resolution and sub-bandwidth of the broadband frequency focusing spatial spectrum, enhancing the robustness of the focusing covariance matrix characteristics under finite fast sorting.

[0093] The design incorporates masking enhancement, broadband frequency focusing spatial spectrum estimation, and CNN-based DOA spatial spectrum estimation processing and training procedures, thereby improving the robustness and fault tolerance of the DOA estimation algorithm.

[0094] Compared to existing technologies, this application proposes a multi-source direction-of-arrival (DOA) estimation method based on broadband frequency-focused spatial spectrum and convolutional neural network by combining signal modeling methods with neural network methods. This method has the following advantages: First, it combines masking enhancement with broadband frequency-focused spatial spectrum, using broadband frequency-focused spatial spectrum as an intermediate representation of sound source spatial information, thereby mitigating the reduction in DOA estimation performance caused by the nonlinear distortion of masking enhancement. Second, to improve the representation ability of broadband frequency-focused spatial spectrum for sound source spatial information, the sub-bandwidths are stacked to improve the frequency resolution of broadband spatial spectrum features, and the mask mean is introduced to weight the focusing covariance matrix, improving the stability of reverberation environment features. Third, by designing DOA spatial spectrum labels with a certain resolution, broadband frequency-focused spatial spectrum features are mapped to DOA spatial spectrum features through CNN, improving the fault tolerance of broadband spatial spectrum features in representing sound source spatial information, thereby obtaining robust DOA estimation performance in complex acoustic environments.

[0095] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by this application.

[0096] Referring to Figure 5, a structural block diagram of a multi-source arrival direction estimation device 200 based on frequency focusing spatial spectrum according to this application is shown. The device includes a data construction module 210, a model construction module 220, a covariance matrix acquisition module 230, a spatial spectrum training module 240, and a spatial spectrum inference module 250, wherein:

[0097] The data construction module 210 is used to generate a microphone array dataset from a single-source clean speech corpus using a preset simulation environment and a microphone array signal simulator, and to construct a training dataset based on the microphone array dataset.

[0098] The model building module 220 is used to generate a mask estimation network model by training the training dataset on sound source masking; the loss function of the mask estimation network model is the mean square error of the complex ideal scale masking of the sound source.

[0099] The covariance matrix acquisition module 230 is used to acquire the enhanced multi-source signal based on the mask estimation network model, and to divide the enhanced multi-source signal into sub-bandwidths by frequency band overlay to acquire the weighted focusing covariance matrix of each sub-bandwidth.

[0100] The spatial spectrum training module 240 is used to construct broadband frequency focusing spatial spectrum features based on the sub-bandwidth weighted focusing covariance matrix of each sub-bandwidth, the sub-bandwidth frequency focusing covariance matrix obtained by beamforming, and to construct a convolutional neural network model based on the broadband frequency focusing spatial spectrum features, and to estimate the DOA spatial spectrum of multiple sound sources using the convolutional neural network model.

[0101] The spatial spectrum inference module 250 is used to predict the spatial spectrum of the sound source DOA based on the decoding process of the mask estimation network model and the sound source DOA estimation network model, and to complete the estimation of the DOA of multiple sound sources.

[0102] A multi-source direction-of-arrival (DOA) estimation device based on frequency-focused spatial spectrum processes multi-source signals and utilizes masking estimation, spatial spectrum training, and inference to construct an accurate multi-source localization and separation model. The device improves the accuracy and robustness of source localization and separation through the collaborative work of multiple modules, from environmental data generation and network model training to final DOA estimation.

[0103] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0104] Optionally, this application also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0105] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0106] Figure 6 is a block diagram of an electronic device 800 according to this application. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0107] Referring to FIG6, the electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0108] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0109] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, images, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0110] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0111] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0112] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0113] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0114] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0115] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast operation information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0116] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0117] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0118] Figure 7 is a block diagram illustrating a computer-readable storage medium 1900 according to this application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0119] Referring to FIG7, the computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0120] The computer-readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer-readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer-readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer-readable storage medium 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0121] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0123] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0124] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0126] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0129] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for estimating the direction of arrival of multiple sound sources based on frequency focusing spatial spectrum, comprising: Using a preset simulation environment and a microphone array signal simulator, a microphone array dataset is generated from a clean speech corpus of a single sound source, and a training dataset is constructed based on the microphone array dataset. A mask estimation network model is generated by training the sound source masking on the training dataset. The loss function of the mask estimation network model is the mean square error of the complex ideal scale masking of the sound source; Based on the mask estimation network model, the enhanced multi-source signal is obtained, and the enhanced multi-source signal is divided into sub-bandwidths by frequency band overlay to obtain the weighted focusing covariance matrix of each sub-bandwidth. Based on the sub-bandwidth weighted focusing covariance matrix, a broadband frequency focusing spatial spectrum feature is constructed from the sub-bandwidth frequency focusing covariance matrix obtained by beamforming, and a convolutional neural network model is constructed based on the broadband frequency focusing spatial spectrum feature. The convolutional neural network model is then used to estimate the DOA spatial spectrum of multiple sound sources. Using the mask estimation network model and the multi-source DOA spatial spectrum estimation network model, the multi-source DOA spatial spectrum is decoded, and significant peak values ​​are extracted to complete the estimation of multi-source DOA.

2. The method as described in claim 1, wherein, The microphone array dataset includes: clean microphone array speech signals, noisy and reverberant microphone array signals, and DOA tags for the sound sources.

3. The method as described in claim 2, wherein, A training dataset is constructed based on the microphone array dataset, including: Using the simulation environment, masking training data is constructed by performing short-time Fourier transform on the clean microphone array speech signal and the noisy and reverberant microphone array signal. Based on the short-time Fourier transform structure of the noisy and reverberant microphone array signal, the active region of the clean microphone array speech is detected and labeled using the endpoint detection algorithm to construct sound source DOA spatial spectrum training data; A training dataset is generated by integrating the masking training data and the sound source DOA spatial spectrum training data.

4. The method of claim 1, wherein, The enhanced multi-source signal is divided into sub-bandwidths using a frequency band overlay method, including: The broadband spatial spectrum is obtained by focusing the sub-bandwidth covariance matrix as an intermediate representation of the spatial orientation information of the sound source. The signal bandwidth is divided into multiple sub-bandwidths by stacking sub-bandwidths, and the speech presence probability of each sub-bandwidth is obtained based on the speech presence probability algorithm. The signals of each sub-bandwidth are weighted to obtain the sub-bandwidth weighted focusing covariance matrix.

5. The method of claim 1, wherein, The broadband frequency focusing spatial spectrum features are constructed from the sub-bandwidth frequency focusing covariance matrix obtained by beamforming, including: The spatial spectrum of the sub-bandwidth is modeled based on the minimum variance distortionless response beamforming method, and the frequency focusing spatial spectrum characteristics of the sub-bandwidth based on beamforming are obtained.

6. The method of claim 1, wherein, The convolutional neural network includes: The convolutional neural network includes convolutional layers for extracting discriminative features and fully connected layers for mapping the DOA spatial spectrum of the sound source wave.

7. The method of claim 1, wherein, Using the mask estimation network model and the multi-source DOA spatial spectrum estimation network model, the multi-source DOA spatial spectrum is decoded, including: The decoding process based on the mask estimation network model and the sound source DOA estimation network model generates the sound source wave DOA prediction result by calculating the significant peak values ​​of the multi-sound source DOA spatial spectrum. Based on the predicted DOA of the sound source waves, the DOA of multiple sound sources is estimated.

8. The method of claim 3, wherein, The constructed sound source DOA spatial spectrum training data includes: For the active region of the speech from the clean microphone array, a pseudo-spectral density (DOA) is generated according to the following formula as training labels for the convolutional neural network model to estimate the DOA spatial spectrum of multiple sound sources: in, For the constructed DOA space pseudospectrum, θ is the ergodic angle of the spatial spectrum. q Let θ be the true DOA angle of the q-th sound source, and σ be a parameter controlling the pseudo-spectral peak width.

9. The method of claim 1, wherein, The mask estimation network model is a feedforward neural network consisting of three hidden layers, wherein: The input signal features of the network model are a complete set of features consisting of the signals from the current frame, the two past frames, and the two future frames; and In the three hidden layers, each hidden layer has 512 neurons.

10. A multi-source direction-of-arrival estimation device based on frequency focusing spatial spectrum, comprising: The data construction module is used to generate a microphone array dataset from a single-source clean speech corpus using a preset simulation environment and a microphone array signal simulator, and to construct a training dataset based on the microphone array dataset. The model building module is used to generate a mask estimation network model by training the training dataset with sound source masking; the loss function of the mask estimation network model is the mean square error of the complex ideal scale masking of the sound source; The covariance matrix acquisition module is used to acquire the enhanced multi-source signal based on the mask estimation network model, and to divide the enhanced multi-source signal into sub-bandwidths by frequency band overlay to acquire the weighted focusing covariance matrix of each sub-bandwidth. The spatial spectrum training module is used to construct broadband frequency focusing spatial spectrum features based on the sub-bandwidth weighted focusing covariance matrix of each sub-bandwidth, and to construct a convolutional neural network model based on the broadband frequency focusing spatial spectrum features, and to estimate the DOA spatial spectrum of multiple sound sources using the convolutional neural network model. The spatial spectrum inference module is used to predict the spatial spectrum of sound source DOA based on the decoding process of the mask estimation network model and the sound source DOA estimation network model, and to complete the estimation of DOA for multiple sound sources.

11. An electronic device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 9.

Images