Methods and apparatus for audio object extraction

Abstract

The disclosure relates to a neural network-based method for audio processing. The method includes receiving an input audio signal comprising audio objects, the input audio signal covering frequency bands along a frequency dimension; and extracting the audio objects from the input audio signal by: extracting a first set of features from the input audio signal, applying an encoding process to the first set of features to obtain high-level feature information indicative of semantic features in the input audio signal; applying, a respective decoding process to obtain, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal; estimating, based on the reconstructed low-level feature information for each audio type, a respective band mask associated with an audio object to be extracted from the input audio signal; and obtaining the extracted audio objects based on the band masks.

Description

D25026W001METHODS AND APPARATUS FOR AUDIO OBJECT EXTRACTION CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to International Application No.PCT / CN2025 / 079934, filed February 28, 2025, of which is incorporated herein by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates to processing of audio signals and more particular to deep learning-based methods and devices for enhancing mixed audio signals containing audio objects of various types.BACKGROUND

[0003] An audio signal may be subjected to a mix of environment caused degradation, such as noise, echo reverberation, and processing related degradation, such as compression, transcoding and further processing steps before being listened to. This may result in a reduced listening experience for a user, as the audio quality’ of the played audio signal is not satisfactory. The audio signal may thus be referred to as a degraded audio or speech signal and enhancement of such a signal may advantageously be performed to reduce noise, reverberation and codec artefacts to improve the listening experience.

[0004] With the advancement of deep learning, speech enhancement algorithms based on deep learning have gained increasing attention and achieved significant progress. These algorithms aim to eliminate unwanted artifacts such as noise, echo, and reverberation, and extract clean speech from mixed audio signals. For example, to improve speech enhancement in these scenarios, speech enhancement based on neural networks has been investigated, where the neural network can be trained with speech comprising all ty pes of degradation, which provides an improved performance of speech enhancement in situations where the actual degradation is unknown to the enhancement method.

[0005] Hoyvever, these existing algorithms / approaches primarily focus on speech extraction and overlook other ty pes of audio signals. With the rapid grow th of the short video industry’, user-generated content (UGC) has become increasingly popular, encompassing a wide variety of audio signals. To deliver more immersive audio experiences, it might be advantageous toD25026W001develop advanced algorithms capable of processing multiple types of audio sources beyond just speech.

[0006] Accordingly, there exists a need for an advanced audio processing approach for audio enhancement, in particular for a deep learning-based audio object extraction system having an improved architecture that enables the simultaneous extraction of multiple audio objects from a mixed audio signal with enhanced accuracy and efficiency.SUMMARY

[0007] In view of the above, the present disclosure provides methods, apparatus, and programs, as well as computer-readable storage media for neural network-based audio processing, having the features of the respective independent claims.

[0008] Broadly speaking, the present disclosure aims at developing deep learning-based audio object extraction methods and systems featuring an improved architecture capable of simultaneously extracting multiple objects from a mixed audio signal.

[0009] According to an aspect of the disclosure, a neural network-based method for audio processing is provided. In particular, the method is directed to audio objection extraction for enhancing audio signals based on a neural network model. The method may, for example, be implemented at an inference stage in an audio processing system. The method may comprise receiving an input audio signal comprising a plurality of audio objects of different audio types. In particular, the input audio signal may cover a plurality of frequency bands along a frequency dimension. The method may also comprise extracting, using a neural network model, the plurality of audio objects from the input audio signal. Specifically, for the extraction of the plurality' of audio objects, the method may further comprise extracting a first set of features from the input audio signal. Herein, each feature in the first set of features may correspond to a combination of a frequency band of the input audio signal and a bin along a time dimension.

[0010] Also, the method may comprise applying an encoding process to the first set of features to obtain high-level feature information indicative of one or more semantic features in the input audio signal (e.g., latent acoustic representation). For example, an encoder can be employed to extract high-level features optimized for the noise suppression task from the raw band energies. In the present disclosure, the term "‘high-level feature information” refers to the compressed, abstract representations generated by an encoder. These features may captureD25026W001semantic and contextual attributes of the input audio, such as 1) Latent embeddings that represent distinct audio objects (e.g., vocals, drums) in a disentangled manner; and / or 2) Higher-order patterns and structural relationships within the audio signal, including temporal dependencies and harmonic content.

[0011] Moreover, the method may further comprise applying, for each of two or more different audio ty pes, a respective decoding process to obtain, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal (e.g.. low-level acoustic features from time-frequency feature representation of the audio signal). For example, a decoder may be applied to project discriminative features at different levels onto a high-resolution space (i.e., at the per-band level at each frame) to get a dense classification, namely the mask values. In the present disclosure, the term ‘low-level feature information" refers to finer-grained, reconstruction-oriented details produced by a decoder. These features may include, for example, 1) Timefrequency representations (e.g., spectrogram details) that preserve granular acoustic characteristics; 2) Transient details, and amplitude envelopes essential for high-fidelity' audio reconstruction; and / or 3) The layer-by-layer outputs of the decoder that progressively refine the extracted audio objects to match the original signal’s perceptual quality.

[0012] The method may also comprise estimating, based on the reconstructed low-level feature information for each audio ty pe, a respective band mask associated with an audio object to be extracted from the input audio signal. Subsequently, the method may comprise obtaining the extracted audio objects based on the band masks.

[0013] In some embodiments, obtaining the extracted audio objects based on the band masks may comprise estimating, based on each band mask, a respective second set of features, each feature of the second set of features corresponding to a frequency bin of the input audio signal; and obtaining the extracted audio objects based on the estimated second sets of features.

[0014] In some embodiments, the method may further comprise, for each of the plurality of audio objects, performing a residual operation subsequent to the respective decoding process, the residual operation comprising one or more sequential residual blocks each including one or more temporal convolution networks to increase receptive fields in the reconstructed low-level feature information.D25026W001

[0015] In some embodiments, a temporal convolution network of the one or more temporal convolution networks may comprise a first one-by-one convolutional layer to compress a channel dimension, a dilated convolutional layer to expand the receptive fields in the time dimension, a last one-by-one convolutional layer to restore the compressed channel dimension, and a residual connection between an input and an output of the temporal convolution network.

[0016] In some embodiments, the one or more temporal convolution networks may be implemented with increasing dilation factors for the dilated convolutional layer in dependence on a number of the temporal convolution networks.

[0017] In some embodiments, the temporal convolution network further may comprise a first batch normalization layer and a first non-linear activation layer following the first one-by-one convolutional layer, and a last batch normalization layer and a last non-linear activation layer following the last one-by-one convolutional layer.

[0018] In some embodiments, the method may further comprise applying an additional residual operation between the encoding process and the respective decoding processes.

[0019] In some embodiments, the method may further comprise applying a down-sampling operation at the encoding process and / or an up-sampling operation at the respective decoding processes.

[0020] In some embodiments, the neural network model may further comprise one or more convolutional neural network (CNN) layers following the respective decoding processes.

[0021] In some embodiments, the neural network model may comprise a deep neural network (DNN) model.

[0022] In some embodiments, the extracted audio objects may be obtained by applying an inverse transformation to the estimated second set of features.

[0023] In some embodiments, the method may further comprise, prior to obtaining the first set of features, transforming the input audio signal into the frequency domain to obtain a transformed audio signal; extracting a bin feature from the transformed audio signal for each frequency bin in the frequency domain; and for each frequency band of the input audio signal, combining features of the bin features corresponding to frequency bins inside the frequency band to obtain band features of the plurality of audio objects for the extraction.D25026W001

[0024] In some embodiments, transforming the input audio signal into the frequency domain may be performed by any one of a short time Fourier transform (STFT), a modified discrete cosine transform (MDCT). a shifted discrete frequency transform (MDXT), or a filter bank based transform.

[0025] In some embodiments, combining features of the bin features corresponding to frequency bins inside the frequency band to obtain band features of the plurality7of audio objects may comprise weighting the features of the bin features corresponding to frequency bins inside the frequency band.

[0026] In some embodiments, the input audio signal contains a mixture of speech and music, and the plurality of audio objects may comprise a first audio object of a speech type and a second audio object of a music type.

[0027] Configured as above, the proposed audio objection extraction approach for enhancing audio signals based on a neural network model with a shared encoder module to extract high-level information utilizes independent decoder modules to reconstruct low-level features for each specific audio object, which is designed to model distinct spectral patterns exhibited in different audio objects, thereby enhancing the accuracy and effectiveness of audio object extraction. Moreover, the receptive field of the model may be further increased by integrating residual modules incorporating temporal convolutional networks after the decoder modules for each audio object, which provides cleaner audio spectrums and reduced content leakage.

[0028] According to another aspect of the disclosure, a neural netw ork-based method for audio processing is provided. In particular, the method is directed to audio objection extraction for enhancing audio signals based on a neural network model. The method may, for example, be implemented at a training stage in an audio processing system. The method may comprise receiving an input audio signal comprising a plurality of audio objects of different audio types. In particular, the input audio signal may cover a plurality' of frequency bands along a frequency dimension. The method may also comprise extracting, using a neural network model, the plurality of audio objects from the input audio signal. Specifically, for the extraction of the plurality' of audio objects, the method may further comprise extracting a first set of features from the input audio signal. Herein, each feature in the first set of features may correspond to a combination of a frequency band of the input audio signal and a bin along a time dimension.D25026W001

[0029] Also, the method may comprise applying an encoding process to the first set of features to obtain high-level feature information indicative of one or more semantic features in the input audio signal (e.g., latent acoustic representation). For example, an encoder can be employed to extract high-level features optimized for the noise suppression task from the raw band energies. In the present disclosure, the term ‘'high-level feature information'’ refers to the compressed, abstract representations generated by an encoder. These features may capture semantic and contextual attributes of the input audio, such as 1) Latent embeddings that represent distinct audio objects (e.g., vocals, drums) in a disentangled manner; and / or 2) Higher-order patterns and structural relationships within the audio signal, including temporal dependencies and harmonic content.

[0030] Moreover, the method may further comprise apply ing, for each of two or more different audio types, a respective decoding process to obtain, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal (e.g., low-level acoustic features from time-frequency feature representation of the audio signal). For example, a decoder may be applied to project discriminative features at different levels onto a high-resolution space (i.e., at the per-band level at each frame) to get a dense classification, namely the mask values. In the present disclosure, the term “low-level feature information” refers to finer-grained, reconstruction-oriented details produced by a decoder. These features may include, for example, 1) Timefrequency representations (e.g., spectrogram details) that preserve granular acoustic characteristics; 2) Transient details, and amplitude envelopes essential for high-fidelity audio reconstruction; and / or 3) The layer-by-layer outputs of the decoder that progressively refine the extracted audio objects to match the original signal’s perceptual quality.

[0031] The method may also comprise estimating, based on the reconstructed low-level feature information for each audio type, a band mask associated with an audio object to be extracted from the input audio signal. Subsequently, the method may comprise estimating, based on each band mask, a respective second set of features for each audio type. Herein, each feature of the second set of features corresponds to a frequency bin of the input audio signal.

[0032] Additionally, the method may comprise receiving a reference audio signal corresponding to the input audio signal; and extracting, for each of the two or more different audio t pes, a third set of features from the reference audio signal. In particular, each feature in the third sets of features corresponds to a frequency bin of the reference audio signal. Furthermore, the method may comprise determining a loss function based on the second setsD25026W001of features and the third sets of features; and updating one or more parameters of the neural network model based on the determined loss function.

[0033] In some embodiments, obtaining the extracted audio objects based on the band masks may comprise estimating, based on each band mask, a respective second set of features, each feature of the second set of features corresponding to a frequency bin of the input audio signal; and obtaining the extracted audio objects based on the estimated second sets of features.

[0034] In some embodiments, the method may further comprise, for each of the plurality of audio objects, performing a residual operation subsequent to the respective decoding process, the residual operation comprising one or more sequential residual blocks each including one or more temporal convolution networks to increase receptive fields in the reconstructed low-level feature information.

[0035] In some embodiments, a temporal convolution network of the one or more temporal convolution networks may comprise a first one-by-one convolutional layer to compress a channel dimension, a dilated convolutional layer to expand the receptive fields in the time dimension, a last one-by-one convolutional layer to restore the compressed channel dimension, and a residual connection between an input and an output of the temporal convolution network.

[0036] In some embodiments, the one or more temporal convolution networks may be implemented with increasing dilation factors for the dilated convolutional layer in dependence on a number of the temporal convolution networks.

[0037] In some embodiments, the temporal convolution network further may comprise a first batch normalization layer and a first non-linear activation layer following the first one-by-one convolutional layer, and a last batch normalization layer and a last non-hnear activation layer following the last one-by-one convolutional layer.

[0038] In some embodiments, the method may further comprise applying an additional residual operation between the encoding process and the respective decoding processes.

[0039] In some embodiments, the method may further comprise applying a down-sampling operation at the encoding process and / or an up-sampling operation at the respective decoding processes.D25026W001

[0040] In some embodiments, the neural network model may further comprise one or more convolutional neural network (CNN) layers following the respective decoding processes.

[0041] In some embodiments, the neural network model may comprise a deep neural network (DNN) model.

[0042] In some embodiments, the extracted audio objects may be obtained by applying an inverse transformation to the estimated second set of features.

[0043] In some embodiments, the method may further comprise, prior to obtaining the first set of features, transforming the input audio signal into the frequency domain to obtain a transformed audio signal; extracting a bin feature from the transformed audio signal for each frequency bin in the frequency domain; and for each frequency band of the input audio signal, combining features of the bin features corresponding to frequency bins inside the frequency band to obtain band features of the plurality of audio objects for the extraction.

[0044] In some embodiments, transforming the input audio signal into the frequency domain may be performed by any one of a short time Fourier transform (STFT), a modified discrete cosine transform (MDCT), a shifted discrete frequency transform (MDXT), or a filter bank based transform.

[0045] In some embodiments, combining features of the bin features corresponding to frequency bins inside the frequency band to obtain band features of the plurality of audio objects may comprise weighting the features of the bin features corresponding to frequency bins inside the frequency band.

[0046] In some embodiments, the input audio signal contains a mixture of speech and music, and the plurality7of audio objects may comprise a first audio object of a speech ty pe and a second audio object of a music ty pe.

[0047] Configured as above, the proposed audio objection extraction approach based on training a neural network model which includes a shared encoder module to extract high-level information and independent decoder modules to reconstruct low-level features for each specific audio object enables simultaneous extraction of multiple audio objects from a mixed audio signal with enhanced accuracy and efficiency. In particular, object extraction from the mixture can be improved and content leakage can be reduced by incorporating the residual module after the decoder module.D25026W001

[0048] According to another aspect of the disclosure, a computer system for audio processing based on a neural network model is provided. Specifically, the computer system may comprise a memory and one or more processors coupled with the memory and configured to perform receiving an input audio signal comprising a plurality of audio objects of different audio t pes, the input audio signal covering a plurality of frequency bands along a frequency dimension; and extracting, using the neural network model, the plurality' of audio objects from the input audio signal.

[0049] More specifically, the neural network model may comprise a feature extraction module for extracting a first set of features from the input audio signal, wherein each feature in the first set of features corresponds to a combination of a frequency band of the input audio signal and a bin along a time dimension; an encoder module for obtaining, from the first set of features, high-level feature information indicative of one or more semantic features in the input audio signal; and a plurality of decoder modules each associated with a respective audio ty pe for obtaining, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal.

[0050] In the present disclosure, the term “high-level feature information” refers to the compressed, abstract representations generated by an encoder. These features may capture semantic and contextual attributes of the input audio, such as 1) Latent embeddings that represent distinct audio objects (e.g., vocals, drums) in a disentangled manner; and / or 2) Higher-order patterns and structural relationships within the audio signal, including temporal dependencies and harmonic content. On the other hand, the term “low-level feature information” refers to finer-grained, reconstruction-oriented details produced by a decoder. These features may include, for example, 1 ) Time-frequency representations (e g., spectrogram details) that preserve granular acoustic characteristics; 2) Transient details, and amplitude envelopes essential for high-fidelity audio reconstruction; and / or 3) The layer-by-layer outputs of the decoder that progressively refine the extracted audio objects to match the original signal’s perceptual quality.

[0051] Furthermore, the one or more processors of the computer system may also be configured to perform estimating, based on the reconstructed low -level feature information for each audio type, a band mask associated with an audio object to be extracted from the input audio signal.D25026W001

[0052] Notably, the computer system may be implemented in a training phase where the neural network model is trained, and in an inference phase where the (trained) neural network model is used for audio object extraction.

[0053] In some embodiments, at an inference stage, the one or more processors may further be configured to perform obtaining the extracted audio objects based on the band masks. Alternatively, at a training stage, the one or more processors may further be configured to perform extracting a third set of features from the reference audio signal, wherein each feature in the third set of features corresponds to a frequency bin of the reference audio signal receiving a reference audio signal corresponding to the input audio signal; determining a loss function based on the second set of features and the third set of features; and updating one or more parameters of the neural network model based on the determined loss function.

[0054] In some embodiments, obtaining the extracted audio objects based on the band masks may comprise estimating, based on each band mask, a respective second set of features, each feature of the second set of features corresponding to a frequency bin of the input audio signal; and obtaining the extracted audio objects based on the estimated second sets of features.

[0055] In some embodiments, the neural network model may further comprise a plurality of residual modules each associated with a respective audio type, said each residual module receiving an output from the respective decoder module, said each residual module comprising one or more sequential residual blocks each including one or more temporal convolution networks to increase receptive fields in the reconstructed low-level feature information.

[0056] In some embodiments, a temporal convolution network of the one or more temporal convolution networks may comprise a first one-by-one convolutional layer to compress a channel dimension, a dilated convolutional layer to expand the receptive fields in the time dimension, a last one-by-one convolutional layer to restore the compressed channel dimension, and a residual connection between an input and an output of the temporal convolution network.

[0057] In some embodiments, the one or more temporal convolution networks may be implemented with increasing dilation factors for the dilated convolutional layer in dependence of a number of the temporal convolution networks.D25026W001

[0058] In some embodiments, the temporal convolution network may further comprise a first batch normalization layer and a first non-linear activation layer following the first one-by-one convolutional layer, and a last batch normalization layer and a last non-linear activation layer following the last one-by-one convolutional layer.

[0059] In some embodiments, the neural network model may further comprise an additional residual module between the encoder module and the respective decoder modules.

[0060] In some embodiments, the neural network model may comprise a deep neural network (DNN) model.

[0061] In some embodiments, the DNN may comprise an up-sampling module for estimating a bin mask for said each audio object present in each frequency bin of the input audio signal.

[0062] In some embodiments, the DNN may further comprise the feature extraction module, followed by the encoder module, followed by the respective decoder modules, and a plurality of convolutional neural network (CNN) layers. Herein, each of the decoder modules may be followed by the up-sampling module, which may be followed by a respective CNN layer of the plurality of CNN layers.

[0063] In some embodiments, the encoder module may comprise at least one down-sample layer and a plurality of CNN layers. In particular, each of the decoder modules may comprise at least one up-sample layer and a plurality of CNN layers.

[0064] In some embodiments, the neural network may have been trained based on a pair of a clean signal and a corresponding degraded signal.

[0065] In some embodiments, the input audio signal may contain a mixture of speech and music, and the plurality of audio objects may comprise a first audio object of a speech type and a second audio object of a music type.

[0066] Aspects of the present disclosure may be implemented via an apparatus. The apparatus may include a processor and memory coupled to the processor. The processor may be adapted to carry out the method according to aspects and embodiments of the present disclosure.

[0067] Aspects of the present disclosure may be implemented via a program. When instructions of the program are executed by a processor, the processor may carry out aspects and embodiments of the present disclosure. A computer-readable storage medium may store the program. Such computer-readable storage media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices,D25026W001read-only memory (ROM) devices, etc.. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented via one or more computer-readable storage media having software stored thereon.

[0068] It should be noted that the methods and apparatus including its preferred embodiments as outlined in the present disclosure may be used stand-alone or in combination with the other methods and apparatus disclosed in this document. Furthermore, all aspects of the methods and apparatus outlined in the present disclosure may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner.

[0069] It will be appreciated that apparatus features and method steps may be interchanged in many ways. In particular, the details of the disclosed method(s) can be realized by the corresponding apparatus, and vice versa, as the skilled person will appreciate. Moreover, any of the above statements made with respect to the method(s) (and, e.g., their steps) are understood to likewise apply to the corresponding apparatus (and. e.g., their blocks, stages, units), and vice versa.BRIEF DESCRIPTION OF DRAWINGS

[0070] Example embodiments of the disclosure are explained below with reference to the accompanying drawings, wherein

[0071] Fig. 1 schematically illustrates an example system 100 for audio processing based on a neural network model according to embodiments of the disclosure,

[0072] Fig. 2 schematically illustrates an example neural network model 200 for audio object extraction according to embodiments of the disclosure,

[0073] Fig. 3 schematically illustrates an example neural network model 300 for audio object extraction according to embodiments of the disclosure,

[0074] Fig. 4 schematically illustrates an example residual module 400 for the proposed neural network model 300 according to embodiments of the disclosure,

[0075] Fig. 5 schematically illustrates an example temporal convolutional network 500 for the proposed residual module 400 according to embodiments of the disclosure,

[0076] Fig. 6 is a flowchart of an example of a process 600 of training a neural network for audio processing according to embodiments of the disclosure,D25026W001

[0077] Fig. 7 schematically illustrates a schematic block diagram of an example electronic device or architecture 700 suitable for implementing example embodiments of the present disclosure, and

[0078] Fig. 8 is a flowchart of an example of a process 800 of using a neural network for audio processing according to embodiments of the disclosure.DETAILED DESCRIPTION

[0079] In the following, example embodiments of the disclosure will be described with reference to the appended figures. It is noted that wherever practicable similar or like reference numbers may be used in the figures and may indicate similar or tike functionality , and repeated description thereof may be omitted.

[0080] The Figures (Figs.) and the following description relate to preferred embodiments (i.e., the disclosed system or method) by way of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of what is claimed.

[0081] In the present disclosure, the “modules” or “systems” illustrated in the Figures do not imply a hardware or software implementation but are merely used for illustration purposes. The functions of those modules can be implemented physically or logically according to the scenarios, which are not limited in the present disclosure.

[0082] As mentioned above, the present disclosure relates generally to deep learning-based audio processing methods / systems for audio object extraction. Recently, deep neural networks (DNNs) have been successfully used in speech enhancement (i.e., DNN-based speech enhancement) that targets the removal of multiple unwanted artifacts such as noise, reverberation, and compression, while preserving the original speech. A commonly used method for DNN-based speech enhancement is time-frequency masking. More specifically, the DNN-based models usually use spectrum bin features of the speech signal as input and estimate a time-frequency mask which can be applied to the spectrum bin features of the speech signal.

[0083] However, for many applications operating in real-time, estimating a mask with the neural network based on the spectrum bin features may be too time consuming. For thisD25026W001reason, existing models have used spectrum band features instead of spectrum bin features as input for the DNN. The DNN then outputs a time-frequency mask which can be applied to the spectrum band features of the speech signal. Thereby, the model complexity of the DNN and the delay introduced by the processing of the speech signal can be greatly decreased. To achieve complexity reduction, a neural network is proposed that estimates a mask on the bin level based on band level input features. As an example for such a neural network, a band-to-bin based LensNet model is proposed to increase the resolution of the estimated masks while keeping complexity as low as that of the LensNet model described in PCT Publication No. WO 2022 / 094290, titled ‘DEEP-LEARNING BASED SPEECH ENHANCEMENT”, which is hereby incorporated by reference in its entirety.

[0084] To further achieve effective audio object extraction, the methods / sy stems as proposed in the present disclosure feature an improved model architecture by means of a neural network model that includes independent decoder modules (with residual modules utilizing temporal convolutional networks in some use cases) for extracting audio objects from a mixed audio signal. That is, for each specific audio object, the processing flow may be conducted independently using the respective decoder modules.

[0085] Specifically, an audio object extraction system featuring a model architecture is proposed, which may be based on a deep neural network model, such as a deep learning-based noise suppression and speech enhancement model. This model architecture employs a U-net structure with a shared encoder module to extract high-level information and utilizes independent decoder modules to reconstruct low-level features for each specific audio object. Given that different objects exhibit distinct spectral patterns, it may be more effective to use separate decoders tailored for each object. Moreover, to further increase the receptive field of the model, residual modules incorporating temporal convolutional networks may additionally be integrated after the decoder modules for each audio object, resulting in cleaner audio spectrums and reduced content leakage.

[0086] Fig. 1 schematically illustrates an example system 100 for audio processing based on a neural network model according to embodiments of the disclosure. According to the present disclosure, the system 100 may refer to a deep learning-based audio object extraction system. As shown in Fig. 1, the system 100 may be run in a training phase and in an inference phase, where the system architecture for a training stage is depicted in Fig. 1(a). while the system architecture for an inference stage is depicted in Fig. 1(b), separated by the dashed line.D25026W001

[0087] Specifically, the system 100 may be implemented by one or more computers that include a memory' and one or more processors (not shown) coupled with the memory to perform neural network-based audio processing as mentioned above. For example, the system 100 may include a neural network model, such as an audio object extraction model 104 (104a for the training stage and 104b for the inference stage) for extracting audio objects of different types from a mixed audio signal.

[0088] In the example of the system 100, the training of the neural network model may be based on a pair of a clean / reference audio signal (e.g.. clean speech, clean music, etc.) and a degraded audio signal (e.g., noisy mixture). The degraded speech signal may be generated based on the reference speech signal. The generation of the degraded speech signal may be based on artificial degradation of the speech signal, e.g. adding a noise floor to the speech signal, or / and may be based on a real degradation due to a system transmission chain. Fig. 1(a) schematically illustrates an example architecture for training the neural network for audio enhancement according to embodiments of the disclosure.

[0089] At the training stage as illustrated in the example of Fig. 1(a), for each training step, clean speech 110-1 and clean music 110-2 (serving as ground truth), as well as room impulse responses (not shown) and noise may be collected, and data augmentation may then be performed to generate the training pairs, which include a noisy mixture 101a, clean speech 110-1, and clean music 110-2. These inputs may be fed into a feature extraction module for feature extraction Ila, 1 la’ to obtain complex bin features, i.e., bin features of the noisy mixture 102a, bin features of clean speech 109-1 and bin features of clean music 109-2. The feature extraction module I la. Ila’ may process the inputs on a frame-by-frame basis, which may operate on a single frame of the input audio signals (e.g., the noisy mixture 102a, clean speech 109-1, clean music 109-2) to determine a feature for each frequency bin of the input audio signal. Notably, the number of frequency bins may depend on a sampling frequency and a frame size.

[0090] To determine the features for the input audio signals, the signals may have to be transformed into the frequency domain. Any suitable discrete frequency transform (Fourier transform, Wavelet transform, etc.,) may be employed. Advantageous examples comprise a short time Fourier transform, SFTF, a modified discrete cosine transform, MDCT, a shifted discrete frequency transform, MDXT, and a filter bank transform. A reason for using MDXT instead of MDCT or DFT is that it provides both the energy compaction property of theD25026W001MDCT and the phase information similar to DFT. The feature determination may comprise determining the complex spectrum value for each frequency bin.

[0091] Accordingly, in the system 100, the waveform may be transformed into timefrequency bin features using a transform function, such as STFT, MDXT. CQMF, or any other time-frequency transform. The complex bin features of the noisy mixture 102a may then be grouped into perceptually motivated band features 103a via a banding process 12a. These band features 103a may then be input into the audio object extraction model 104a to predict band masks 105a associated with the noisy mixture 101a for speech and music content (105a-1 indicating the estimated band masks for speech, 105a-2 indicating the estimated band masks for music content). The band masks 105a-l, 105a-2 may then be converted to bin masks 106a associated with the noisy mixture 101a (106a-l indicating the estimated bin masks for speech, 106a-2 indicating the estimated bin masks for music content) through inverse banding 13a. The bin mask may then be applied to the bin features to generate enhanced bin features. In some cases, the generation of the enhanced bin features may be based on an enhancement module (not shown) to determine the (enhanced) bin features corresponding to the noisy mixture 101a (e.g., a speech / music signal with degradations), which may apply the bin mask to the bin features by multiplication of each value of the bin mask with each corresponding bin feature. The bin feature may be the complex values of each frequency bin. The enhanced bin features may therefore also correspond to complex values of each frequency bin.

[0092] Thus, as further shown in Fig. 1(a), the predicted bin masks 106a-l, 106a-2 may then be multiplied by the spectrum of the noisy mixture 101a (as indicated by 16a-l, 16a-2) to derive the predicted speech and music spectrums associated with the noisy mixture 101a. as the estimated bin features for speech content 107a-l and the estimated bin features for music content 107a-2, respectively. Similarly, spectrums associated with the clean speech and the clean music may be derived from the respective bin features 109-1, 109-2. Finally, the predicted spectrums associated with the noisy mixture 14-1. 14-2 and the corresponding clean speech and music spectrums 15-1, 15-2 may be used for a loss function 108 to compute a specific loss for use to update one or more parameters of the neural network model (e.g., the audio object extraction model 104a).

[0093] As the aim of the speech enhancement framework is an output speech signal as close as possible to the reference / clean audio signal, in a final step the enhanced bin features (e.g., the predicted spectrums associated with the noisy mixture 14-1, 14-2) and the bin features of the reference / clean audio signal (e.g., the corresponding clean speech and music spectrumsD25026W00115-1, 15-2) have to be compared. The comparison is performed by the loss function 108. Any suitable loss function may be used to evaluate the performance of the audio object extraction in the system 100 for audio enhancement, such as the mean square error (MSE).Advantageously, a hybrid loss function may be employed to further improve the performance of the audio object extraction system 100.

[0094] The result of the loss function will be evaluated. Evaluation may be performed on the result of the loss function over multiple pairs of reference / clean audio signal and degraded / noisy audio signal and over multiple frames of the respective pairs. In some examples, the pairs may capture a large variety of speech, e g., gender, age etc., and a large variety of degradations for each clean speech sample. In other words, for each sample of a clean audio signal, multiple samples of degradations of this specific audio sample may be provided.

[0095] Depending on the evaluation result, parameters of the neural network model 104a may be updated. Updating the parameters may include updating of weights in the neural network model. The neural network model 104a may be trained until the result of the loss function 108 reaches a threshold or until the result of the loss function 108 does not decrease substantially anymore.

[0096] Fig. 2 schematically illustrates an example neural network model 200 for audio object extraction according to embodiments of the disclosure. Specifically, the neural network model 200 may be implemented as the audio object extraction model 104a, 104b of the system 100 in Fig. 1, and may implemented by a deep learning-based model (which also refers to a deep neural network (DNN) model), such as the LensNet model as described above. The DNN may be any suitable DNN for mask-based speech enhancement that is able to upscale the relatively few number of band features to a mask with values for each frequency bin.

[0097] As illustrated in Fig. 2, the model architecture may include a feature extraction module 201, an encoder module 202, a decoder module 203 and a final convolutional neural network (CNN) layer 204. The feature extraction module 201 may obtain the band features 103a as described above. The encoder module 202 may include one or more down-sampling layers, combined with additional CNN layers and dense connections. Similarly, the decoder module 203 may include one or more up-sampling layers, along with other CNN layers and dense connections. The final CNN layer 204 may be utilized to estimate the speech masks or the music masks. Accordingly, the output of the final CNN layer 204 may be a mask or multiple masks (for speech or other types of audio objects), as indicated by 205. The mask may haveD25026W001the same resolution as the input features. In other words, if the input features are the band features, the output mask may have a value corresponding to each frequency band.

[0098] Fig. 3 schematically illustrates an example neural network model 300 for audio object extraction according to embodiments of the disclosure. Specifically, the neural network model 300 may be implemented as the audio object extraction model 104a, 104b of the system 100 in Fig. 1, and, similar to the neural network model 200, may implemented by a DNN model, e.g., the LensNet model as described above, which may likewise include a feature extraction module 301 having similar functionality to the feature extraction module 201 of the neural network model 200. However, compared to the structure of the neural network model 200 in Fig. 2, the architecture of the neural network model 300 may employ a shared encoder module 302 (which may also have a similar structure to conduct similar functions as the encoder module 202) to extract high-level information (e.g., latent acoustic representation) from the noisy features and utilizes independent decoder modules 303-1, 303-2 to reconstruct low-level features (e.g., low-level acoustic features from time-frequency feature representation of the audio signal) for each specific audio object. For example, the decoder module 303-1 may be used to reconstruct the low-level features for the speech, and the decoder module 303-2 may be used to reconstruct the low-level features for the music content. Specifically, the independent decoder modules 303-1 , 303-2 may also have a similar structure / functionality to the decoder module 203 in the model 200. An exemplary encoder for the encoder module 302 may include a series of blocks that produce input feature maps corresponding to increasingly larger receptive fields in the input audio data along the frequency dimension. An exemplary decoder for the decoder modules 303-1, 303-2 may include a series of blocks that produce output feature maps from the input feature maps.

[0099] As indicated above, the term “high-level feature information” refers to the compressed, abstract representations generated by an encoder. These features may capture semantic and contextual attributes of the input audio, such as 1) Latent embeddings that represent distinct audio objects (e.g., vocals, drums) in a disentangled manner; and / or 2) Higher-order patterns and structural relationships within the audio signal, including temporal dependencies and harmonic content. On the other hand, the term “low-level feature information” refers to finer-grained, reconstruction-oriented details produced by a decoder. These features may include, for example, 1) Time-frequency representations (e.g., spectrogram details) that preserve granular acoustic characteristics; 2) Transient details, and amplitude envelopes essential for high-fidelity' audio reconstruction; and / or 3) The layer-by-D25026W001layer outputs of the decoder that progressively refine the extracted audio objects to match the original signal's perceptual quality.

[0100] Accordingly, different decoder modules for different objects may be used to account for the distinct spectral patterns exhibited by different objects. Compared to the architecture of the model 200 with a shared decoder as shown in Fig. 2, the neural network model 300 using independent decoders can distinguish these patterns of different audio objects more effectively, and can learn the specific patterns of each object and reconstruct object signals more accurately based on these learned patterns. In this way, using independent decoder modules may significantly enhance the object extraction performance.

[0101] Furthermore, to increase the receptive field of the neural network model in the time dimension, residual modules 306-1, 306-2 may also be utilized subsequent to the decoder modules 303-1. 303-2 for each audio object (e.g., for speech and music content, respectively). In some embodiments, the residual modules 306-1, 306-2 may be integrated with temporal convolutional networks as described below in relation to Fig.4 and Fig. 5. Subsequent to the residual modules 306-1, 306-2, a respective final convolutional neural network (CNN) layer 304-1, 304-2 may be applied to estimate the speech masks (indicated by 305-1) and the music masks (indicated by 305-2), similar to the final CNN layer 204 in the model 200.

[0102] Fig. 4 schematically illustrates an example residual module 400 for the proposed neural network model 300 according to embodiments of the disclosure. Specifically, the residual module 400 may be implemented as the residual modules 306-1, 306-2 of the audio object extraction model 104a, 104b as shown in Fig. 1. As illustrated in Fig. 4, the structure of the residual module 400 may include one or more repeated / sequential residual blocks 41-1, 41-2, ... , 41-N. In detail, each residual block 41-1, 41-2, ... , 41-N may include one or more temporal convolution networks 401-1, ... , 401-M wdth increasing dilation factors, such as 1, 2, 4, ... , 2M, where M represents the number of the temporal convolutional netw orks within each residual block 41-1. 41-2. .... 41-N.

[0103] Fig. 5 schematically illustrates an example temporal convolutional network 500 for the proposed residual module 400 according to embodiments of the disclosure. Specifically, the temporal convolutional netw ork 500 may be implemented as the one or more temporal convolution networks 401-1, ... , 401-M within the residual modules 306-1, 306-2 of the audio object extraction model 104a, 104b as shown in Fig. 1. As illustrated in Fig. 5, the structure of the temporal convolutional network 500 may include a 1x1 convolutional layer 501, a dilated convolutional layer 504, an output 1x1 convolutional layer 507, and a residual connection 510D25026W001between input and output. More specifically, each convolutional layer may be followed by a batch normalization layer 502, 505, 508 and anon-linear activation layer 503, 506, 509. The input convolutional layer 501 may compress the channel dimension to a smaller number, while the subsequent dilated convolutional layer 504 may expand the receptive field in the time dimension. To accommodate the residual connection, the output convolutional layer 507 may restore the channel number to its original size. Since 1x1 convolutional layers are used, the model complexity’ is not significantly increased. By stacking multiple temporal convolutional networks 401-1, ... , 401-M, the receptive field can be substantially enlarged, which improves object extraction from the mixture and reduces content leakage by incorporating the residual module after the decoder module.

[0104] After training of the audio object extraction model, the audio object extraction model may be used for extracting the audio objects of various types (from different sources for example) from a mixed audio signal for enhancing the mixed audio signal. Fig. 1(b) schematically illustrates an example architecture for using the neural network for audio enhancement according to embodiments of the disclosure. Certain modules in Fig. 1(b) may be identical to modules in Fig. 1(a). For a detailed explanation of these modules, it is referred to the embodiment corresponding to Fig. 1(a).

[0105] For example, similar to the architecture of Fig. 1(a), an audio signal may be processed by a feature extraction module similar to the feature extraction module for feature extraction 1 la, Ila’, which may output bin features of the audio signal. The bin features may likewise be grouped to band features. The audio signal may be degraded due to. e.g., background noise, and may further be degraded by a lossy compression or by compression artifacts during transmission.

[0106] Accordingly, at the inference stage in the system 100 as illustrated in the example of Fig. 1(b), a test signal 101b may be fed into a feature extraction module for feature extraction 1 lb to obtain complex bin features, i.e., bin features of the test signal 102b. Functionality of the feature extraction module 1 lb may be identical to functionality of the feature extraction module I la, I la’. Similar to the procedure performed at the training stage, these features may be grouped into band features 103b and fed into the trained audio object extraction model 104b (which may have been trained based on the architecture as shown in Fig. 1(a), for example) to predict speech and music band masks associated with the test signal 101b, as indicated by 105b-l, 105b-2, respectively. Notably, the audio object extraction model 104b may have the same structure as the neural network model 104a, i.e., it may be a DNN, andD25026W001preferably structured as depicted in Figs. 2, 3 and 4. In particular, the audio object extraction model 104b may be a trained version of the neural network model 104a, i.e., the weights of neural network 104b have been optimized for audio enhancement based on training data.

[0107] The band masks 105b-l, 105b-2 may then be converted to bin masks 106b associated with the test signal 101b (106b-l indicating the estimated bin masks for speech, 106b-2 indicating the estimated bin masks for music content) using inverse banding 13b. Similar to the architecture of Fig. 1(a), the bin mask and the bin features may be processed by an enhancement module (not shown) which determines enhanced bin features. The enhanced bin features may correspond to an enhanced version of the input audio signal. As illustrated in Fig. 1(b), the predicted bin masks 106b and the spectrum of the test signal (as indicated by 16b-l, 16b-2) may be used (e.g., by a multiplication operation) to generate the predicted spectrums associated with the test signal 101b, as the estimated (enhanced) bin features for speech content 107b-l and the estimated (enhanced) bin features for music content 107b-2, respectively, which may then be used to obtain the extracted speech 120-1 and music waveform 120-2 through a transform-inverse processing 17.

[0108] It should be further noted that to generate the enhanced audio signal, an inverse frequency transform (inverse time-frequency transform) may be performed, which may correspond to the frequency transform performed to the input test audio signal. The output enhanced audio signal containing the extracted speech 120-1 and music waveform 120-2 may then be the in the time domain. In some cases, the processing based on the architecture of Fig.1(b) may operate on a single frame of the input audio signal, or on multiple consecutive frames at the same time. The number of consecutive frames may depend on a content of the audio signal. Further, the number of consecutive frames may be chosen such that the delay introduced by the audio enhancement may not be noticeable by users of the audio object extraction system 100 in a real-time application.

[0109] It should be noted that the architecture of the system 100 as illustrated in Fig. 1 is a non-limiting example for implementing audio object extraction as proposed in the present disclosure. While speech and music extraction are used as examples here, the system 100 can be adapted for other types of audio extraction. That is, the proposed audio object extraction approach may also be applied to process various mixed audio signals that may contain different types of audio objects other than speech / music, and that may contain different numbers of audio objects to be extracted.D25026W001

[0110] It should be further noted that the proposed audio object extraction system according to the present disclosure may employ other types of deep learning-based models for audio object extraction. For example, any DNN-based audio object extraction model utilizing a Linet structure (as described above) may benefit from the proposed model with independent decoder module architecture, as illustrated in Fig.3.[OHl] In addition, any DNN-based audio object extraction or speech enhancement model, regardless of its structure, may also benefit from the proposed residual module, as illustrated in Figs. 4 and 5, as it can be seamlessly integrated after any model architecture.

[0112] Fig. 6 is a flowchart of an example of a process 600 of training a neural network for audio processing according to embodiments of the disclosure. Process 600 may correspond to the steps performed according to training of the neural network model 104a in the audio object extraction system 100 as shown in Fig. 1(a). In some implementations, blocks of process 600 may be performed by a speech enhancement device. Alternatively, blocks of process 600 may be performed by another device, and the parameters for the trained neural network are provided to the speech enhancement device.

[0113] In S610, process 600 may receive an input audio signal comprising a plurality of audio objects of different audio types. The input audio signal may cover a plurality of frequency bands along a frequency dimension. Examples of audio types include music and speech. An additional example of an audio type is noise.

[0114] Process 600 may then extract the plurality of audio objects from the input audio signal using a neural network model. Specifically, in S620, process 600 may extract a first set of features from the input audio signal. Herein, each feature in the first set of features may correspond to a combination of a frequency band of the input audio signal and a bin along a time dimension.

[0115] In S630, process 600 may apply an encoding process to the first set of features to obtain high-level feature information indicative of one or more semantic features in the input audio signal.

[0116] In S640, process 600 may apply, for each of two or more different audio types, a respective decoding process to obtain, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal.D25026W001

[0117] In S650, process 600 may estimate, based on the reconstructed low-level feature information for each audio ty pe, a band mask associated with an audio object to be extracted from the input audio signal.

[0118] In S660, process 600 may estimate, based on each band mask, a respective second set of features for each audio type. Herein, each feature of the second set of features may correspond to a frequency bin of the input audio signal.

[0119] In S670, process 600 may perform similar steps as the previous steps as S610 and S660 to a reference audio signal corresponding to the input audio signal. That is, process 600 may receive a reference audio signal corresponding to the input audio signal, extract, for each of the two or more different audio types, a third set of features from the reference audio signal, determine a loss function based on the second sets of features and the third sets of features, and update one or more parameters of the neural network model based on the determined loss function. Herein, each feature in the third sets of features may correspond to a frequency bin of the reference audio signal.

[0120] Fig. 8 is a flowchart of an example of a process 800 of using a neural network for audio processing according to embodiments of the disclosure. Process 800 may correspond to the steps performed according to training of the neural network model 104a in Fig.6. In some implementations, blocks of process 800 may be performed by a playback device.Alternatively, blocks of process 800 may be performed by another device, and the enhanced audio signal may be provided to the playback device.

[0121] In S810, process 800 may receive an input audio signal comprising a plurality of audio objects of different audio types. The input audio signal may cover a plurality of frequency bands along a frequency dimension.

[0122] Process 800 may then extract the plurality of audio objects from the input audio signal using a neural network model. Specifically, in S820, process 800 may extract a first set of features from the input audio signal. Herein, each feature in the first set of features may correspond to a combination of a frequency band of the input audio signal and a bin along a time dimension.

[0123] In S830, process 800 may apply an encoding process to the first set of features to obtain high-level feature information indicative of one or more semantic features in the input audio signal.D25026W001

[0124] In S840, process 800 may apply, for each of two or more different audio types, a respective decoding process to obtain, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal.

[0125] In S850, process 800 may estimate, based on the reconstructed low-level feature information for each audio type, a respective band mask associated with an audio object to be extracted from the input audio signal.

[0126] In S860, process 800 may obtain the extracted audio objects based on the band masks.

[0127] Fig. 7 schematically illustrates a schematic block diagram of an example electronic device or architecture 700 (e.g., an apparatus 700) suitable for implementing example embodiments of the present disclosure. Architecture 700 includes but is not limited to servers and client devices, systems, modules and methods as described in reference to Figs. 1-6. As shown, the architecture 700 includes central processing unit (CPU) 701 which is capable of performing various processes in accordance with a program stored in, for example, read only memory (ROM) 702 or a program loaded from, for example, storage unit 708 to random access memory (RAM) 703. The CPU 701 may be, for example, an electronic processor 701, which may include one or more processor cores, and in some examples the processor 701 may be multiple processors. In RAM 703, the data used when CPU 701 performs the various processes is also stored, as required. CPU 701, ROM 702 and RAM 703 are connected to one another via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0128] The following components are connected to I / O interface 705: input unit 706, that may include a keyboard, a mouse, or the like; output unit 707 that may include a display such as a liquid crystal display (LCD) and one or more speakers; storage unit 708 including a hard disk, or another suitable storage device; and communication unit 709 which may include a network interface card such as a network card (e.g., wired or wireless).

[0129] In some implementations, input unit 706 includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g.. mono, stereo, spatial, immersive, and other suitable formats).

[0130] In some implementations, output unit 707 include systems with various number of speakers. Output unit 707 (depending on the capabilities of the host device) can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).D25026W001

[0131] In some embodiments, communication unit 709 is configured to communicate with other devices (e.g., via a network). Drive 710 is also connected to I / O interface 705, as required. Removable medium 711, such as a magnetic disk, an optical disk, a magnetooptical disk, a flash drive or another suitable removable medium is mounted on drive 710, so that a computer program read therefrom is installed into storage unit 708, as required. A person skilled in the art would understand that although apparatus 700 is described as including the above-described components, in real applications, it is possible to add, remove, and / or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.

[0132] In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, the computer program may be downloaded and mounted from the network via the communication unit 709, and / or installed from the removable medium 711, as shown in Fig. 7.

[0133] Generally, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuitry), software, logic or any combination thereof. For example, the units discussed above can be executed by control circuitry (e.g., CPU 701 in combination with other components of FIG. 7), thus, the control circuitry may be performing the actions described in this disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, a processor and / or other computing device(s), which may include control circuitry. While various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques, or methods described herein may be implemented in, as non-limiting examples, hardw are, softw are, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0134] Additionally, various blocks shown in the flowcharts may be viewed as method steps, and / or as operations that result from operation of computer program code, and / or as a plurality of coupled logic circuit elements constructed to carry out the associated function(s). ForD25026W001example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above.

[0135] In the context of the disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to one or more processors of a general-purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by one or more processors of the computer or other programmable data processing apparatus, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and / or servers.

[0137] Interpretation

[0138] A computing device implementing the techniques described above can have the following example architecture. Other architectures are possible, including architectures with more or few er components. In some implementations, the example architecture includes one or more processors (e.g., dual-core Intel® Xeon® Processors), one or more output devices (e.g., LCD), one or more network interfaces, one or more input devices (e.g., mouse, keyboard, touch-sensitive display) and one or more computer-readable mediums (e.g., RAM,D25026W001ROM, SDRAM, hard disk, optical disk, flash memory', etc.). These components can exchange communications and data over one or more communication channels (e.g., buses), which can utilize various hardware and software for facilitating the transfer of data and control signals between components.

[0139] The term ‘‘computer-readable medium” refers to a medium that participates in providing instructions to processor for execution, including without limitation, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory ) and transmission media. Transmission media includes, without limitation, coaxial cables, copper wire and fiber optics.

[0140] Computer-readable medium can further include operating system (e.g., a Linux® operating system), network communication module, audio interface manager, audio processing manager and live content distributor. Operating system can be multi-user, multiprocessing, multitasking, multithreading, real time, etc. Operating system performs basic tasks, including but not limited to: recognizing input from and providing output to network interfaces and / or devices; keeping track and managing files and directories on computer-readable mediums (e.g., memory' or a storage device); controlling peripheral devices; and managing traffic on the one or more communication channels. Network communications module includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols, such as TCP / IP, HTTP, etc.).

[0141] Architecture can be implemented in a parallel processing or peer-to-peer infrastructure or on a single device with one or more processors. Software can include multiple software components or can be a single body of code.

[0142] The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language (e.g., Objective-C, Java), including compiled or interpreted languages, and it can be deployed in any' form, including as a stand-alone program or as a module, component, subroutine, a browser-based web application, or other unit suitable for use in a computing environment.D25026W001

[0143] Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors or cores, of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory7, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magnetooptical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

[0144] To provide for interaction with a user, the features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or a retina display device for displaying information to the user. The computer can have a touch surface input device (e.g., a touch screen) or a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer. The computer can have a voice input device for receiving voice commands from the user.

[0145] The features can be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be connected by any form or medium of digital data communication such as a communication network. Examples of communication networks include, e.g., a LAN, a WAN, and the computers and networks forming the Internet.

[0146] The computing system can include clients and servers. A client and sen7er are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the clientD25026W001device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0147] A system of one or more computers can be configured to perform particular actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0148] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0149] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0150] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the present invention discussions utilizing terms such as “processing”, “computing”, “calculating”, “determining”, “analyzing” or the like, refer to the action and / or processes of a computer or computing system, or similar electronic computingD25026W001devices, that manipulate and / or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

[0151] Reference throughout this invention to “one example embodiment”, “some example embodiments” or “an example embodiment” means that a particular feature, structure or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present invention. Thus, appearances of the phrases “in one example embodiment”, “in some example embodiments” or “in an example embodiment” in various places throughout this invention are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this invention, in one or more example embodiments.

[0152] As used herein, unless otherwise specified the use of the ordinal adjectives “first”, “second”, “third”, etc., to describe a common object, merely indicate that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

[0153] Also, it is to be understood that the phraseology and terminology7used herein are for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless specified or limited otherwise, the terms “mounted”, “connected”, “supported”, and “coupled” and variations thereof are used broadly and encompass both direct and indirect mountings, connections, supports, and couplings.

[0154] In the claims below and the description herein, any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements / features that follow, but not excluding others. Thus, the term comprising, when used in the claims, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements / features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.D25026W001

[0155] It should be appreciated that in the above description of example embodiments of the present invention, various features of the present invention are sometimes grouped together in a single example embodiment. Fig., or description thereof for the purpose of streamlining the present invention and aiding in the understanding of one or more of the various inventive aspects. This method of invention, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed example embodiment. Thus, the claims following the Description are hereby expressly incorporated into this Description, with each claim standing on its own as a separate example embodiment of this invention.

[0156] Furthermore, while some example embodiments described herein include some but not other features included in other example embodiments, combinations of features of different example embodiments are meant to be within the scope of the present invention, and form different example embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed example embodiments can be used in any combination.

[0157] In the description provided herein, numerous specific details are set forth. However, it is understood that example embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been show n in detail in order not to obscure an understanding of this description.

[0158] Thus, while there has been described what are believed to be the best modes of the present invention, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the present invention, and it is intended to claim all such changes and modifications as fall within the scope of the present invention. For example, any formulas given above are merely representative of procedures that may be used. Functionality may be added or deleted from the block diagrams and operations may be interchanged among functional blocks. Steps may be added or deleted to methods described within the scope of the present disclosure.

[0159] Enumerated Example Embodiments

[0160] Various aspects and implementations of the present disclosure may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims.

[0161] EEE1. A neural netw ork based method for audio processing, the method comprising:D25026W001receiving an input audio signal comprising a plurality of audio objects of different audio types, the input audio signal covering a plurality of frequency bands along a frequency dimension; andextracting, using a neural network model, the plurality of audio objects from the input audio signal by:extracting a first set of features from the input audio signal, wherein each feature in the first set of features corresponds to a combination of a frequency band of the input audio signal and a bin along a time dimension;applying an encoding process to the first set of features to obtain high-level feature information indicative of one or more semantic features in the input audio signal;applying, for each of two or more different audio types, a respective decoding process to obtain, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal;estimating, based on the reconstructed low-level feature information for each audio type, a respective band mask associated with an audio object to be extracted from the input audio signal; andobtaining the extracted audio objects based on the band masks.

[0162] EEE2. A neural network based method for audio processing, the method comprising:receiving an input audio signal comprising a plurality of audio objects of different audio types, the input audio signal covering a plurality’ of frequency bands along a frequency dimension; andextracting, using a neural network model, the plurality of audio objects from the input audio signal by:extracting a first set of features from the input audio signal, wherein each feature in the first set of features corresponds to a combination of a frequency band of the input audio signal and a bin along a time dimension;applying an encoding process to the first set of features to obtain high-level feature information indicative of one or more semantic features in the input audio signal;applying, for each of two or more different audio types, a respective decoding process to obtain, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal;D25026W001estimating, based on the reconstructed low-level feature information for each audio type, a respective band mask associated with an audio object to be extracted from the input audio signal; andestimating, based on each band mask, a respective second set of features for each audio type, wherein each feature of the second set of features corresponds to a frequency bin of the input audio signal;receiving a reference audio signal corresponding to the input audio signal; extracting, for each of the two or more different audio types, a third set of features from the reference audio signal, wherein each feature in the third sets of features corresponds to a frequency bin of the reference audio signal;determining a loss function based on the second sets of features and the third sets of features; andupdating one or more parameters of the neural network model based on the determined loss function.

[0163] EEE3. The method of EEE1, wherein obtaining the extracted audio objects based on the band masks comprises:estimating, based on each band mask, a respective second set of features, each feature of the second set of features corresponding to a frequency bin of the input audio signal; and obtaining the extracted audio objects based on the estimated second sets of features.

[0164] EEE4. The method of any one of the preceding EEEs, further comprising, for each of the plurality of audio objects, performing a residual operation subsequent to the respective decoding process, the residual operation comprising one or more sequential residual blocks each including one or more temporal convolution networks to increase receptive fields in the reconstructed low-level feature information.

[0165] EEE5. The method of EEE4, wherein a temporal convolution network of the one or more temporal convolution networks comprises a first one-by-one convolutional layer to compress a channel dimension, a dilated convolutional layer to expand the receptive fields in the time dimension, a last one-by-one convolutional layer to restore the compressed channel dimension, and a residual connection between an input and an output of the temporal convolution network.D25026W001

[0166] EEE6. The method of EEE 5, wherein the one or more temporal convolution networks are implemented with increasing dilation factors for the dilated convolutional layer in dependence on a number of the temporal convolution networks.

[0167] EEE7. The method of EEE5 or EEE6, wherein the temporal convolution network further comprises a first batch normalization layer and a first non-linear activation layer following the first one-by-one convolutional layer, and a last batch normalization layer and a last non-linear activation layer following the last one-by-one convolutional layer.

[0168] EEE8. The method of any one of the preceding EEEs, further comprising applying an additional residual operation between the encoding process and the respective decoding processes.

[0169] EEE9. The method of any one of the preceding EEEs, further comprising applying a down-sampling operation at the encoding process and / or an up-sampling operation at the respective decoding processes.

[0170] EEE10. The method of any one of the preceding EEEs, wherein the neural netw ork model further comprises one or more convolutional neural network (CNN) layers following the respective decoding processes.

[0171] EEE11. The method of any one of the preceding EEEs, wherein the neural netw ork model comprises a deep neural network (DNN) model.

[0172] EEE12. The method of EEE1 or any one of EEE3 to EEE10 when depending on EEE1, wherein the extracted audio objects are obtained by applying an inverse transformation to the estimated second set of features.

[0173] EEE13. The method of any one of the preceding EEEs, further comprising, prior to obtaining the first set of features:transforming the input audio signal into the frequency domain to obtain a transformed audio signal;extracting a bin feature from the transformed audio signal for each frequency bin in the frequency domain; andfor each frequency band of the input audio signal, combining features of the bin features corresponding to frequency bins inside the frequency band to obtain band features of the plurality of audio objects for the extraction.D25026W001

[0174] EEE14. The method of EEE 13, wherein transforming the input audio signal into the frequency domain is performed by any one of a short time Fourier transform (STFT). a modified discrete cosine transform (MDCT), a shifted discrete frequency transform (MDXT), or a filter bank based transform.

[0175] EEE15. The method of EEE 13 or 14, wherein combining features of the bin features corresponding to frequency bins inside the frequency band to obtain band features of the plurality of audio objects comprises weighting the features of the bin features corresponding to frequency bins inside the frequency band.

[0176] EEE16. The method of any one of the preceding EEEs, wherein the input audio signal contains a mixture of speech and music, and the plurality of audio objects comprises a first audio object of a speech ty pe and a second audio object of a music type.

[0177] EEE17. A computer system for audio processing based on a neural network model, comprising:a memory;one or more processors coupled with the memory and configured to perform: receiving an input audio signal comprising a plurality of audio objects of different audio ty pes, the input audio signal covering a plurality of frequency bands along a frequency¬ dimension; andextracting, using the neural network model, the plurality of audio objects from the input audio signal, the neural network model comprising:a feature extraction module for extracting a first set of features from the input audio signal, wherein each feature in the first set of features corresponds to a combination of a frequency band of the input audio signal and a bin along a time dimension;an encoder module for obtaining, from the first set of features, high-level feature information indicative of one or more semantic features in the input audio signal; anda plurality of decoder modules each associated with a respective audio type for obtaining, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal; and estimating, based on the reconstructed low-level feature information for each audio type, a band mask associated with an audio object to be extracted from the input audio signal.D25026W001

[0178] EEE18. The system of EEE17, wherein, at an inference stage, the one or more processors are further configured to perform obtaining the extracted audio objects based on the band masks, orwherein, at a training stage, the one or more processors are further configured to perform the following:extracting a third set of features from the reference audio signal, wherein each feature in the third set of features corresponds to a frequency bin of the reference audio signal receiving a reference audio signal corresponding to the input audio signal;determining a loss function based on the second set of features and the third set of features; andupdating one or more parameters of the neural network model based on the determined loss function.

[0179] EEE19. The system of EEE18, wherein obtaining the extracted audio objects based on the band masks comprises:estimating, based on each band mask, a respective second set of features, each feature of the second set of features corresponding to a frequency bin of the input audio signal; and obtaining the extracted audio objects based on the estimated second sets of features.

[0180] EEE20. The system of any one of EEE17 to EEE19, wherein the neural network model further comprises a plurality of residual modules each associated with a respective audio type, said each residual module receiving an output from the respective decoder module, said each residual module comprising one or more sequential residual blocks each including one or more temporal convolution networks to increase receptive fields in the reconstructed low-level feature information.

[0181] EEE21. The system of EEE20, wherein a temporal convolution network of the one or more temporal convolution networks comprises a first one-by-one convolutional layer to compress a channel dimension, a dilated convolutional layer to expand the receptive fields in the time dimension, a last one-by-one convolutional layer to restore the compressed channel dimension, and a residual connection between an input and an output of the temporal convolution network.

[0182] EEE22. The system of EEE21, wherein the one or more temporal convolution networks are implemented with increasing dilation factors for the dilated convolutional layer in dependence of a number of the temporal convolution networks.D25026W001

[0183] EEE23. The system of EEE21 or EEE22, wherein the temporal convolution network further comprises a first batch normalization layer and a first non-linear activation layer following the first one-by-one convolutional layer, and a last batch normalization layer and a last non-linear activation layer following the last one-by-one convolutional layer.

[0184] EEE24. The system of any one of EEE17 to EEE23, wherein the neural network model further comprises an additional residual module between the encoder module and the respective decoder modules.

[0185] EEE25. The system of any one of EEE17 to EEE24, wherein the neural network model comprises a deep neural network (DNN) model.

[0186] EEE26. The system of EEE25, wherein the DNN comprises an up-sampling module for estimating a bin mask for said each audio object present in each frequency bin of the input audio signal.

[0187] EEE27. The system of EEE25 or EEE26, wherein the DNN further comprises the feature extraction module, followed by the encoder module, followed by the respective decoder modules, and a plurality of convolutional neural network (CNN) layers, wherein each of the decoder modules is followed by the up-sampling module, which is followed by a respective CNN layer of the plurality of CNN layers.

[0188] EEE28. The system of any one of EEE17 to EEE27, wherein the encoder module comprises at least one down-sample layer and a plurality of CNN layers, and wherein each of the decoder modules comprises at least one up-sample layer and a plurality of CNN layers.

[0189] EEE29. The system of any one of EEE 17 to EEE28, wherein the neural network has been trained based on a pair of a clean signal and a corresponding degraded signal.

[0190] EEE30. The system of any one of EEE17 to EEE29, wherein the neural network has been trained based on a loss function.

[0191] EEE31. The system of any one of EEE17 to EEE30, wherein the input audio signal contains a mixture of speech and music, and the plurality of audio objects comprises a first audio object of a speech type and a second audio object of a music type.

[0192] EEE32. A program comprising instructions that, when executed by a processor, cause the processor to cany' out the method according to any one of EEE1 to EEE16.D25026W001

[0193] EEE33. A computer-readable storage medium storing the program according to EEE32.

[0194] EEE-B1. A deep learning-based audio object extraction system with a novel model architecture, comprising:a U-net structure with a shared encoder module to extract high-level information from the mixture and independent decoder modules to reconstruct low-level features for each specific audio object; andresidual modules incorporating temporal convolutional networks to increase the model’s receptive field in the time dimension.

[0195] EEE-B2. The system of EEE-B1, comprising a shared encoder module and independent decoder modules for each audio object, wherein the output of the encoder is fed as input into the independent decoders.

[0196] EEE-B3. The system of EEE-B1 or EEE-B2, wherein residual modules are employed after the decoder modules for each specific audio object.

[0197] EEE-B4. The system of EEE-B2, wherein residual connections are also established between the encoder module and independent decoder modules.

[0198] EEE-B5. The system of EEE-B3, wherein each residual module consists of one or more repeated residual blocks.

[0199] EEE-B6. The system of EEE-B5, wherein, each residual block includes one or more temporal convolution networks with increasing dilation factors, such as 1, 2, 4, ... , 2M, where M represents the number of the temporal convolutional network within each residual block.

[0200] EEE-B7. The system of EEE-B6, wherein each temporal convolutional network comprises a 1x1 convolutional layer, a dilated convolutional layer, an output 1x1 convolutional layer, and a residual connection between input and output. Each convolutional layer is followed by a batch normalization layer and a non-linear activation layer.

[0201] EEE-C1. A deep learning-based audio object extraction system for extracting a plurality of audio objects from an audio signal, the system comprising:an encoder module configured to extract high-level features from the audio signal; a plurality of decoder modules, wherein each decoder module of the plurality ofD25026W001decoder modules is configured to reconstruct low-level features of an audio object of the plurality of audio objects based on the high-level features; anda plurality’ of residual modules, wherein each residual module of the plurality of residual modules outputs an audio object mask in response to receiving the low-level features of an audio object of the plurality of audio objects.

[0202] EEE-C2. The system of EEE-C1, wherein each residual module is configured to increase a receptive field.

[0203] EEE-C3. The system of EEE-C1 or EEE-C2, further comprising a plurality of residual connections, wherein each residual connection of the plurality of residual connections is established between the encoder module and a decoder module of the plurality of decoder modules.

[0204] EEE-C4. The system of any one of EEE-C1 to EEE-C3, wherein the audio signal comprises a mixture of speech, noise, and / or music.

[0205] EEE-C5. The system of any one of EEE-C1 to EEE-C4, wherein each residual module of the plurality of residual modules consists of one or more repeated residual blocks.

[0206] EEE-C6. The system of EEE-C5, wherein each residual block includes one or more temporal convolution networks with increasing dilation factors, such as 1, 2, 4, 2M, where M represents a plurality of temporal convolutional networks within each residual block.

[0207] EEE-C7. The system of EEE-C6, wherein each temporal convolutional network of the plurality of temporal convolutional networks comprises a 1x1 convolutional layer, a dilated convolutional layer, an output 1x1 convolutional layer, and a residual connection.

[0208] EEE-C8. The system of EEE-C7, wherein each convolutional layer is followed by a batch normalization layer and a non-linear activation layer.

[0209] EEE-C9. A method for extracting a plurality of audio objects from an audio signal via a deep learning-based audio object extraction model, the method comprising:extracting, via an encoder module, high-level features from the audio signal; reconstructing, via each decoder module of a plurality of decoder modules, low-level features of an audio object of the plurality of audio objects based on the high-level features; andD25026W001outputing, via each residual module of a plurality of residual modules, an audio object mask in response to receiving the low-level features of an audio object of the plurality of audio objects.

[0210] EEE-C10. An apparatus comprising a processor and a memory’, configured to perform the method according to EEE-C9.

[0211] EEE-C11. A computer program product comprising instructions which, when the program is executed by a computer, causes the computer to carry out the method according to EEE-C9.

[0212] EEE-C12. A computer-readable storage medium storing the computer program product according to EEE-C 11.

Claims

D25026W001CLAIMS1. A neural network-based method for audio processing, the method comprising: receiving an input audio signal comprising a plurality of audio objects of different audio types, the input audio signal covering a plurality of frequency bands along a frequency dimension; andextracting, using a neural network model, the plurality of audio objects from the input audio signal by:extracting a first set of features from the input audio signal, wherein each feature in the first set of features corresponds to a combination of a frequency band of the input audio signal and a bin along a time dimension;applying an encoding process to the first set of features to obtain high-level feature information indicative of one or more semantic features in the input audio signal;applying, for each of two or more different audio t pes, a respective decoding process to obtain, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal;estimating, based on the reconstructed low-level feature information for each audio type, a respective band mask associated with an audio object to be extracted from the input audio signal; andobtaining the extracted audio objects based on the band masks.

2. A neural network-based method for audio processing, the method comprising: receiving an input audio signal comprising a plurality of audio objects of different audio types, the input audio signal covering a plurality of frequency bands along a frequency dimension;extracting, using a neural network model, the plurality of audio objects from the input audio signal by:extracting a first set of features from the input audio signal, wherein each feature in the first set of features corresponds to a combination of a frequency band of the input audio signal and a bin along a time dimension;applying an encoding process to the first set of features to obtain high-level feature information indicative of one or more semantic features in the input audio signal;D25026W001applying, for each of two or more different audio pes, a respective decoding process to obtain, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal;estimating, based on the reconstructed low-level feature information for each audio ty pe, a band mask associated yvith an audio object to be extracted from the input audio signal; andestimating, based on each band mask, a respective second set of features for each audio type, wherein each feature of the second set of features corresponds to a frequency bin of the input audio signal;receiving a reference audio signal corresponding to the input audio signal; extracting, for each of the two or more different audio types, a third set of features from the reference audio signal, wherein each feature in the third sets of features corresponds to a frequency bin of the reference audio signal;determining a loss function based on the second sets of features and the third sets of features; andupdating one or more parameters of the neural network model based on the determined loss function.

3. The method according to claim 1, wherein obtaining the extracted audio objects based on the band masks comprises:estimating, based on each band mask, a respective second set of features, each feature of the second set of features corresponding to a frequency bin of the input audio signal; and obtaining the extracted audio objects based on the estimated second sets of features.

4. The method of any one of claims 1 to 3. further comprising, for each of the plurality of audio objects, performing a residual operation subsequent to the respective decoding process, the residual operation comprising one or more sequential residual blocks each including one or more temporal convolution networks to increase receptive fields in the reconstructed low-level feature information.

5. The method of claim 4, wherein a temporal convolution network of the one or more temporal convolution networks comprises a first one-by-one convolutional layer to compress a channel dimension, a dilated convolutional layer to expand the receptive fields in the time dimension, a last one-by-one convolutional layer to restore the compressed channelD25026W001dimension, and a residual connection between an input and an output of the temporal convolution network.

6. The method of claim 5, wherein the one or more temporal convolution networks are implemented with increasing dilation factors for the dilated convolutional layer in dependence on a number of the temporal convolution networks.

7. The method of claim 5 or 6, wherein the temporal convolution network further comprises a first batch normalization layer and a first non-linear activation layer following the first one-by-one convolutional layer, and a last batch normalization layer and a last non-linear activation layer following the last one-by-one convolutional layer.

8. The method of any one of the preceding claims, further comprising applying an additional residual operation between the encoding process and the respective decoding processes.

9. The method of any one of the preceding claims, further comprising applying a down-sampling operation at the encoding process and / or an up-sampling operation at the respective decoding processes.

10. The method of any one of the preceding claims, wherein the neural network model further comprises one or more convolutional neural network, CNN, layers following the respective decoding processes.

11. The method of any one of the preceding claims, wherein the neural network model comprises a deep neural network, DNN, model.

12. The method of claim 1 or any one of claims 3 to 10 when depending on claim 1, wherein the extracted audio objects are obtained by applying an inverse transformation to the estimated second set of features.

13. The method of any one of the preceding claims, further comprising, prior to obtaining the first set of features:D25026W001transforming the input audio signal into the frequency domain to obtain a transformed audio signal;extracting a bin feature from the transformed audio signal for each frequency bin in the frequency domain; andfor each frequency band of the input audio signal, combining features of the bin features corresponding to frequency bins inside the frequency band to obtain band features of the plurality’ of audio objects for the extraction.

14. The method of claim 13, wherein transforming the input audio signal into the frequency domain is performed by any one of a short time Fourier transform, STFT, a modified discrete cosine transform, MDCT, a shifted discrete frequency transform, MDXT, or a filter bank based transform.

15. The method of claim 13 or 14, wherein combining features of the bin features corresponding to frequency bins inside the frequency band to obtain band features of the plurality of audio objects comprises weighting the features of the bin features corresponding to frequency bins inside the frequency band.

16. The method of any one of the preceding claims, wherein the input audio signal contains a mixture of speech and music, and the plurality of audio objects comprises a first audio object of a speech type and a second audio object of a music type.

17. A computer system for audio processing based on a neural network model, comprising:a memory;one or more processors coupled with the memory and configured to perform: receiving an input audio signal comprising a plurality of audio objects of different audio types, the input audio signal covering a plurality of frequency bands along a frequency dimension; andextracting, using the neural network model, the plurality of audio objects from the input audio signal, the neural network model comprising:a feature extraction module for extracting a first set of features from the input audio signal, wherein each feature in the first set of features corresponds to a combination of a frequency band of the input audio signal and a bin along a time dimension;D25026W001an encoder module for obtaining, from the first set of features, high-level feature information indicative of one or more semantic features in the input audio signal; and a plurality of decoder modules each associated with a respective audio type for obtaining, based on the high-level feature information, reconstructed low-level feature information indicative of one or more acoustic features in the input audio signal; and estimating, based on the reconstructed low-level feature information for each audio type, a band mask associated with an audio object to be extracted from the input audio signal.

18. The system of claim 17, wherein, at an inference stage, the one or more processors are further configured to perform obtaining the extracted audio objects based on the band masks, orwherein, at a training stage, the one or more processors are further configured to perform the following:extracting a third set of features from the reference audio signal, wherein each feature in the third set of features corresponds to a frequency bin of the reference audio signal receiving a reference audio signal corresponding to the input audio signal;determining a loss function based on the second set of features and the third set of features; andupdating one or more parameters of the neural network model based on the determined loss function.

19. The system of claim 18, wherein obtaining the extracted audio objects based on the band masks comprises:estimating, based on each band mask, a respective second set of features, each feature of the second set of features corresponding to a frequency bin of the input audio signal: and obtaining the extracted audio objects based on the estimated second sets of features.

20. The system of any one of claims 17 to 19, wherein the neural network model further comprises a plurality of residual modules each associated with a respective audio type, said each residual module receiving an output from the respective decoder module, said each residual module comprising one or more sequential residual blocks each including one or more temporal convolution networks to increase receptive fields in the reconstructed low-level feature information.D25026W00121. The system of claim 20, wherein a temporal convolution network of the one or more temporal convolution networks comprises a first one-by-one convolutional layer to compress a channel dimension, a dilated convolutional layer to expand the receptive fields in the time dimension, a last one-by-one convolutional layer to restore the compressed channel dimension, and a residual connection between an input and an output of the temporal convolution network.

22. The system of claim 21, wherein the one or more temporal convolution networks are implemented with increasing dilation factors for the dilated convolutional layer in dependence of a number of the temporal convolution networks.

23. The system of claim 21 or 22, wherein the temporal convolution network further comprises a first batch normalization layer and a first non-linear activation layer following the first one-by-one convolutional layer, and a last batch normalization layer and a last non-linear activation layer following the last one-by-one convolutional layer.

24. The system of any one of claims 17 to 23, wherein the neural network model further comprises an additional residual module between the encoder module and the respective decoder modules.

25. The system of any one of claims 17 to 24, wherein the neural network model comprises a deep neural network, DNN, model.

26. The system of claim 25, wherein the DNN comprises an up-sampling module for estimating a bin mask for said each audio object present in each frequency bin of the input audio signal.

27. The system of claim 25 or 26, wherein the DNN further comprises the feature extraction module, followed by the encoder module, followed by the respective decoder modules, and a plurality of convolutional neural network, CNN, layers, wherein each of the decoder modules is followed by the up-sampling module, which is followed by a respective CNN layer of the plurality of CNN layers.D25026W00128. The system of any one of claims 17 to 27, wherein the encoder module comprises at least one down-sample layer and a plurality of CNN layers, and wherein each of the decoder modules comprises at least one up-sample layer and a plurality of CNN layers.

29. The system of any one of claims 17 to 28, wherein the neural network has been trained based on a pair of a clean signal and a corresponding degraded signal.

30. The system of any one of claims 17 to 29, wherein the neural network has been trained based on a loss function.

31. The system of any one of claims 17 to 30, wherein the input audio signal contains a mixture of speech and music, and the plurality of audio objects comprises a first audio object of a speech type and a second audio object of a music type.

32. A program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1 to 16.

33. A computer-readable storage medium storing the program according to claim 32.

Images