Method and device for generating processed audio waveforms

Abstract

The present disclosure describes a method and a device for generating processed audio waveforms. The method comprises receiving, from one or more sources, an input audio waveform corresponding to a target audio waveform; generating, using a encoder model, a low-dimensional latent space representation corresponding to the input audio waveform modifying the low-dimensional latent space representation for generating one or more modified latent space representations corresponding to one or more audio features associated with the input audio waveform; generating, using a decoder model, one or more processed audio waveforms based on the one or more modified latent space representations; and generating, using a first adaptor model, a first target audio waveform for a first audio processing task associated with the first adaptor model, based on at least one of the one or more processed audio waveforms.

Description

METHOD AND DEVICE FOR GENERATING PROCESSED AUDIO WAVEFORMS

[0001] The present disclosure generally relates to the field of audio signal processing, and more particularly relates to building an Artificial Intelligence (AI) based-unified foundation model for audio signal processing.

[0002] In recent times, audio / video editing, restoration, and calling applications are being widely used for connecting people through calling, multi-media content sharing on social media sites, podcasts, vlogs etc. Currently most existing mobile devices have multiple Machine Learning (ML)-based models and different applications for performing different audio processing tasks such as separation, enhancement, in-painting for audio / video editing. Most of the above models perform similar processing steps such as feature extraction, core ML processing, post-processing, and the like.

[0003] These different ML models for each task such as separation, enhancement running sequentially for editing, restoration etc., lead to higher computational requirements, power consumption, and latency. This, therefore, results in overall low computational efficiency due to high redundancy, higher power consumption, and increased on-device latency. The overall low-computational efficiency leads to poor user experience due to frequent battery drain, overheating, and excessive application latency.

[0004] Further, the state-of-the-art ML models rely on generating tokens from additional modalities such as text and images / video, together with audio which increases the computational complexity and requires multi-modal datasets with high quality matched audio-visual or audio-text pairs for training.

[0005] The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.

[0006] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

[0007] In one non-limiting embodiment of the present disclosure, a method of generating processed audio signals is disclosed. The method comprises receiving, from one or more sources, an input audio waveform. The method further comprises generating, using an encoder model, a low-dimensional latent space representation corresponding to the input audio waveform. The method further comprises modifying the low-dimensional latent space representation for generating one or more modified latent space representations corresponding to one or more audio features associated with the input audio waveform. The method further comprises generating, using a decoder model, one or more processed audio waveforms based on the one or more modified latent space representations. In an exemplary aspect, each of the one or more processed audio waveforms corresponds to one or more features among the one or more audio features associated with the input audio waveform. The method further comprises generating, using a first adaptor model among the one or more adaptor models, a first target audio waveform for a first audio processing task associated with the first adaptor model, based on at least one of the one or more processed audio waveforms.

[0008] In an non-limiting embodiment of the present disclosure, a device for generating processed audio signals is disclosed. The device comprises memory comprising one or more storage mediums storing one or more instructions and processor comprising a processing circuitry and being operatively coupled to the memory. The one or more instructions may cause, when executed by the processor individually or collectively, the device to to receive, from one or more sources, an input audio waveform. The one or more instructions may cause, when executed by the processor individually or collectively, the device to generate, using an encoder model, a low-dimensional latent space representation corresponding to the input audio waveform. The one or more instructions may cause, when executed by the processor individually or collectively, the device to modify the low-dimensional latent space representation for generating one or more modified latent space representations corresponding to one or more audio features associated with the input audio waveform. The one or more instructions may cause, when executed by the processor individually or collectively, the device to generate, using a decoder model, one or more processed audio waveforms based on the one or more modified latent space representations. In an exemplary aspect, the one or more processed audio waveforms corresponds to one or more features among the one or more audio features associated with the input audio waveform. The one or more instructions may cause, when executed by the processor individually or collectively, the device to generate using a first adaptor model among the one or more adaptor models, a first target audio waveform for a first audio processing task associated with the first adaptor model, based on at least one of the one or more processed audio waveforms.

[0009] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

[0010] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of at least one of device and methods in accordance with embodiments of the present subject matter are now described, by way of example only, and with reference to the accompanying figures, in which:

[0011] Figure 1A depicts an exemplary environment for implementation of the proposed foundation model for processing audio signals, in accordance with an embodiment of the present disclosure,

[0012] Figure 1B depicts an architectural diagram of the proposed foundation model for processing audio signals, in accordance with an embodiment of the present disclosure,

[0013] Figure 2 illustrates an exemplary block diagram of a device for generating processed audio signals, in accordance with an embodiment of the present disclosure,

[0014] Figures 3A and 3B depict architectural diagrams, respectively of training an encoder-decoder network in accordance with embodiments of the present disclosure,

[0015] Figure 4A depicts an architectural diagram for modification of low-dimensional latent space representation during a training phase, using clustering technique, in accordance with an embodiment of the present disclosure,

[0016] Figure 4B depicts an architectural diagram for modification of low-dimensional latent space representation during inference, in accordance with an embodiment of the present disclosure,

[0017] Figure 5A depicts an architectural diagram for implementation of a first method of modification of low-dimensional latent space representation during a training phase, using a generative model, in accordance with an embodiment of the present disclosure,

[0018] Figure 5B depicts an architectural diagram for implementation of a second method of modification of low-dimensional latent space representation during a training phase, using a generative model, in accordance with an embodiment of the present disclosure,

[0019] Figure 5C depicts an architectural diagram for modification of low-dimensional latent space representation during inference, using a generative model, in accordance with an embodiment of the present disclosure,

[0020] Figure 6 depicts an architectural diagram for training a foundation model based on an end-to-end approach, in accordance with an embodiment of the present disclosure,

[0021] Figures 7A-7E depict architectural diagrams of an adaptor model for generating finetuned separated audio signals, in accordance with some embodiments of the present disclosure,

[0022] Figure 8A depicts architectural diagram of an adaptor model for generating finetuned enhanced and / or in-painted audio signals during a training phase, in accordance with some embodiments of the present disclosure,

[0023] Figure 8B depicts architectural diagram of an adaptor model for generating finetuned enhanced and / or in-painted audio signals during inference, in accordance with some embodiments of the present disclosure,

[0024] Figure 9 depicts, by way of a flowchart, an exemplary method for generating processed audio signals, in accordance with an embodiment of the present disclosure, and

[0025] Figure 10 illustrates a block diagram of an exemplary computer system for implementing embodiments consistent with the present disclosure.

[0026] The figures depict embodiments of the disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the disclosure described herein.

[0027] Existing mobile devices utilize several different Machine Learning (ML) models for audio enhancement, eraser, and audio in-painting, which are often triggered sequentially during audio / video editing and audio restoration applications. Most of the above models perform similar processing steps such as feature extraction, core ML processing, post-processing etc.

[0028] These different ML models for each task such as separation, enhancement running sequentially for editing, restoration etc., lead to higher computational requirements, power consumption, and latency. Therefore, this results in low overall computational efficiency due to high redundancy, higher power consumption, and increased on-device latency. This further leads to bad user experiences due to frequent battery drain, overheating, excessive application latency.

[0029] Further, the state-of-the-art ML models rely on generating tokens from additional modalities such as text and images / video, together with audio which increases the computational complexity and requires multi-modal dataset with high quality matched audio-visual or audio-text pairs for training. Therefore, several ML models require additional modality such as text / vision in addition to audio for training and inference.

[0030] Thus, there exists a need to overcome the above-mentioned limitations of the current technology and provide technique(s) for enhancing the audio quality using audio signals alone.

[0031] The present disclosure describes a system and a method for building a foundation artificial intelligence (AI) model for audio signal processing that uses only audio signals as inputs. In the present disclosure, novel technique for utilizing foundation models using audio signals only as inputs to learn general audio signal representations and adaptor models to fine-tune performance for specific tasks such as audio separation model, enhancement and in-painting applications is described. In the present disclosure, the combination of encoder-decoder and generative models or clustering algorithms for deriving the low-dimensional compact audio signal representations is described. The encoder blocks learn the low-dimensional audio signal latent representations in a way to generalize across different downstream tasks using shared weights or weighted average of learned weights. The modification of the latent space using generative / clustering algorithms is done in a way to derive optimal and efficient latent space representations such that they are easily used for all the target downstream tasks. The decoder is used for generating audio waveforms using the optimal latent space representations, with the psychoacoustic metric-based loss functions to achieve similar overall audio quality as target audio signals. The adaptor models are used to further fine-tune the outputs based on target application.

[0032] The present disclosure provides a single unified foundational model which generates target audio signals for several downstream applications with only degraded audio signals as input. The downstream applications such as audio editing, restoration, calling etc. is easily expanded to other applications such as speech style transfer using the same foundation model architecture by modifying the adaptor models. The source separation U-net based on adaptor model architectures, of the present disclosure achieves >3 dB average improvement in terms of Scale Invariant Signal-to-Noise (SNR).

[0033] The present disclosure provides a foundation model based on audio signal inputs only and multiple encoder-decoder architecture with modification of learned audio representations using generative / clustering-based algorithms and psychoacoustic-metric based loss functions to generate audio waveforms with similar audio quality as target audio which is fine-tuned for downstream tasks using different adaptor models easily for different audio processing tasks.

[0034] In one embodiment of the present disclosure, a method for building a foundation AI model for audio signal processing is disclosed. The method comprises feeding as input, a plurality of raw audio waveforms into an encoder. The method further comprises generating using the encoder, a single low-dimensional latent space representation of the raw audio waveforms. The method further comprises transforming using a diffusion model, the single low-dimensional latent space representation of the raw audio waveforms into multiple low-dimensional target latent space representations, each target latent space representative of distinct audio features present in each of the input audio waveforms. The method further comprises generating using a decoder, output audio waveforms having distinct audio features that are substantially closer to the target audio waveforms.

[0035] In an exemplary embodiment, a method for generating processed audio signals from degraded audio mixtures corresponding to target audio source signals is described. The method comprises receiving as an input, training data comprising a plurality of raw audio waveforms corresponding to target audio waveforms in the input audio waveforms. The method further comprises transforming using an encoder model, each of the raw audio into a low-dimensional latent space representation. The method further comprises generating one or more target latent space representations corresponding to distinct audio features present in each of the input audio waveforms, by modifying the low dimensional latent space representations using a generative or clustering model. The method further comprises generating using a decoder model, processed audio waveforms corresponding to distinct audio features of the input audio waveforms, wherein the generated processed audio waveforms are perceptually similar to the target audio waveforms.

[0036] In an embodiment of the present disclosure, the processed audio waveforms are generated using psychoacoustic-based loss functions.

[0037] In an embodiment of the present disclosure, the processed audio waveforms are used to further train one or more task-specific adaptor models.

[0038] In recent times, audio / video editing, restoration, and calling applications are being widely used for connecting people through calling, multi-media content sharing on social media sites, podcasts, vlogs etc. Currently most existing mobile devices have multiple ML-based models and different applications for performing different audio processing tasks such as separation, enhancement, in-painting for audio / video editing, restoration and calling applications.

[0039] However, there is a lot of redundancy in terms of processing blocks, types of features extracted, derivation and modification of latent space across the different audio processing tasks / target apps etc. which leads to increase in computational complexity, power inefficiency and latency for the mobile device applications. From user perspective, they have to switch between different applications to perform the audio processing tasks, and even if a single application supports multiple processing tasks, application and switching between different models lead to higher waiting time, heating and lag issues for users leading to poor experience.

[0040] Further, the several AI models either utilize separate discriminative AI models for separation, enhancement and in-painting tasks or utilize the text / image / video-based input signals together with audio signals to derive tokens for generative AI-based audio generation tasks.

[0041] For instance, the current separation and enhancement models in literature comprise of encoder, separation / enhancement network and decoder, which estimate and apply a mask for separation or enhancement of input signals. These methods mostly work well for specific sound-classes such as music, speech etc. but not for mixtures of various sound classes such as different background noise, speech, music, wind etc.

[0042] The universal sound source separation models have been implemented recently based on dilated convolutions and U-net architectures which achieve better accuracy for sound separation for different sound class mixtures, however they suffer from poor generalization for separation performance for equal to or more than 3 source classes, in presence of reverberation, low signal-to-noise ratio especially below 10-15 dB, high spectral and temporal overlap between signals greater than 60-70%. Additionally, the enhancement networks also suffer from poor performance in challenging conditions such as low Signal-to-Noise (SNR), high reverberation, and high spectral correlations between the target signals and noise signals.

[0043] The existing ML models typically utilize multi-modal features such as text / video / images together with audio inputs to generate tokens and processed further to derive a latent space representation. However, the processing of multiple input modalities and multi-modal based learned representation often leads to high computational complexity, large number of Neural network (NN) parameters and higher storage space requirements.

[0044] It would be appreciated that the desired target audio signal quality is similar for several audio processing tasks, namely, source separation, enhancement, in-painting etc. across different apps such as separation, enhancement and in-painting (target: target audio signals without any leakage / distortion / missing samples). Hence, a learned optimal latent space representation using an encoder-decoder combined with generative / clustering approach to modify the latent space derives efficient embedding vectors, which is used to generate estimated target signals which have similar audio quality as target audio signals and is further processed and fine-tuned for several downstream tasks and applications as desired. The techniques of the present disclosure utilize this insight for training of audio signals and for deriving inference of foundation and adaptor models for several downstream tasks. Specifically, the techniques of the present disclosure utilize generative or clustering based algorithms to derive efficient and optimal latent space representations from audio signals.

[0045] The technique of the present disclosure utilizes a combination of learned and derived psychoacoustic based loss functions with decoupled computation of interference and leakage to generate signals which have similar overall perceptual audio quality to the target audio signals.

[0046] The present disclosure describes a system, and a method provide an audio foundation model for enhancing the audio quality and uses only audio signals as inputs. In the present disclosure, novel technique for utilizing foundation models using audio signals only as inputs to learn general audio signal representations and adaptor models to fine-tune performance for specific tasks such as audio separation model, enhancement and in-painting applications. In the present disclosure, the combination of encoder-decoder and generative model such as generative models or clustering algorithms for deriving the low-dimensional compact audio signal representations. The encoder blocks learn the low-dimensional audio signal latent representations in a way to generalize across different downstream tasks using shared weights or weighted average of learned weights. The modification of the latent space using generative / clustering algorithms is done in a way to derive optimal and efficient latent space representations such that they are easily used for all the target downstream tasks. The decoder is used for generating audio waveforms using the optimal latent space representations, with the psychoacoustic metric-based loss functions to achieve similar overall audio quality as target audio signals. The adaptor models are used to further fine-tune the outputs based on target application.

[0047] The present disclosure provides a single unified foundational model which generates target audio signals for several downstream applications with degraded audio signals only as input. The downstream applications such as audio editing, restoration, calling etc. are easily expanded further to other applications such as speech style transfer using the same foundation model architecture by modifying the adaptor models. The source separation U-net based on adaptor model architectures, of the present disclosure achieves more than 3 dB average improvement in terms of Scale Invariant Signal-to-Noise (SNR).

[0048] The present disclosure provides a foundation model based on audio signal inputs only and multiple encoder-decoder architecture with modification of learned audio representations using generative / clustering-based algorithms and psychoacoustic-metric based loss functions to generate audio waveforms with similar audio quality as target audio which is fine-tuned for downstream tasks using different adaptor models easily for different audio processing tasks.

[0049] One exemplary use case for the techniques disclosed in the present disclosure is editing of recorded audio / video signals by allowing users to enhance / remove certain sources, add missing samples for audio signals in a single application.

[0050] In one embodiment of the present disclosure, a method for building a foundation AI model for audio signal processing is disclosed. The method further comprises feeding as input, a plurality of raw audio waveforms into an encoder. The method further comprises generating using the encoder, a single low-dimensional latent space representation of the raw audio waveforms. The method further comprises transforming, using a diffusion model, the single low-dimensional latent space representation of the raw audio waveforms into multiple low-dimensional target latent space representations, each target latent space representative of distinct audio features present in each of the input audio waveforms. The method further comprises generating using a decoder, output audio waveforms having distinct audio features that are substantially closer to the target audio waveforms.

[0051] The present disclosure provides modification of the low-dimensional latent space using deep clustering or generative algorithms in a way to derive optimal and efficient representations which is easily used for several target downstream tasks.

[0052] In one embodiment, the present disclosure utilizes a decoder model for converting latent space representations to audio waveforms using psychoacoustic metric-based loss functions, which help to derive estimated audio signals with similar overall perceptual quality as target audio signals.

[0053] In an exemplary embodiment, a method for generating processed audio signals from degraded audio mixtures corresponding to target audio source signals is described. The method comprises receiving as an input, training data comprising a plurality of raw audio waveforms corresponding to target audio waveforms in the input audio waveforms. The method further comprises transforming using an encoder model, each of the raw audio into a low-dimensional latent space representation. The method further comprises generating one or more target latent space representations corresponding to distinct audio features present in each of the input audio waveforms, by modifying the low dimensional latent space representations using a generative or clustering model. The method further comprises generating using a decoder model, processed audio waveforms corresponding to distinct audio features of the input audio waveforms, wherein the generated processed audio waveforms are perceptually similar to the target audio waveforms.

[0054] In an embodiment of the present disclosure, the processed audio waveforms are generated using psychoacoustic-based loss functions.

[0055] In an embodiment of the present disclosure, the processed audio waveforms are used to further train one or more task-specific adaptor models.

[0056] In one embodiment, a method for audio quality enhancement using foundation model, which takes audio signal as input and produces the desired target audio output signals is described. The present disclosure utilizes the encoder models trained across several different audio processing tasks to derive generalized model weights. The audio representations learned using the trained encoder models are compact and generalizable using clustering and generative models together with decoder models and psychoacoustic-based loss functions for generating target audio signals for several downstream tasks such as audio separation, in-painting and enhancement across different mobile device applications such as calling, editing, restoration, speech style transfer etc.

[0057] In one embodiment, the foundational model of the present disclosure aims to help the user to obtain target audio source signals free from different degradations such as different types of sound sources, missing audio samples, reverberation, poor SNR etc. using single foundation model, thereby improving overall model efficiency, and providing several functionalities and seamless user experience.

[0058] In an embodiment, an input mixture is fed to an encoder and a separator network. The output of the separator network is two masking signals that are obtained after separation and classification of the input mixture. The two masking signals are the fed to decoder to obtain the estimated sound sources of the first and second signals, respectively.

[0059] In an embodiment, several exemplary commercial applications use similar techniques. For instance, an audio eraser application performs source signal separation and classification and allows user to conduct manual gain control for any source to create custom audio mixtures using separated sound sources for recorded videos. However, such applications provide poor performance when considering a source signal having more than three sound sources.

[0060] Therefore, it is known in the art that the several foundation models perform functionalities of source separation and classification. However, the present disclose describes training of a neural network to generate sound embedding representations. The present disclosure describes how to derive the embedding, which is used for several downstream tasks, which include but not limited to classification, separation, enhancement etc.

[0061] The present disclosure provides using the clustering or generative methods for modifying the latent space embedding. An exemplary method according to the present disclosure modifies the latent space embedding to make the latent space embedding (or representation) more optimal for other downstream tasks.

[0062] In an embodiment, psychoacoustic-metric based loss functions based on perceptual distortion, interference or learned audio quality metrics may be used to match the perceptual audio quality to the target audio signals.

[0063] In an embodiment, the foundation model architecture utilizes degraded audio signals as input during inference, and the pairs of target audio signals & degraded audio signals during training.

[0064] Universal encoder according to the present disclosure transforms raw audio waveforms / audio features to efficient low-dimensional latent space representations using the universal encoder-decoder models of the present disclosure.

[0065] The present disclosure generates one or more latent space representations corresponding to the distinct learned features present in the input audio waveform, by modifying the low-dimensional latent space using pre-trained clustering or generative models such that the derived compact latent space representations is decoded to generate target audio signals for wide variety of downstream audio processing tasks such as separation, enhancement and in-painting.

[0066] Decoding the generated latent space representations corresponding to distinct features to generate audio waveforms which are perceptually similar to target waveform / features is implemented using psychoacoustic-based loss functions and universal decoders.

[0067] Task-specific adaptor models is used for further fine-tuning the output from foundation models for downstream audio processing tasks, namely, audio source separation, in-painting, and enhancement. These outputs are utilized as inputs for further processing for target use-case applications such as audio / video calling, editing, restoration, speech style transfer etc.

[0068] In an embodiment, the encoder-decoder model transforms the input (degraded audio waveform) into a lower-dimensional latent space through a universal encoder network and subsequently reconstructs the target audio waveform from this latent representation using a universal decoder network. The lower-dimensional latent space should be semantically rich so that it generalizes across several target downstream tasks while achieving desired performance.

[0069] Neural network architectures, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Convolutional Recurrent Neural Networks (CRNNs) are employed to learn the weights required for transforming the input waveform into a lower-dimensional latent space and reconstructing it back into the target waveform according to different downstream tasks.

[0070] The encoder-decoder model is trained to reconstruct the input audio signal, similar to a Variational Autoencoder. The key difference from Variational Autoencoder and the present approach lies in the derivation of encoder-decoder weights using the below-mentioned methods to improve generalization of latent space across multiple tasks

[0071] The encoder and decoder are trained for different tasks like audio sound separation, in-painting models in multiple iterations. An exemplary way to train is to share the weights across multiple tasks. First, the weights will get updated in iteration 1 (e.g., an iteration focusing on in-painting task), followed by iteration 2 (e.g., an iteration focusing on enhancement task) and last in iteration 3 (e.g., an iteration focusing on sound separation). The weights are updated in a shared manner across iterations, with each iteration focusing on a specific task. This allows the encoder model to generalize better across multiple tasks.

[0072] In an exemplary training method, the encoder and decoder with same architecture separately with same pre-trained initial weights will be trained separately to avoid random weight updates (for example, the encoder's weights are taken from the above-mentioned method described earlier as it has already learned semantics from each task or take any other pre-trained model weights for any of the target tasks). In this case, each encoder / decoder model (namely encoder / decoder model 1, 2 and 3) will get trained separately, followed by weighted average of encoder and decoder weights from each training to get the generalised weights of universal encoder and decoder. The weights for universal encoder and decoder are derived as:

[0073]

[0074]

[0075] In math figures 1 and 2, 'encoder1' and 'decoder1' refers to weights of the encoder model 1 and the decoder model 1, respectively. 'Encoder2' and 'Decoder2' refers to weights of the encoder model 2 and the decoder model 2, respectively. 'Encoder3' and 'Decoder3' refers to weights of the encoder model 3 and the decoder model 3, respectively. The learned weights which are derived from multiple tasks in this manner will be semantically rich.

[0076] The inputs and outputs for the methods are described as follows.Input will be degraded audio signals which consist of different audio sources at different SNR, reverberation, diffuse and directional noise etc. which will be transformed into a low-dimensional compact representation and transformed back to time domain to generate the target audio and obtained the learned encoder and decoder weights for each target task.

[0077] In the first method, the training methodology of universal encoder and decoder model is in iterative manner with shared weights.

[0078] In the second method, the training methodology with weighted average of the learned weights of separately trained encoder and decoder.

[0079] The present disclosure modifies the latent space using clustering and task-specific latent space model. The objective is to modify the latent space learned using universal encoder to generalize well with desired performance across several downstream tasks using clustering and adaptor models.

[0080] For each task, a pre-trained encoder block is utilized for each task to obtain task-specific latent space, which is further compressed using clustering technique into a compact representation called codebook. Clustering is a technique that groups similar data points together based on their similarity.

[0081] The derived codebook acts as a reference for specifying the appropriate TSLSM (Task Specific Latent Space Model) model for a given task. By mapping the latent space of a new task to the nearest cluster in the codebook, which TSLSM model to use for that task, is determined.

[0082] The codebook not only helps in specifying the TSLSM model but also prevents the need to re-train a new encoder for adding a new task to the foundation model. Instead of starting from scratch, the existing universal encoder is leveraged and adapt it to the new task using the specified TSLSM model.

[0083] The universal latent space is then obtained from the encoder and is modified to a task-specific latent space using a low complexity TSLSM model. The TSLSM model fine-tunes the universal latent space to better suit the requirements of a specific task. This ensures maintenance of the benefits of the universal encoder while tailoring it to the needs of each task.

[0084] For clustering of latent space and utilizing codebook for task specific latent space modification, the TSLSM blocks are introduced, which modify the semantically rich universal latent space to task specific latent space.

[0085] The input to the system is the universal latent space obtained using the universal encoder. The output is the task-specific latent space, which is used for the desired downstream task.

[0086] In an embodiment, processed for modification of latent space using generative models is described. The objective of processes is totransform the universal latent space into a task-specific latent space using generative methods. For illustration purposes, Task 1 is considered as separation. The aim is to train a generator network to convert the universal latent space into the separation (Task 1) latent space.

[0087] In an embodiment, a first method for training for Generator and Discriminator Models is depicted. First, an initial analysis is done by the Discriminator model to distinguish whether the latent feature obtained from the Generator model is similar to source separation's task latent space.

[0088] Target and generated latent space features are passed to decoder 1 to generate the audio files. The decoder weights will be frozen during training since it is previously trained by the task 1. Secondly, discriminator and generator models update their weights using backpropagation based on result of initial analysis.

[0089] In an embodiment, an exemplary method for training for Generator and Discriminator Models is depicted. The initial analysis for the second method is same as mentioned above for first method.

[0090] In the second level analysis, after converting both the latent spaces to audio signal (time domain) using a decoder, a diffusion Markov chain process is used to iteratively inject Gaussian noise into the decoded audio signal output of the latent spaces which help to further improve the learning capability of generator model to bypass the discriminator model.

[0091] Additionally, in second level analysis, discriminator 2 is used to provide the additional check whether the output audio from decoder 1 is real or fake.

[0092] Losses that are applied for the first and second methods are Mean Square Error (MSE), Scale-Invariant-Signal to Noise Ratio (SI-SNR), SNR, Mel loss and multi-resolution Short-Term Fourier Transform (STFT) loss. The inputs and outputs for the processes is the universal latent space. The output is the task-specific latent space, which is used for the desired task, such as separation in this case.

[0093] The present approach introduces a generative method to transform the universal latent space into a task-specific latent space, allowing for efficient adaptation to various tasks without extensive retraining.

[0094] The present approach employs single / multiple discriminators and ensures that the generated audio closely matches the target audio for the specific task. Additionally, the use of a diffusion Markov chain in the second method further enhances the quality of the generated audio, making our approach robust and effective for transforming the universal latent space into task-specific representations.

[0095] In an embodiment, an alternative approach to the Universal Encoder-Decoder with Latent Space Modification Module is described. The objective is to derive end-to-end approach for generating lower-dimensional latent space and deriving compact representation that should be semantically rich enough to be used in several tasks of foundation model.

[0096] Theencoder from the method of iterative training of encoder and decoder for multiple tasks is taken and the initial layers of the encoder is frozen and then retrained to smaller EEN (Encoder Extension Networks) to produce task wise latent space l1, l2 and l3 respectively for target downstream task1, task2 and task3.

[0097] The task-specific latent spaces l1, l2 and l3 are averaged together (e.g., weighted average) to get the universal latent space and then passed to universal decoder and then finally to DEN (Decoder Extension Network) to convert it back to target audio signals. Different Neural network architectures, such as CNNs or RNNs are employed to learn the weights required for transforming the waveform into a lower-dimensional latent space and reconstructing it back into the waveform according to different tasks.

[0098] Training is done for all permutation cases of task1, task2 and task3. E.g., in one iteration one permutation of task (e.g., enhancement followed by separation) will update the EEN and DEN weights and in the next iteration, another permutation of task (e.g., separation followed by in-painting) will update the weights.

[0099] This permuted input-based training will help to generalize the learned universal latent space.

[0100] The input and output to the model is all permutation (based on different tasks) of audio files along with the task specific hyper parameters which are transformed to lower-dimension and again reconstructed back to generate input signals to enable the EEN and DEN networks to learn the weights for each task.

[0101] The method of training universal encoder and decoder blocks by using task specific EEN and DEN blocks in end-to-end manner is a novel approach.

[0102] Further, the present disclosure describes the techniques for Psychoacoustic-based loss functions for estimating the target signals. The objective of the technique is implementing a novel linear combination of three loss function - Perceptual Distortion (PD), Perceptual Interference ( ) and DNN-based learned psychoacoustic audio quality metric.

[0103] In the state of the art, the 2f Model estimates the perceived quality of separated source signals driven by 2 Model Output Variables from Perceptual Evaluation of Audio Quality Metric Model:Average Block Distortion(ADB)and Average Modulation Difference (AvgModDiff1).

[0104] The non-linear relationship betweenAvgModDiff1andMean Opinion Scores (MMS)is modelled and combined withADBusing a linear regression model represented by the following math figure:

[0105]

[0106] In the 2f Model. to calculate the Perceptual Distortion (PD), the reference signal and reconstructed signal output are passed to the 2f model and MMSscore is computed which is independent of the interference.

[0107] To calculate the Perceptual Interference ( ), reference signal free from noise and artefacts and reconstructed signal output free from artefacts are computed and passed to the 2f Model to obtain an MMS score decoupled from perceptual distortion.

[0108] To compute MMS scores (where is which denotes artifacts introduced by model forPDcalculation and is interference between sources used for computing ), : .

[0109] In addition to the PI and PD computation described above, a DNN-based model based on CNN or Long Short-Term Memory (LSTM) architectures are trained offline using a database of human audio quality perceptual judgements such as open-source databases to derive a learned Mean Opinion Score (MOS)( ).

[0110] The learned MOS is combined with PI and PD using a linear combination term together with the learned MOS, which is represented as:

[0111]

[0112] where and , are parameters which are tuned based on empirical results on test dataset.

[0113] The loss function is integrated in the encoder and decoder training and is applied to the input and target signal. It may also be used to further train the adaptor models for source separation and enhancement tasks.

[0114] The novel combination of loss functions ensures that the estimated signals minimize perceptual interference, perceptual distortion and have same perceptual audio quality as target audio signals.

[0115] The method for calculating the perceptual interference in a way which is decoupled from perceptual distortion is a major novelty for the loss functions.

[0116] The adaptor models aims to improve sound separation performance by reducing spectral leakage and improve the overall audio quality of separated audio signals. In the present disclosure, the source separation is implemented with up to 4 source mixtures with categories: speech, music, wind, and noise.

[0117] In the state of the art, the U-net architecture is used for speech enhancement. The present disclosure utilizes this U-net architecture with time-frequency features derived using Short Time Fourier Transform (STFT) processing block in combination with additional convolutional / LSTM layers or additional features such as Filter banks.

[0118] In an embodiment, Long Short Term Memory (LSTM) Layers are added in bottleneck and performance is improved by learning long-term dependencies.

[0119] In an embodiment, Filter bank-based processing such as Equivalent Rectangular Bandwidth (ERB), Mel Filter bank Cepstral Coefficients (MFCC) based on human audio processing mechanisms is integrated to give model perceptually motivated audio features for enhanced separation performance.

[0120] In an embodiment, Conformer (e.g., a combination of a term 'Transformer; and a term 'Convolutional') and attention layers are added in bottleneck layer of U-net models for improving separation performance by including self-attention, attention across time and frequency channels and convolution-augmented transformer to help model to learn long-range dependencies across audio signals, which improves consistency of separated audio signals.

[0121] Theattention layers of the adaptor model convolute across time and frequency domain channels separately to extract better sound source features and dependencies across time and frequency channels.

[0122] Additional time domain and frequency domain attention layers in bottleneck layer with skip connections to get the better bottleneck feature maps and to improve separation performance and reduce spectral leakage.

[0123] The input to the system during inference is the estimated separated audio signal obtained from the foundation model which might contain spectral leakage mix of other sound sources. The output during inference is the fine-tuned separated audio signals with minimum spectral leakage across different source signals and desired overall audio quality.

[0124] The adaptor models enhance or remove reverberation or to fill in from noisy / reverberant / masked audio signals to produce target audio using a conditional diffusion based generative model. The diffusion model iteratively removes noise, reverberation or in-paints the missing audio samples from an audio signal to reconstruct the original signal.

[0125] The technique initializes the audio signal with Gaussian noise and iteratively removes the Gaussian noise by obtaining a de-noised estimate and conditioning it based on the input observations provided during training (which will be target audio signals for enhancement and / or in-painting task). The final output signal is obtained when the added Gaussian noise level becomes imperceptible.

[0126] The technique allows for joint and separate processing of in-painting and enhancement tasks. If one task is required, the respective pre-trained encoder together with desired target signal is used for training the adaptor model.

[0127] A diffusion-based generative model is employed, where a stochastic process transforms target audio into noisy / reverberant / masked audio through a forward process governed by a Stochastic Differential Equation (SDE).

[0128] The reverse process, guided by a score model, generates target audio from the noisy / reverberant / masked input.

[0129] U-Net architecture having a hierarchical encoder / decoder structure, Noise Conditional Score Network based on multi resolution U-Net are common networks used to model score function.

[0130] In one embodiment, the objective functions such as denoising score matching are used to train diffusion models. ODE solvers such as Eulers method is used to solve reverse diffusion process to estimate target signal.

[0131] Techniques like Constant Q transform, STFT, Mel-spectrogram are used to reduce feature size and model complexity.

[0132] Additional conditioning inputs from pre-trained encoders for enhancement and in-painting tasks generate the desired latent representations, which are used to reduce number of iterations in reverse process to speed up the diffusion process.

[0133] Scaling of latent vectors during conditioning helps to jointly train for in-painting and enhancement tasks.

[0134] The input to the modelisestimated in-painted signal obtained from foundational model which contains residual background noise, reverberation etc.

[0135] The output from the model istarget enhanced and / or in-painted audio signal generated by the conditional diffusion adaptor model.

[0136] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0137] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood, however that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternative falling within the spirit and the scope of the disclosure.

[0138] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a device or system or apparatus proceeded by "comprises a" does not, without more constraints, preclude the existence of other elements or additional elements in the device or system or apparatus.

[0139] In the following detailed description of the embodiments of the disclosure, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0140] As described in the background section, currently, mobile devices use multiple Machine Learning (ML)- models and different applications for performing different audio processing tasks such as separation, enhancement, in-painting for audio / video editing. However, there is a lot of redundancy in terms of processing blocks, types of features extracted, derivation and modification of latent space across the different audio processing tasks / target applications etc., which leads to increase in computational complexity, power inefficiency and latency for the mobile device applications. Further, from user perspective, users have to switch between different applications to perform the audio processing tasks, and even if a single application supports multiple processing tasks, switching between different models leads to higher waiting time, heating and lag issues for users leading to poor experience.

[0141] Further, several ML models utilize text / image / video-based input signals together with audio signals to derive tokens for generative AI-based audio generation tasks that are processed further to derive a latent space representation. However, the processing of multiple input modalities and multi-modal based learned representation often leads to high computational complexity, large number of Neural network (NN) parameters and higher storage space requirements.

[0142] To overcome the above-mentioned limitations, the present disclosure provides a method and a system for building a unified foundation AI model for audio signal processing that uses only audio signals as inputs. For example, the present disclosure provides a single unified foundational model which generates target audio signals for several downstream applications with only degraded audio signals as input. At the outset, the present disclosure provides a novel technique for utilizing foundation models using audio signals only as inputs to learn general audio signal representations and adaptor models to fine-tune performance for specific tasks such as audio separation model, enhancement and in-painting applications. A detailed description of the proposed solution is provided in the upcoming paragraphs in conjunction with Figures 1-9.

[0143] Figure 1Adepicts an exemplary environment 100a for implementation of the proposed foundation model for processing audio signals, in accordance with an embodiment of the present disclosure. The exemplary environment 100a depicts a device 101 where a foundation model 102 and task-specific adaptor models 110 (interchangeably referred to as "adaptor models 110") are implemented. Figure 1A further depicts that the device 101 receives a degraded audio waveform (interchangeably referred to as "audio signal") from one or more sources 103. The degraded audio waveform may be also referred to as "an input audio waveform". In an embodiment, the one or more sources 103 may comprise a server, one or more devices, a database, and the like. The degraded audio waveform is processed by the foundation model 102 and the adaptor models 110 to generate an output audio waveform. In an embodiment, the output audio waveform may correspond to (or may be used for) a specific audio processing task.

[0144] Figure 1Bdepicts an architectural diagram 100b of the proposed foundation model 102 for processing audio signals, in accordance with an embodiment of the present disclosure. In an embodiment, the architectural diagram 100b depicts the foundation model 102 comprising an encoder 104, a modification block 106 and a decoder 108. The architectural diagram 100b further depicts multiple adaptor models 110 for different audio processing tasks / applications, including, but not limited to in-painting, separation and enhancement. In an embodiment, the foundation model 102 and the adaptor models 110 may be trained to process audio signals in order to generate target audio signals for specific audio processing tasks. During training, an input provided to the foundation model 102 may comprise a degraded audio waveform along with a corresponding target audio waveform. However, once the foundation model 102 is trained, for example, during inference process, an input provided to the foundation model 102 may comprise a degraded audio waveform (e.g., a raw audio waveform from an audio recording device or an external electronic device). only. In an embodiment, the training of the foundation model 102 may be performed in order to allow the foundation model 102 to generate an output audio waveform that is perceptually similar to the target audio waveform.

[0145] At the outset, the encoder 104 of the foundation model 102 generates a low-dimensional latent space representation corresponding to the degraded audio waveform. For example, the encoder 104 may receive an input audio waveform, and generate a low-dimensional latent representation corresponding to the input audio waveform. Next, the modification block 106 modifies the low-dimensional latent space representation to generate one or more modified latent space representations corresponding to one or more audio features present in the degraded audio waveform. For example, each of the one or more modified latent space representations may correspond to one or more audio features associated with the input audio waveform. Further, the decoder 108 of the foundation model 102 decodes the generated one or more modified latent space representations to generate one or more processed audio waveforms. Each of the one or more processed audio waveform is specific to be used for a corresponding audio processing task. For example, the decoder 108 may generate one or more task-specifically processed audio waveforms, such as estimated audio signals for audio in-painting, estimated audio signals for audio separation, or estimated signals for audio enhancement. In an embodiment, the decoder 108 is trained using psychoacoustic loss functions in order to enable the decoder 108 to generate the one or more processed audio waveforms perceptually similar to the target audio waveform. Furthermore, task-specific adaptor models 110 are used for finetuning the one or more processed audio waveforms generated by the foundation model 102, for downstream audio processing tasks, such as, audio source separation, in-painting or enhancement. For example, an adaptor model for in-painting included in the adaptor models 110 may generate finetuned audio signals for audio in-painting from the estimated audio signals for in-painting generated by the decoder 108. These outputs can then be utilized as inputs for further processing for target use-case applications / application such as audio / video calling, editing, restoration, speech style transfer, and the like. A detailed description of the training and inference of the encoder 104, the modification block 106, the decoder 108 and the adaptor models 110 is described in the upcoming paragraphs in conjunction with Figures 2-8.

[0146] Figure 2illustrates an exemplary block diagram 200 of a device 202 (analogous to the device 101 depicted in Figure 1A) for generating processed audio signals, in accordance with an embodiment of the present disclosure. In an embodiment, the device 202 comprises a communication interface 204, an input / output (I / O) module 206, a memory 208 and a processor 210. It shall be noted that, in some embodiments, the device 202 may include more or fewer components than those depicted herein. The various components of the device 202 may be implemented using hardware, software, firmware, or any combinations thereof. Further, the various components of the device 202 may be operably coupled with each other. More specifically, various components of the device 202 may be capable of communicating with each other using communication channel media (such as buses, interconnects, etc.).

[0147] In an embodiment, the communication interface 204 may include a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter) that enables the device 202 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 204 may permit the device 202 to receive information from another device and / or provide information to another device. For example, the communication interface 204 may receive or obtain one or more audio signals (or waveforms) (e.g., degraded (raw) audio signals and / or target audio signals) from other devices. The received audio signals may be stored in the memory 208, and be provided as input for the foundation model 102. The communication interface 204 may transmit or provide one or more audio signals (or waveforms) generated using the foundation model 102 and / or the adaptor models 110 to other devices. In an embodiment, the communication interface 204 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, a high-definition multimedia interface (HDMI), a secure digital (SD) card interface, or the like. In an embodiment, the communication interface 204 may be a communication 'circuit' used to connect the device 202 with the other devices.

[0148] In an embodiment, the I / O module 206 may comprise a circuitry or device to receive a command or data to be used by another component (e.g., the processor 210) of the device 202, from the outside (e.g., a user or other devices) of the device 202. For example, the I / O module 206 may receive or obtain one or more audio signals (or waveforms) (e.g., degraded (raw) audio signals and / or target audio signals) from outside of the device 202. The received audio signals may be stored in the memory 208, and be provided as input for the foundation model 102. In an embodiment, the I / O module 206 may include, a microphone, a mouse, a keyboard, a key (e.g., a button), a touch screen, or a digital pen (e.g., a stylus pen). The I / O module 206 may comprise a circuitry or device to provide any signals or data generated by another component (e.g., the processor 210) of the device, to the outside of the device 202. For example, the I / O module 206 may transmit or provide one or more audio signals (or waveforms) generated using the foundation model 102 and / or the adaptor models 110 to outside of the device 202. In an embodiment, the I / O module 206 include a sound output circuit (e.g., a speaker) or a display device to provide information visually to the outside.

[0149] In one embodiment, the memory 208 is capable of storing machine executable instructions. In an embodiment, the memory 208 may also store the foundation model 102 and the adaptor models 110 along with the essential data or information required for execution of these models. In an embodiment, the processor 210 is embodied as an executor of software instructions. As such, the processor 210 is capable of executing the instructions stored in the memory 208 to perform one or more operations described herein.

[0150] In an embodiment, the memory 208 may include a random-access memory (RAM), a read only memory (ROM), and / or another type of dynamic or static storage device (e.g., a flash memory, a magnetic memory, and / or an optical memory) that stores information and / or instructions for use by the processor 210. In an embodiment, the memory 208 may contain information and / or software related to the operation and use of the device 202. For example, the memory 208 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, and / or a solid-state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, or another type of non-transitory computer-readable medium, along with a corresponding drive.

[0151] The memory 208 may store various data used by at least one component (e.g., the processor 120) of the device 202. The various data may include, for example, software and input data or output data for a command related thereto. For example, a program may be stored in the memory 208 as software, and may include, for example, an operating system (OS), middleware, or an application.

[0152] In one embodiment, the processor 210 may be embodied as a multi-core processor, a single core processor, or a combination of one or more multi-core processors and one or more single core processors. For example, the processor 210 may be embodied as one or more of various processing devices, such as a coprocessor, a microprocessor, a controller, a Digital Signal Processor (DSP), a processing circuitry with or without an accompanying DSP, or various other processing devices including, a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like.

[0153] In an embodiment, the processor 210 may include one or a plurality of processors. At this time, one or a plurality of processors may be a general-purpose processor, such as a Central Processing Unit (CPU), an Application Processor (AP), or the like, a graphics-only processing unit such as a Graphics Processing Unit (GPU), a Visual Processing Unit (VPU), and / or an AI-dedicated processor such as a neural processing unit (NPU).

[0154] In an embodiment, the device 202 may comprise a plurality of processors. For example, the processor 210 may be implemented as one or more multi-core processors that include one or more cores (e.g., homogeneous multi-cores or heterogeneous multi-cores). When the plurality of cores are included in the processor 210, each of the cores may include a cache memory, and a common cache shared by the cores may be included in the processor 210. Each of the cores may independently read and execute program instructions or each of the cores may read and execute one or more portions of program instructions. For example, the memory 208 may store one or more instructions, and the one or more instructions may be executed by one or more cores individually or collectively.

[0155] The processor 210 may execute, for example, software to control at least one other component (e.g., a hardware or software component) of the device 202 coupled with the processor 210, and may perform various data processing or computation. In one embodiment, as at least part of the data processing or computation, the processor 210 may load a command or data received from another component (e.g., the communication interface 204 or the I / O module 206) in the memory 208, process the command or the data stored in the memory 208, and store resulting data in the memory 210.

[0156] For example, the processor 210 of the device 202 may invoke at least one of the one or more instructions stored in the memory 208, and execute the at least one of the one or more instructions, with or without using one or more other components under the control of the processor 210. This allows the device 202 to be operated to perform at least one function according to the at least one instruction invoked. The one or more instructions may include a code generated by a compiler or a code executable by an interpreter. The memory 208, which may be a machine-readable storage medium, may be provided in the form of a non-transitory storage medium. Wherein, the term "non-transitory" simply means that the storage medium is a tangible device, and does not include a signal (e.g., an electromagnetic wave), but this term does not differentiate between where data is semi-permanently stored in the memory 208 (the storage medium) and where the data is temporarily stored in the memory 208. In an embodiment, the device 202 may comprise one or more processors, and the one or more instructions may be executed by the one or more processors individually or collectively, thereby causing the device 202 to perform any combination of one or more operations (or functions, steps) described herein.

[0157] According to the present disclosure, a computer-readable storage medium may store one or more instructions. The one or more instructions, when executed by at least one processor individually or collectively, cause the at least one processor to perform any method, operation, step, function, or combination of thereof described herein.

[0158] According to the present disclosure, a device for generating processed audio signals may comprise memory and a processor. The memory may comprise one or more storage mediums storing one or more instructions. The processor may comprise a processing circuitry and being operatively coupled to the memory. When executed by the processor individually or collectively, the one or more instructions may cause the device to receive, from one or more sources, an input audio waveform. When executed by the processor individually or collectively, the one or more instructions may cause the device to generate, using an encoder model, a low-dimensional latent space representation corresponding to the input audio waveform. When executed by the processor individually or collectively, the one or more instructions may cause the device to modify the low-dimensional latent space representation for generating one or more modified latent space representations corresponding to one or more audio features associated with the input audio waveform. When executed by the processor individually or collectively, the one or more instructions may cause the device to generate, using a decoder model, one or more processed audio waveforms based on the one or more modified latent space representations. Each of the one or more processed audio waveforms may correspond to one or more features among the one or more audio features associated with the input audio waveform. When executed by the processor individually or collectively, the one or more instructions may cause the device to generate, using a first adaptor model among the one or more adaptor models a first target audio waveform for a first audio processing task associated with the first adaptor model, based on at least one of the one or more processed audio waveforms.

[0159] In an embodiment, the processor 210 is configured to perform various operations as elaborated in detail in the upcoming paragraphs.

[0160] Figures 3A and 3Bdepict architectural diagrams 300a and 300b respectively of training of the encoder-decoder network in accordance with embodiments of the present disclosure. For example, the architectural diagrams 300a and 300b illustrate different training approaches of the encoder-decoder network, with the objective to transform the input (e.g., a degraded audio waveform or a raw audio waveform) into a low-dimensional latent space representation through the encoder 104 and subsequently reconstruct the target audio waveform from the low-dimensional latent space representation using the decoder 108. Further, it may be noted by a skilled person that the encoder 104 and the decoder 108 may employ Neural Network architectures such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Convolutional Recurrent Neural Networks (CRNNs) to learn the weights required for transforming the input degraded audio waveform into a low-dimensional latent space representation and reconstructing it back into the target audio waveform according to different downstream tasks.

[0161] Referring to Figure 3A, the architectural diagram 300a depicts a first approach for training the encoder-decoder network. It may be appreciated by a skilled person that the training of the encoder-decoder network may be performed by the processor 210.

[0162] In the first approach, the encoder-decoder network is trained iteratively, where each iteration corresponds to a distinct audio processing task. For example, during a first iteration (i.e., 'Iteration-1'), an input audio waveform 302 (e.g., a degraded audio waveform) which comprises different audio sources at different Signal to Noise Ratio (SNR), reverberation, diffuse and directional noise etc., is provided to the encoder 104 to generate a low-dimensional latent space representation 310. The low-dimensional latent space representation 310 may be transformed to the time domain to generate a target audio waveform 314 corresponding to audio in-painting task, for example, using an audio in-painting model 312. The generated low-dimensional latent space representation 310 is decoded by the decoder 108 to generate an output audio waveform 334. During the first iteration, a set of one or more weights of the encoder 104 and decoder 108 may be updated (or learned), respectively. The set of weights utilized by the encoder 104 and the decoder 108 are updated after the first iteration and utilized during a second iteration (i.e., Iteration 2). For example, during the second iteration, an input audio waveform 304 is provided to the sub-encoder 306 to generate a low-dimensional latent space representation 316. The low-dimensional latent space representation 316 may be transformed to the time domain to generate a target audio waveform 320 corresponding to audio enhancement task, for example, using a sound enhancement model 318. The generated low-dimensional latent space representation 316 is decoded by the decoder 108 to generate an output audio waveform 336. Next, post the second iteration, the updated set of weights utilized by the encoder 104 and the decoder 108 are again updated and utilized during a third iteration (i.e., Iteration 3). For example, during the second iteration, an input audio waveform 306 is provided to the encoder 104 to generate a low-dimensional latent space representation 322. The low-dimensional latent space representation 322 may be transformed to the time domain to generate a target audio waveform 326 corresponding to audio separation task, for example, using an audio separation model 324. The generated low-dimensional latent space representation 322 is decoded by the decoder 108 to generate an output audio waveform 338.

[0163] In an embodiment, training of the encoder 104 and the decoder 108 may be repeated iteratively for the same input audio waveform (e.g., for the input audio waveform 302) till output audio waveforms for each audio processing tasks have been generated. Once the output audio waveforms for each audio processing tasks have been generated, the process may repeat for another input audio waveform (e.g., for the input audio waveform 304 and / or the input waveform 306) till the encoder 104 and the decoder 108 have been efficiently trained. In an embodiment, the efficiency by which the encoder 104 and the decoder 108 can perform their respective tasks is judged based on psychoacoustic loss functions and the training of the encoder 104 and the decoder 108 is performed to reduce the values of psychoacoustic loss functions (described in detail in the upcoming paragraphs). For example, during each iteration, a loss may be calculated based on corresponding output audio waveform and corresponding target audio waveform. Based on the calculated loss, the set of weights of the encoder 104 and the set of weights of the decoder 108 may be updated.

[0164] In an embodiment, each model for generating corresponding target audio waveform may be a pre-trained task-specific model or pre-defined task-specific algorithm to generate, from a latent space representation, an audio waveform which is to be utilized for specific task. For example, the audio in-painting model 312 may pre-trained or pre-defined to generate, from any latent space representation (e.g., the latent space representation 310), an audio waveform (e.g., the target audio waveform 314) to be utilized for audio in-painting process. Similarly, the audio enhancement model 318 may generate the target audio waveform 320 to be utilized for sound enhancement process from the latent space representation 316, and the audio separation model 324 may generate the target audio waveform 316 to be utilized for sound separation process for the latent space representation 322.

[0165] In an embodiment, one or more task-specific target audio waveforms may be input with the input audio waveforms 302, 304, and 306 as training dataset. For example, during the iteration 1, the loss may be calculated based on the output audio waveform 334 and a target audio waveform which is generated or obtained, with regard to the corresponding input waveform, for training any model focusing on audio in-painting process.

[0166] Referring to Figure 3B, the architectural diagram 300b depicts a second approach for training the encoder-decoder network. It may be appreciated by a skilled person that the training of the encoder-decoder network may be performed by the processor 210.

[0167] In the second approach, each sub-encoder and sub-decoder network is trained independently of another sub-encoder and sub-decoder network. For instance, as depicted in Figure 3B, the set of weights of the encoder 104 may be determined based on the sets of weights of sub-encoders 344, 358 and 372. On similar lines, the set of weights of the decoder 108 may be determined based on the sets of weights of sub-decoders 352, 366 and 382. It may, however, be noted by a skilled person that the number of sub-encoders and the number of sub-encoders can be higher or lower depending on the number of audio processing tasks.

[0168] An input audio waveform 342 may be provided to each sub-encoder, and each sub-encoder may generate low-dimensional latent space representations 346, 360 and 374 and target audio waveforms 350, 364 and 380 corresponding to distinct audio processing tasks, respectively. For instance, the first sub-encoder 344 may generate the low dimensional latent space representation 346 from an input audio waveform 342. The low-dimensional latent space representation 336 may be transformed to the time domain to generate and the target audio waveform 350 corresponding to audio in-painting task, for example, using an audio in-painting model 348. The generated low-dimensional latent space representation 336 is decoded by the first sub-decoder 352 to generate the output audio waveform 350. On similar lines, the second sub-encoder 358 may generate the low dimensional latent space representation 360 from the input audio waveform 342. The low-dimensional latent space representation 360 may be transformed to the time domain to generate the target audio waveform 364 corresponding to audio enhancement task, for example, using an audio in-painting model 362. The generated low-dimensional latent space representation 360 is decoded by the second sub-decoder 358 to generate the output audio waveform 368. Further, the third sub-encoder 372 may generate the low dimensional latent space representation 374 from the input audio waveform 342. The low-dimensional latent space representation 374 may be transformed to the time domain to generate the target audio waveforms 380 corresponding to audio separation task, for example, using an audio in-painting model 376. The generated low-dimensional latent space representation 360 is decoded by the third sub-decoder 382 to generate the output audio waveform 384.

[0169] In an embodiment, each sub-encoder-sub-decoder network may start with a set of initial pre-trained weights and the set of initial pre-trained weights may be updated after each sub-encoder-sub-decoder network has performed their respective tasks. For example, during each iteration, a loss may be calculated based on corresponding output audio waveform and corresponding target audio waveform. Based on the calculated loss, the sets of weights of each sub-encoder-decoder pair may be updated. The weights of the encoder 104 may be taken as a combination (or a weighted average) of the weights of each of the sub-encoders and the weights of the decoder 108 may be taken as a combination (or a weighted average) of the weights of each of the sub-decoders as depicted in math figures (5) and (6) below .

[0170]

[0171]

[0172] In an embodiment, the weight parameters α1, α2, α3, β1, β2, and β3 of math figures 5 and 6 may be predefined or adjusted based on one or more properties of each downstream task. For example, each weight parameter may be fixed or adjusted based on importance, frequency of application, and / or difficulty of corresponding audio processing task.

[0173] As described in the preceding paragraphs, in order to ensure efficient training of the encoder 104 and the decoder 108, For example, in order for the generated processed (or output) waveforms to perceptually be similar to the input audio waveforms, a loss function is integrated with the encoder 104 and the decoder 108 during their training. This loss function is a linear combination of three loss functions viz., Perceptual Distortion (PD), Perceptual Interference ( ) and Deep Neural Network (DNN)-based learned psychoacoustic audio quality metric. The methodology to generate the loss function is as follows -

[0174] The starting point for generating the loss function is a pre-existing 2f Model which estimates the perceived quality of separated source signals, driven by two Model Output Variables (MOVs) from Perceptual Evaluation of Audio Quality (PEAQ) Metric Model. These MOVs are Average Block Distortion (ADB) and Average Modulation Difference (AvgModDiff1). A non-linear relationship between AvgModDiff1 and Mean Opinion Scores (MMS)is modelled and combined with ADB using a linear regression model represented by math figure (7):

[0175]

[0176] In the 2f Model, to calculate the Perceptual Distortion (PD), a reference signal and a reconstructed signal output are passed to the 2f model and MMSscore is computed which is independent of the interference.

[0177] To calculate the Perceptual Interference (PI(decoupled)), a reference signal free from noise and artefacts and reconstructed signal output free from artefacts are computed and passed to the 2f Model to obtain a MMS score decoupled from perceptual distortion.

[0178] In addition to the PI and PD computation described above, a DNN-based model based on CNN or Long Short-Term Memory (LSTM) architectures are trained using a database of human audio quality perceptual judgements such as open source databases to derive a learned MOS score ( ).

[0179] The learned MOS score ( ) is then combined with PI and PD using a linear combination term together with the learned MOS score, which is represented by the following math figure (8):

[0180]

[0181] where and , parameters which can be tuned based on empirical results on test dataset.

[0182] The loss function as represented by math figure (8) is integrated during training of the encoder 104 and the decoder 108 and ensures that the processed audio waveforms generated by the foundation model 102 minimize perceptual interference, perceptual distortion and have same perceptual audio quality as the target audio waveforms.

[0183] Figure 4Adepicts an architectural diagram 400a for modification of low-dimensional latent space representation during the training phase, using a clustering technique, in accordance with an embodiment of the present disclosure. For example, in the architectural diagram 400a, the modification block 106 is implemented as a clustering block that uses clustering techniques for modifying the low-dimensional latent space representation generated by the encoder 104. In an embodiment, the modification of the low-dimensional latent space representation is performed in order to generalize low-dimensional latent space representation with desired performance across several downstream tasks.

[0184] During the training phase, as depicted in Figure 4A, the modification block 104 may employ one or more sub-encoders such as sub-encoders 402, 404 and 406 in order to generate task-specific latent space representations 408, 410 and 412, where each task-specific latent space representation corresponds to a distinct audio processing task. In an embodiment, the sub-encoders 402, 404, and 406 may correspond to the encoder 104 which has been trained for audio processing tasks 1-3 using the training method described in reference to FIG. 3A. In an embodiment, the sub-encoders 402, 404, and 406 may each be independently pre-trained for their respective audio processing tasks 1-3, for example, using the training method described in reference to FIG. 3B. The generated task-specific latent space representations 408, 410 and 412 are further compressed using the clustering technique into a compact representation called codebook 414, based on a similarity between data points of each task-specific latent space representation.

[0185] The derived codebook 414 acts as a reference for specifying an appropriate Task Specific Latent Space Model (TSLSM) for a given audio processing task, during model inference. For instance, referring to Figure 4B which depicts modification of low-dimensional latent space representation during model inference by mapping the latent space representation of a new task to a nearest cluster in the codebook 414, a most relevant TSLSM may be determined in order to derive a task-specific latent space representation corresponding to a new audio processing task 430.

[0186] Thus, the codebook 414 not only helps in specifying the TSLSM but also prevents the need to retrain a new encoder for adding a new task to the foundation model 102.

[0187] Further, as also depicted in Figure 4B, a low-dimensional latent space representation 416 obtained from the encoder 104 may be modified to a task-specific latent space representation such as task-specific latent space representations 424, 426 and 428, using a low-complexity TSLSM such as TSLSMs 418, 420 and 422. The TSLSM fine-tunes the low-dimensional latent space representation to better suit the requirements of a specific audio processing task. This allows to maintain the benefits of the encoder 104 while tailoring it to the needs of each audio processing task.

[0188] In an embodiment, the codebook 414 may include a set of compressed representations corresponding to generated task-specific latent space representations 408, 410 and 412. Each compressed representation may represent a cluster or a TSLSM for the corresponding task. For example, a compressed representation of the task-specific latent space representation 408 may correspond to a cluster or a TSLSM associated with the audio processing task 1. During inference, a low-dimensional latent space representation generated by the encoder 104 is mapped to the nearest cluster among one or more clusters associated with a set of compressed representations included in the codebook 414. A TSLSM corresponding to the mapped compression may be selected for the low-dimensional latent space representation. The low-dimensional latent space representation may be modified using the selected TSLSM.

[0189] For the modification of the low-dimensional latent space representation, the modification block 106 may employ an exemplary approach, based on a generative model as described in the forthcoming paragraphs in conjunction with Figures 5A-5C.

[0190] Figure 5Adepicts an architectural diagram 500a for implementation of a first method of modification of low-dimensional latent space representation during the training phase, using a generative model, in accordance with an embodiment of the present disclosure. During the training phase, training data comprising a plurality of low-dimensional latent space representations corresponding to a plurality of degraded audio waveforms is provided to a generator 506 of the generative model. The generator 506 generates a plurality of task-specific latent space representations based on the plurality of low-dimension latent space representations. For instance, as depicted in Figure 5A, a low-dimensional latent space representation 504 is provided to the generator 506. The generator 506 generates a task-specific latent space representation corresponding to an audio processing task. Upon generation of the task-specific latent space representation, a discriminator 508 may evaluate the authenticity of the generated task-specific latent space representation by comparing the generated task-specific latent space representation to a target task-specific latent space representation 502. If the generated task-specific latent space representation is determined to be authentic, the task-specific latent space representation and the target task-specific latent space representation may be provided to a decoder 510 to generate corresponding audio waveforms. In an embodiment, the weights of the decoder 510 are frozen during training, since it is trained as shown in Figure 3A and 3B. For example, the decoder 510 may correspond to the decoder 108 which has been trained using the method described with reference to Fig. 3A and / or FIG. 3B. Further, during training, a set of weights associated with the generator 506 and the discriminator 508 are updated using backpropagation based on a result of the generation and the analysis performed correspondingly by the generator 506 and the discriminator 508. For example, one or more weights of the generator 506 and one or more weights of the discriminator 508 are updated using backpropagation based on at least one of the evaluation or the plurality of generated audio waveforms from the decoder 510.

[0191] Figure 5B depicts an architectural diagram 500b for implementation of a second method of modification of low-dimensional latent space representation during the training phase, using a generative model, in accordance with an embodiment of the present disclosure. The initial steps of the second method are same as those performed in the first method. For example, on similar lines as recited with reference to Figure 5A, a low-dimensional latent space representation 504 is provided to the generator 506. The generator 506 generates a task-specific latent space representation corresponding to an audio processing task. Upon generation of the task-specific latent space representation, a discriminator 508 may evaluate the authenticity of the generated task-specific latent space representation by comparing the generated task-specific latent space representation to a target task-specific latent space representation 502. If the generated task-specific latent space representation is determined to be authentic, the task-specific latent space representation and the target task-specific latent space representation may be provided to a decoder 510 to generate corresponding audio waveforms. In an embodiment, the weights of the decoder 510 are frozen during training, since it is trained as shown in Figure 3A and 3B. After converting the task-specific latent space representation and the target task-specific latent space representation to audio waveforms using the decoder 510, a diffusion Markov chain process may be used to iteratively inject a predefined Gaussian noise into the audio waveforms to further improve the learning capability of the generator 506 to bypass the discriminator. For example, upon injecting predefined Gaussian noise into the audio waveforms, a second discriminator i.e., discriminator 512 may evaluate the authenticity of the audio waveforms such that the learning capability of the generator 506 may be improved. Further, it may be appreciated by a skilled person that for the injection of Gaussian noise, any suitable process other than the diffusion Markov chain process may be used. In an embodiment, one or more weights of the generator 506, one or more weights of the discriminator 508, and / or one or more weights of the discriminator 512 are updated using backpropagation based on at least one of: the evaluation by the discriminator 508, the plurality of generated audio waveforms from the decoder 510, or the evaluation by the discriminator 512.

[0192] Further, it may be noted by a skilled person that during training of the generator 506 either by the first method or the second method, loss functions may be integrated in order to train the generator 506 to effectively generate task-specific latent space representations. In an embodiment, the loss functions that may be integrated may comprise Mean Square Error (MSE), Scale-Invariant-Signal to Noise Ratio (SI-SNR), SNR, Mel loss and multi-resolution Short-Term Fourier Transform (STFT) loss.

[0193] Figure 5Cdepicts an architectural diagram 500c for modification of low-dimensional latent space representation during model inference, using a generative model, in accordance with an embodiment of the present disclosure. For example, Figure 5C depicts a trained generator 514 that may be implemented during inference to generate task-specific latent space representation 516 from the low-dimensional latent space representation 504 received from the encoder 104.

[0194] The task-specific latent space representations generated by the modification block 106 either through clustering technique or via generative model may be fed to the decoder 108 for generating estimated audio signals for different audio processing tasks as depicted in Figure 1.

[0195] In an embodiment, instead of separately training the encoder-decoder network and the modification block 106, an end-to-end training approach may be employed for simultaneously training the foundation model 102 comprising encoder 104 - decoder 108 network and the modification block 106 as depicted in Figure 6. For example, Figure 6 depicts an architectural diagram 600 for training the foundation model 102 based on an end-to-end approach, in accordance with an embodiment of the present disclosure. The architectural diagram 600 comprises the encoder 104, encoder extension network 608 comprising sub-encoders E1, E2 and E3, the decoder 108 and decoder extension network 618 comprising sub-decoders D1, D2 and D3. It may be noted by a skilled person that the number of sub-encoders and the number of sub-decoders may depend on the number of audio processing tasks for which the foundation model 102 is to be trained. For instance, in the scenario depicted in Figure 6, the foundation model 102 is to be trained for three audio processing tasks viz, audio in-painting, audio enhancement and audio separation and hence, the encoder extension network 608 and the decoder extension network 618 comprises three sub-encoders and three sub-decoders, respectively.

[0196] In the end-to-end approach depicted in Figure 6, one or more input audio signals such as the input audio signals 602 and 604 along with task-specific hyperparameters, such as the hyperparameters 606 are fed to the encoder 104. In an embodiment, the encoder 104 may be trained by following the methodology recited in the preceding paragraphs in conjunction with Figure 3A. The initial layers of the encoder 104 may be frozen and the sub-encoders E1, E2 and E3 of the encoder extension network 606 may be trained to produce task-specific latent space representations 610, 612 and 614, where each task-specific latent space representation corresponds to a distinct audio processing task and are associated with corresponding weights w1, w2 and w3. Next, a weighted average of the task-specific latent space representations 610, 612 and 614 may be taken to generate a low-dimensional latent space representation 616 which may then be passed to the decoder 108, and subsequently to the decoder extension network 618 so as to convert the low-dimensional latent space representation 616 into one or more output (or target) audio signals, such as the output audio signals 620 and 622. In an embodiment, the decoder 108 may be trained by following the methodology recited in the preceding paragraphs in conjunction with Figure 3A. Further, it may be appreciated by a skilled person that the encoder extension network 608 and the decoder extension network 618 may employ different neural network architectures, such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs) to learn the weights required for transforming the task-specific latent space representations 610, 612 and 614 into a low-dimensional latent space representation 616 and reconstructing the low-dimensional latent space representations 610, 612 and 614 into output audio signals 620 and 622 according to different tasks.

[0197] Further, in an embodiment, in the end-to-end approach, the training of the encoder extension network 608 and the decoder extension network 618 may be performed for all combination cases of the audio processing tasks. For instance, in a first iteration, the weights of the encoder extension network 608 and the decoder extension network 618 may be updated for the combination of audio enhancement task and audio separation task. Further, in the next iteration, the weights of the encoder extension network 608 and the decoder extension network 618 may be updated for the combination of audio separation task and audio in-painting task, and so on.

[0198] Now, as described in the preceding paragraphs, the decoder 108 may generate estimated audio signals for different audio processing tasks as depicted in Figure 1. For example, as depicted in Figure 1, the decoder 108 may be trained to generate estimated in-painting audio signal, estimated enhanced audio signal and estimated separated audio signals. These estimated audio signals may then be finetuned or processed by the adaptor models 110 in order to obtain finetuned in-painting audio signal, finetuned enhanced audio signal and finetuned separated audio signals that can be subsequently utilized as inputs for further processing for target use-case applications such as audio / video calling, editing, restoration, speech style transfer, and the like.

[0199] Figures 7A-7Edepict architectural diagrams 700a-700e of an adaptor model 110 for generating finetuned separated audio signals, in accordance with some embodiments of the present disclosure. For example, Figures 7A, 7B and 7C-7E depict different embodiments for implementation of the adaptor model 110 for generating finetuned separated audio signals based on a U-net architecture comprising primarily a complex encoder 706, a complex decoder 708 and a complex ratio mask 710. For instance, Figure 7A depicts a first embodiment of the adaptor model 110 for generating finetuned separated audio signals where time-frequency audio features are extracted from an input audio signal 702 using a Short Time Fourier Transform (STFT) processing block 704 in combination with additional convolutional / Long Short-Term Memory (LSTM) layers. In an embodiment, the input audio signal 702 is the estimated separated audio signal obtained from the foundation model 102 which contains spectral leakage mix of other sound sources. Figure 7B depicts a second embodiment of the adaptor model 110 for generating finetuned separated audio signals which additionally integrates filterbanks 712 to facilitate processing of input audio signal 702 based on human audio processing mechanisms to give the adaptor model 110 perceptually motivated audio features for enhanced separation performance. Figures 7C-7E depict a third embodiment of the adaptor model 110 for generating finetuned separated audio signals where Conformer (correspond to a combination of a term 'Transformer' and a term 'Convolutional') and attention layers are added in bottleneck layer of the U-net architecture with skip connections for improving separation performance by including self-attention, attention across time and frequency channels and convolution-augmented transformer to help the adaptor model 110 to obtain improved bottleneck feature maps in order to improve separation performance and reduce spectral leakage. In the embodiments depicted in Figures 7A-7E, input audio signal 702 is separated into four categories viz., speech, music, wind and noise. However, it may be appreciated by a skilled person that said four categories are not limiting, and the sole objective of the adaptor model 110 is to generate finetuned separated audio signals with minimum spectral leakage across different source signals and desired audio quality.

[0200] Figure 8Adepicts architectural diagram 800a of adaptor model 110 for generating finetuned enhanced and / or in-painted audio signals during training phase, in accordance with some embodiments of the present disclosure. The objective of the adaptor model 110 for generating finetuned enhanced and / or in-painted audio signals is to enhance or remove reverberation or to fill in the noisy / reverberant / masked audio signals to produce a target audio signal using a conditional diffusion-based generative model 810. During the training phase, as depicted in Figure 8A, a target audio signal 804 is initialized with Gaussian noise by the conditional diffusion-based generative model 810. In an embodiment, the conditional diffusion-based generative model 810 employs a stochastic process to transform the target audio signal 804 into noisy / reverberant / masked audio signal through a forward process governed by a Stochastic Differential Equation (SDE). The Gaussian noise is then iteratively removed by obtaining a de-noised estimate of the target audio signal 804 (which will be target audio signals for enhancement and / or in painting task) and subsequently conditioning the same based on the input observations provided during the training. In an embodiment, a reverse process, guided by a score model, generates output signal audio from the noisy / reverberant / masked target audio signal 804. Further, the output audio signal is obtained when the added Gaussian noise level becomes imperceptible. In an embodiment, the conditional diffusion-based generative model 810 employs a U-Net architecture comprising a hierarchical encoder / decoder structure and a Noise Conditional Score Network (NCSN) to model score function. Further, objective functions such as denoising score matching are used to train conditional diffusion-based generative model 810. Furthermore, Ordinary Differential Equation (ODE) solvers such as Eulers method may be used to solve reverse diffusion process to estimate target audio signal 804. Also, techniques like Constant Q transform, STFT, Mel-spectrogram may be used to reduce feature size and model complexity.

[0201] The above-described approach allows for joint and separate processing of in-painting and enhancement tasks. If one task is required, the respective pre-trained encoder together with desired target audio signal may be used for training the adaptor model 110.

[0202] Figure 8Bdepicts architectural diagram 800b of an adaptor model 110 for generating finetuned enhanced and / or in-painted audio signals during model inference, in accordance with some embodiments of the present disclosure. For example, the architectural diagram 800b depicts that during the model inferenceof the adaptor model 110, only features extracted from an input audio signal 802 is fed to a trained conditional diffusion-based generative model 816 to generate an output audio signal 818. In an embodiment, the input audio signal 802 may comprise an estimated in-painted audio signal or an estimated enhanced audio signal obtained from the foundation model 102 and the output audio signal 818 may comprise a finetuned in-painted audio signal or a finetuned enhanced audio signal.

[0203] The solution described in the present disclosure thus provide a foundation model that enables generation of a target audio signal quality for several audio processing tasks, namely, source separation, enhancement, in-painting etc., across different applications such as audio / video calling, editing, restoration, speech style transfer etc. Such techniques may prove to be advantageous in various scenarios as described by the following use cases -

[0204] Audio editing. For audio / video editor application, the proposed foundation model 102 can perform high quality source separation to generate separated audio signals, and provide a manual user using an additional module for gain equalizer to control the gain of each source and create their own audio mixture.

[0205] Audio / Video Calling. Using long audio in-painting adaptor model with foundation model, the proposed foundation model 102 can estimate the dropped audio packets during calls due to network issues, especially for longer durations packet drops (e.g., greater than 100-200ms). Long duration packet drops allow larger latency requirements for the proposed in-painting application for the audio foundation models. The proposed foundation and adaptor models for enhancement can also remove background noise, reverberation etc., from a caller's voice signal to enhance call clarity for the user.

[0206] Audio Restoration. Due to errors during data transfer, audio data in recorded audio / video files from voice / video recorder applications may be corrupted or missing, or even contain some added noise due to compression, poor Signal to Noise Ratio (SNR) etc. In such cases, the proposed foundation model 102 can help to restore missing and / or noisy audio segments of large lengths (e.g., greater than 200ms) with offline processing.

[0207] Figure 9 depicts, by way of a flowchart, an exemplary method 900 for generating processed audio signals, in accordance with an embodiment of the present disclosure. The method 900 is implemented at the device 202. The method 900 may comprise one or more steps. The method 900 may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, and functions, which perform particular functions or implement particular abstract data types.

[0208] Further, the order in which the method 900 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method. Additionally, individual blocks may be deleted from the methods without departing from the scope of the subject matter described herein. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof.

[0209] At step 902, the method 900 may comprise receiving, from one or more sources, a degraded audio waveform corresponding to a target audio waveform. In an embodiment, said receiving may be performed by the processor 210. Further, the one or more sources may comprise a server, one or more devices, a database, and the like.

[0210] At step 904, the method 900 may comprise generating a low-dimensional latent space representation corresponding to the degraded audio waveform. In an embodiment, for generating the low-dimensional latent space representation, the processor 210 may employ the encoder 104 of the trained foundation model 102.

[0211] At step 906, the method 900 may comprise modifying the low-dimensional latent space representation for generating one or more modified latent space representations corresponding to one or more audio features present in the degraded audio waveform. In an embodiment, the processor 210 may employ one of: a clustering technique or a generative model for generating one or more modified latent space representations. Further, the one or more modified latent space representations may also be referred to as one or more task-specific latent space representations such that each of the one or more task-specific latent space representations correspond to a distinct audio processing task, including but not limited to, audio in-painting, audio enhancement and audio separation.

[0212] At step 908, the method 900 may comprise generating one or more processed audio waveforms based on the one or more modified latent space representations. In an embodiment, the one or more processed audio waveforms are similar to the target audio waveform. Further, for generating the one or more processed audio waveforms, the processor 210 may employ the decoder 108 of the trained foundation model 102.

[0213] At step 910, the method 900 may comprise performing a set of predefined operations on each of the one or more processed audio waveforms for utilization by a respective audio processing task. In an embodiment, the set of predefined operations may be performed by the processor 210 by employing the adaptor model 110 for finetuning the generated one or more processed audio waveforms.

[0214] According to an embodiment of the present disclosure, a method of generating processed audio signals may comprise receiving, from one or more sources, an input audio waveform. The method may comprise generating, using an encoder model, a low-dimensional latent space representation corresponding to the input audio waveform. The method may comprise modifying the low-dimensional latent space representation for generating one or more modified latent space representations corresponding to one or more audio features associated with the input audio waveform. The method may comprise generating, using a decoder model, one or more processed audio waveforms based on the one or more modified latent space representations, wherein each of the one or more processed audio waveforms corresponds to one or more features among the one or more audio features associated with the input audio waveform. The method may comprise generating, using a first adaptor model among the one or more adaptor models, a first target audio waveform for a first audio processing task associated with the first adaptor model, based on at least one of the one or more processed audio waveforms.

[0215] Alternatively or additionally, the method may comprise generating, using a second adaptor model among the one or more adaptor models. a second target audio waveform for a second audio processing task associated with the second adaptor model, based on at least one of the one or more processed audio waveforms. The first audio processing task and the second audio processing task may be associated with different audio processing operation.

[0216] Alternatively or additionally, the encoder model may be trained by providing, to the encoder model, training data comprising a plurality of training audio waveforms and a plurality of corresponding target audio waveforms. The encoder model may be trained by iteratively training the encoder model for generating a low-dimensional latent space of each training audio waveform of the plurality of training audio waveforms. Each iteration may be associated with a distinct audio processing task. The set of weights updated after a first iteration may be utilized during a subsequent iteration.

[0217] Alternatively or additionally, the decoder model may be trained by providing, to the decoder model, low-dimensional latent space representations corresponding to each of a plurality of training audio waveforms generated by the encoder model. The decoder model may be trained by iteratively training the decoder model for generating processed audio waveforms based on the corresponding low-dimensional latent space representations such that each of the processed audio waveforms are similar to target audio waveforms associated with the respective training audio waveforms. Each iteration may be associated with a distinct audio processing task . A set of weights updated after a first iteration may be utilized during a subsequent iteration.

[0218] Alternatively or additionally, the encoder model may be trained by providing to each of one or more sub-encoder models, training data comprising a plurality of training audio waveforms and a plurality of corresponding target audio waveforms. The encoder model may be trained by training, each of the one or more sub-encoder models for generating respective low-dimensional latent space representations of each training audio waveform of the plurality of degraded audio waveforms. The low-dimensional latent space representations generated by each sub-encoder model may be associated with a distinct audio processing task. Each sub-encoder model may be trained independently of another sub-encoder model. The trained encoder model may utilize a set of weights which is a combination of weights of each of one or more sub-encoder models.

[0219] Alternatively or additionally, the decoder model may be trained by providing to each of one or more sub-decoder models, low-dimensional latent space representations corresponding to each of a plurality of training audio waveforms generated by a respective one or more sub-encoder models. The decoder model may be trained by training, each of the one or more sub-decoder models for generating processed audio waveforms based on the corresponding low-dimensional latent space representations. The processed audio waveforms generated by each sub-decoder model may be associated with a distinct audio processing task. Each sub-decoder model may be trained independently of another sub-decoder model. The trained decoder model may utilize a set of weights which is a combination of weights of each of the one or more sub-decoder models.

[0220] Alternatively or additionally, the encoder and the decoder model may be trained by integrating a loss function with the encoder and the decoder model. The loss function may be a combination of Perceptual Distortion (PD) loss, decoupled Perceptual Interference (PIdecoupled) loss and / or a Deep Neural Network (DNN)-based learned psychoacoustic audio quality metric loss.

[0221] Alternatively or additionally, the one or more modified latent space representations may be generated by implementing one of: a trained generative model or a clustering technique on the low-dimensional latent space representation.

[0222] Alternatively or additionally, generating the one or more modified latent space representations using the clustering technique may comprise mapping each of the one or more low-dimensional latent space representations to the nearest cluster among one or more clusters associated with a set of compressed representations. The set of compressed representations may be generated by obtaining, using one or more pre-trained sub-encoder models, one or more training low-dimensional latent space representations. Each of the one or more training low-dimensional latent space representations may be associated with a distinct audio processing task. The set of compressed representations may be generated by clustering the one or more low-dimensional latent space representations to generate the set of compressed representations, wherein each compressed representation corresponds to each of the one or more training low-dimensional latent space representations.

[0223] Alternatively or additionally, wherein the generative model may comprise a generator and a first discriminator. The generative model may be trained by providing to the generator, training data comprising a plurality of training low-dimensional latent space representations corresponding to a plurality of training audio waveforms. The generative model may be trained by generating, by the generator, a plurality of task-specific latent space representations based on the plurality of training low-dimension latent space representations. Each task-specific latent space representation may be associated with a distinct audio processing task. The generative model may be trained by evaluating, by the first discriminator, authenticity of each of the generated plurality of task-specific latent space representations by comparing each of the generated plurality of task-specific latent space representations with a corresponding target task-specific latent space representation. The generative model may be trained by generating, using the decoder model, a plurality of audio waveforms corresponding to each of the target task-specific latent space representations based on the evaluation. The generative model may be trained by updating one or more weights of the generator and one or more weights of the first discriminator using backpropagation based on at least one of the evaluation or the plurality of generated audio waveforms from the decoder model.

[0224] Alternatively or additionally, the generative model may comprise a generator, a first discriminator, and a second discriminator. The generative model may be trained by providing to the generator, training data comprising a plurality of training low-dimensional latent space representations corresponding to a plurality of training audio waveforms. The generative model may be trained by generating, by the generator, a plurality of task-specific latent space representations based on the plurality of training low-dimension latent space representations. Each task-specific latent space representation may be associated with a distinct audio processing task. The generative model may be trained by evaluating, by the first discriminator, authenticity of each of the generated plurality of task-specific latent space representations by comparing each of the generated plurality of task-specific latent space representations with a corresponding target task-specific latent space representation. The generative model may be trained by generating, using the decoder model , a plurality of audio waveforms corresponding to each of the target task-specific latent space representations based on the evaluation. The generative model may be trained by superimposing a predefined noise on each of the plurality of audio waveforms generated by the decoder model. The generative model may be trained by evaluating, by a second discriminator, authenticity of each of the plurality of audio waveforms generated by the decoder model. The generative model may be trained by updating one or more weights of the generator, one or more weights of the first discriminator, and one or more weights of the second discriminator, using backpropagation based on the evaluation by the first discriminator, the evaluation by the second discriminator, and the plurality of generated audio waveforms from the decoder model.

[0225] Figure 10illustrates a block diagram of an exemplary computer system 1000 for implementing embodiments consistent with the present disclosure. In an embodiment, the computer system 1000 may be used to implement the device 202. Thus, the computer system 1000 may be used for generating the foundation model 102. The computer system 1000 may comprise a Central Processing Unit 1004 (also referred as "CPU" or "processor"). The processor 1004 may comprise at least one data processor. The processor 1004 may include specialized processing units such as integrated system (bus) controllers, memory management control units, floating point units, graphics processing units, digital signal processing units, etc.

[0226] The processor 1004 may be disposed in communication with one or more input / output (I / O) devices (not shown) via I / O interface 1002. The I / O interface 1002 may employ communication protocols / methods such as, without limitation, audio, analog, digital, monoaural, RCA, stereo, IEEE (Institute of Electrical and Electronics Engineers) -1394, serial bus, universal serial bus (USB), infrared, PS / 2, BNC, coaxial, component, composite, digital visual interface (DVI), high-definition multimedia interface (HDMI), Radio Frequency (RF) antennas, S-Video, VGA, IEEE 1016.n / b / g / n / x, Bluetooth, cellular (e.g., code-division multiple access (CDMA), high-speed packet access (HSPA+), global system for mobile communications (GSM), long-term evolution (LTE), WiMAX, or the like), etc.

[0227] Using the I / O interface 1002, the computer system 1000 may communicate with one or more I / O devices. For example, the input device 1020 may be an antenna, keyboard, mouse, joystick, (infrared) remote control, camera, card reader, fax machine, dongle, biometric reader, microphone, touch screen, touchpad, trackball, stylus, scanner, storage device, transceiver, video device / source, etc. The output device 1022 may be a printer, fax machine, video display (e.g., cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), plasma, Plasma display panel (PDP), Organic light-emitting diode display (OLED) or the like), audio speaker, etc.

[0228] The processor 1004 may be disposed in communication with the communication network 1018 via a network interface 1006. The network interface 1006 may communicate with the communication network 1018. The network interface 1006 may employ connection protocols including, without limitation, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 1016.11a / b / g / n / x, etc. The communication network 1018 may include, without limitation, a direct interconnection, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), the Internet, etc. The network interface 1006 may employ connection protocols include, but not limited to, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 1016.11a / b / g / n / x, etc.

[0229] The communication network 1018 includes, but is not limited to, a direct interconnection, an e-commerce network, a peer to peer (P2P) network, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), the Internet, Wi-Fi, and such. The first network and the second network may either be a dedicated network or a shared network, which represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), Wireless Application Protocol (WAP), etc., to communicate with each other. Further, the first network and the second network may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, etc. The communication network 1018 may be in communication with the plurality of sensors 104 to monitor vibrations

[0230] In some embodiments, the processor 1004 may be disposed in communication with a memory 1010 (e.g., RAM, ROM, etc. not shown in Figure 10) via a storage interface 1008. The storage interface 1008 may connect to memory 1010 including, without limitation, memory drives, removable disc drives, etc., employing connection protocols such as serial advanced technology attachment (SATA), Integrated Drive Electronics (IDE), IEEE-1094, Universal Serial Bus (USB), fiber channel, Small Computer Systems Interface (SCSI), etc. The memory drives may further include a drum, magnetic disc drive, magneto-optical drive, optical drive, Redundant Array of Independent Discs (RAID), solid-state memory devices, solid-state drives, etc.

[0231] The memory 1010 may store a collection of program or database components, including, without limitation, user interface 1012, an operating system 1014, web browser 1016 etc. In some embodiments, computer system 1000 may store user / application data, such as, the data, variables, records, etc., as described in this disclosure. Such databases may be implemented as fault-tolerant, relational, scalable, secure databases such as Oracle ® or Sybase®.

[0232] The operating system 1014 may facilitate resource management and operation of the computer system 1000. Examples of operating systems include, without limitation, APPLE MACINTOSHROS X, UNIXR, UNIX-like system distributions (E.G., BERKELEY SOFTWARE DISTRIBUTIONTM(BSD), FREEBSDTM, NETBSDTM, OPENBSDTM, etc.), LINUX DISTRIBUTIONSTM(E.G., RED HATTM, UBUNTUTM, KUBUNTUTM, etc.), IBMTMOS / 2, MICROSOFTTMWINDOWSTM(XPTM, VISTATM / 7 / 8, 10 etc.), APPLERIOSTM, GOOGLERANDROIDTM, BLACKBERRYROS, or the like.

[0233] In some embodiments, the computer system 1000may implement the web browser 1016stored program component. The web browser 1016may be a hypertext viewing application, for example MICROSOFTRINTERNET EXPLORERTM, GOOGLERCHROMETM0, MOZILLARFIREFOXTM, APPLERSAFARITM, etc. Secure web browsing may be provided using Secure Hypertext Transport Protocol (HTTPS), Secure Sockets Layer (SSL), Transport Layer Security (TLS), etc. Web browsers 1016may utilize facilities such as AJAXTM, DHTMLTM, ADOBERFLASHTM, JAVASCRIPTTM, JAVATM, Application Programming Interfaces (APIs), etc. In some embodiments, the computer system 1000may implement a mail server (not shown in Figure) stored program component. The mail server may be an Internet mail server such as Microsoft Exchange, or the like. The mail server may utilize facilities such as ASPTM, ACTIVEXTM, ANSITMC++ / C#, MICROSOFTR, .NETTM, CGI SCRIPTSTM, JAVATM, JAVASCRIPTTM, PERLTM, PHPTM, PYTHONTM, WEBOBJECTSTM, etc. The mail server may utilize communication protocols such as Internet Message Access Protocol (IMAP), Messaging Application Programming Interface (MAPI), MICROSOFTRexchange, Post Office Protocol (POP), Simple Mail Transfer Protocol (SMTP), or the like. In some embodiments, the computer system 1000may implement a mail client stored program component. The mail client (not shown in Figure) may be a mail viewing application, such as APPLERMAILTM, MICROSOFTRENTOURAGETM, MICROSOFTROUTLOOKTM, MOZILLARTHUNDERBIRDTM, etc.

[0234] The illustrated steps are set out to explain the exemplary embodiments shown, and it should be anticipated that ongoing technological development will change the manner in which particular functions are performed. These examples are presented herein for purposes of illustration, and not limitation. Further, the boundaries of the functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.

[0235] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the invention.

[0236] When a single device or article is described herein, it will be readily apparent that more than one device / article (whether or not they cooperate) may be used in place of a single device / article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device / article may be used in place of the more than one device or article, or a different number of devices / articles may be used instead of the shown number of devices. The functionality and / or the features of a device may be alternatively embodied by one or more other devices which are not explicitly described as having such functionality / features. Thus, other embodiments of the invention need not include the device itself.

[0237] Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based here on. Accordingly, the disclosure of the embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

[0238] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

[0239] In an embodiment, the present disclosure provides a single unified foundational model which can generate target audio signals for several downstream applications with degraded audio signals only as input.

[0240] In an embodiment, the present disclosure allows expansion of various downstream applications such as audio editing, restoration, calling etc., to other applications such as speech style transfer using the foundation model architecture by modifying the adaptor models.

Claims

A method of generating processed audio signals, the method comprising:receiving, from one or more sources (103), an input audio waveform;generating, using an encoder model (104), a low-dimensional latent space representation corresponding to the input audio waveform;modifying the low-dimensional latent space representation for generating one or more modified latent space representations corresponding to one or more audio features associated with the input audio waveform;generating, using a decoder model (108), one or more processed audio waveforms based on the one or more modified latent space representations, wherein each of the one or more processed audio waveforms corresponds to one or more features among the one or more audio features associated with the input audio waveform, andgenerating, using a first adaptor model among one or more adaptor models (110), a first target audio waveform for a first audio processing task associated with the first adaptor model, based on at least one of the one or more processed audio waveforms.The method as claimed in claim 1, further comprising:generating, using a second adaptor model among the one or more adaptor models (110), a second target audio waveform for a second audio processing task associated with the second adaptor model, based on at least one of the one or more processed audio waveforms,wherein the first audio processing task and the second audio processing task are associated with different audio processing operation.The method as claimed in claim 1 or claim 2, wherein the encoder model (104) is trained by:providing, to the encoder model (104), training data comprising a plurality of training audio waveforms and a plurality of corresponding target audio waveforms; anditeratively training the encoder model (104) for generating a low-dimensional latent space representation (310, 316, 322) of each training audio waveform of the plurality of training audio waveforms,wherein each iteration is associated with a distinct audio processing task, andwherein a set of weights updated after a first iteration are utilized during a subsequent iteration.The method as claimed in any one of claims 1 to 3, wherein the decoder model (108) is trained by:providing, to the decoder model (108), low-dimensional latent space representations (310, 316, 322) corresponding to each of a plurality of training audio waveforms generated by the encoder model (104); anditeratively training the decoder model (108) for generating processed audio waveforms (334, 336, 338) based on the corresponding low-dimensional latent space representations (310, 316, 322) such that each of the processed audio waveforms are similar to target audio waveforms associated with the respective training audio waveforms,wherein each iteration is associated with a distinct audio processing task andwherein a set of weights updated after a first iteration are utilized during a subsequent iteration.The method as claimed in claim 1 or claim 2, wherein the encoder model (104) is trained by:providing to each of one or more sub-encoder models (344, 358, 372), training data comprising a plurality of training audio waveforms and a plurality of corresponding target audio waveforms; andtraining, each of the one or more sub-encoder models (344, 358, 372) for generating respective low-dimensional latent space representations (346, 360, 374) of each training audio waveform of the plurality of degraded audio waveforms,wherein the low-dimensional latent space representations (346, 360, 374) generated by each sub-encoder model are associated with a distinct audio processing task, andwherein each sub-encoder model is trained independently of another sub-encoder model.The method as claimed in claim 5, wherein the trained encoder model (104) utilizes a set of weights which is a combination of weights of each of one or more sub-encoder models (344, 358, 372).The method as claimed in any one of claims 1, 2, 5, and 6, wherein the decoder model (108) is trained by:providing to each of one or more sub-decoder models (352, 366, 382), low-dimensional latent space representations (346, 360, 374) corresponding to each of a plurality of training audio waveforms generated by a respective one or more sub-encoder models (344, 358, 372); andtraining, each of the one or more sub-decoder models (352, 366, 382) for generating processed audio waveforms (350, 364, 380) based on the corresponding low-dimensional latent space representations,wherein the processed audio waveforms (350, 364, 380) generated by each sub-decoder model are associated with a distinct audio processing task, andwherein each sub-decoder model is trained independently of another sub-decoder model.The method as claimed in claim 7, wherein the trained decoder model (108) utilizes a set of weights which is a combination of weights of each of the one or more sub-decoder models.The method as claimed in any one of claims 1 to 8, wherein the encoder (104) and the decoder model (108) are trained by:integrating a loss function with the encoder (104) and the decoder model (108), wherein the loss function is a combination of Perceptual Distortion (PD) loss, decoupled Perceptual Interference loss and a Deep Neural Network (DNN)-based learned psychoacoustic audio quality metric loss.The method as claimed in any one of claims 1 to 9, wherein the one or more modified latent space representations are generated by implementing one of: a trained generative model or a clustering technique on the low-dimensional latent space representation.The method as claimed in claim 10, wherein generating the one or more modified latent space representations using the clustering technique comprises mapping each of the one or more low-dimensional latent space representations to the nearest cluster among one or more clusters associated with a set of compressed representations,wherein the set of compressed representations are generated by:obtaining, using one or more pre-trained sub-encoder models (402, 404, 406), one or more training low-dimensional latent space representations (408, 410, 412), wherein each of the one or more training low-dimensional latent space representations (408, 410, 412) is associated with a distinct audio processing task; andclustering the one or more low-dimensional latent space representations (408, 410, 412) to generate the set of compressed representations, wherein each compressed representation corresponds to each of the one or more training low-dimensional latent space representations.The method as claimed in claim 10, wherein the generative model comprises a generator (506) and a first discriminator (508), and wherein the generative model is trained by:providing to a generator (506), training data comprising a plurality of training low-dimensional latent space representations corresponding to a plurality of training audio waveforms;generating, by the generator (506), a plurality of task-specific latent space representations based on the plurality of training low-dimension latent space representations, wherein each task-specific latent space representation is associated with a distinct audio processing task;evaluating, by the first discriminator (508), authenticity of each of the generated plurality of task-specific latent space representations by comparing each of the generated plurality of task-specific latent space representations with a corresponding target task-specific latent space representation;generating, using the decoder model (510 / 108), a plurality of audio waveforms corresponding to each of the target task-specific latent space representations based on the evaluation; andupdating one or more weights of the generator (506) and one or more weights of the first discriminator (508) using backpropagation based on at least one of the evaluation or the plurality of generated audio waveforms from the decoder model (510 / 108).The method as claimed in claim 10, wherein the generative model comprises a generator (506), a first discriminator (508), and a second discriminator (512), and wherein the generative model is trained by:providing to the generator (506), training data comprising a plurality of training low-dimensional latent space representations corresponding to a plurality of training audio waveforms;generating, by the generator (506), a plurality of task-specific latent space representations based on the plurality of training low-dimension latent space representations, wherein each task-specific latent space representation is associated with a distinct audio processing task;evaluating, by the first discriminator (508), authenticity of each of the generated plurality of task-specific latent space representations by comparing each of the generated plurality of task-specific latent space representations with a corresponding target task-specific latent space representation;generating, using the decoder model (510 / 108) , a plurality of audio waveforms corresponding to each of the target task-specific latent space representations based on the evaluation;superimposing a predefined noise on each of the plurality of audio waveforms generated by the decoder model (510 / 108);evaluating, by a second discriminator (512), authenticity of each of the plurality of audio waveforms generated by the decoder model (510 / 108); andupdating one or more weights of the generator (506), one or more weights of the first discriminator (508), and one or more weights of the second discriminator (512), using backpropagation based on the evaluation by the first discriminator (508), the evaluation by the second discriminator (512), and the plurality of generated audio waveforms from the decoder model (510 / 108).A computer-readable storage medium storing one or more instructions, wherein the one or more instructions, when executed by at least one processor individually or collectively, cause the at least one processor to perform the method of any one of claims 1 to 13.A device for generating processed audio signals, the device (202) comprises:memory (208) comprising one or more storage mediums storing one or more instructions; anda processor (210) comprising a processing circuitry and being operatively coupled to the memory (208), wherein, when executed by the processor (210) individually or collectively, the one or more instructions cause the device to:receive, from one or more sources (103), an input audio waveform;generate, using an encoder model (104), a low-dimensional latent space representation corresponding to the input audio waveform;modify the low-dimensional latent space representation for generating one or more modified latent space representations corresponding to one or more audio features associated with the input audio waveform;generate, using a decoder model (108), one or more processed audio waveforms based on the one or more modified latent space representations, wherein each of the one or more processed audio waveforms corresponds to one or more features among the one or more audio features associated with the input audio waveform, andgenerate, using a first adaptor model among one or more adaptor models (110), a first target audio waveform for a first audio processing task associated with the first adaptor model, based on at least one of the one or more processed audio waveforms.

Images