Method for restoring distorted speech by using gaussian guidance, system for performing same, and computer program

Abstract

A method for restoring distorted speech by using Gaussian guidance, and a system for performing same are disclosed. According to one disclosed embodiment, the method by the system including at least one processor restores distorted speech by using Gaussian guidance comprises: a training process for inputting a clean speech signal and a distorted speech signal into a speech restoration model in which a second network is combined on the basis of a first network, generating Gaussian guidance on the basis of latent features extracted from the first network and the clean speech signal and providing same to the first network as conditioning information, thereby training the speech restoration model; and an inference process for excluding the second network from the trained speech restoration model, using, as conditioning information, the first network that uses Gaussian guidance sampled from a Gaussian distribution, so as to restore a distorted speech signal, thereby providing a clean speech signal, wherein the first network is trained to restore the clean speech signal by using the clean speech signal and the distorted speech signal, and, after being trained, generates a clean speech signal in which distortion has been restored, and the second network provides, to the first network as Gaussian guidance, a Gaussian distribution including information about the clean speech by using the latent features extracted from the first network and the clean speech signal.

Description

Method for restoring distorted speech using Gaussian guidance, and a system and computer program for executing the same

[0001] The disclosed invention relates to a method for restoring distorted speech using Gaussian guidance for restoring various speech distortions that occur in combination, and a system for implementing the same.

[0002] Recently, there have been widespread attempts to move away from traditional signal processing methods and apply deep learning technology based on deep neural networks, which is rapidly evolving, to audio signal compression. The simplest way to apply deep learning technology to audio codecs is to replace the frequency transformation and inverse transformation processes with deep learning networks. For example, using a simple structure known as an autoencoder can replace the encoding and decoding processes of an audio codec. However, since this method does not offer high performance, there is a need to use a more suitable method: a deep learning network based on a generative model.

[0003] For example, a speech signal processing technology using flow-based deep learning generative models is WaveGlow, a neural vocoder technology. WaveGlow models the probability distribution using a function that bipartites the data dimension and performs an affine transformation. By designing an affine coupling layer for 1x1 invertible filtered data, it can effectively model the probability distribution of the data by generalizing the combination of each component. In particular, it generates speech signals by passing Mel-spectrograms, which serve as condition vectors determining the characteristics of the samples to be generated, through the deep learning network of the flow-based generative model. While WaveGlow has yielded efficient results as a neural vocoder that generates speech waveforms using only Mel-spectrograms, it has the problem of performance limitations in generating original speech.

[0004] Furthermore, speech enhancement technology, which improves signal quality by removing noise from noisy speech signals, has long been a core technology in the fields of voice communications, broadcasting, and media, and has been consistently in demand. With the recent advancement of machine learning technology, research is widely being conducted to design speech enhancement methods through learning, moving away from traditional signal processing techniques; generally, machine learning-based speech enhancement technologies demonstrate superior performance compared to signal processing-based technologies. Speech enhancement using machine learning is being attempted through various structures and methods. These include methods that remove noise components by predicting noise masks in the spectrogram domain, methods that generate enhanced speech signals by analyzing signals at various resolutions using U-Net, methods that generate high-quality speech signals using WaveNet to analyze temporal dependencies over a wide time domain, methods utilizing Generative Adversarial Networks (GANs), or methods that combine these approaches. However, these methods merely apply techniques widely used and proven effective in the existing machine learning field to speech signals and are not the optimal methods specifically tailored for speech signals. For example, WaveNet utilizes the characteristic that speech signals depend on past signals, but this is closer to utilizing the universal characteristics of audio signals rather than considering the unique characteristics of speech signals alone.

[0005] Conventional machine learning-based speech enhancement technology improves the performance of generative models by combining two different networks while utilizing various additional information as conditions to process various distorted speech that causes noise, reverberation, and bandwidth degradation. However, there are problems such as the need to use information known in advance or an increase in model complexity, such as the number of parameters or computational load.

[0006] The disclosed embodiment for solving these problems relates to a method for restoring distorted speech using Gaussian guidance and a system for executing the same, which can improve speech restoration performance by effectively utilizing conditioning information through Gaussian guidance obtained from clean speech.

[0007] A method for restoring distorted speech using Gaussian guidance according to a disclosed embodiment is a method for restoring distorted speech using Gaussian guidance performed by a system comprising at least one processor, comprising: a learning process in which a clean speech signal and a distorted speech signal are input to a speech restoration model combined with a second network based on a first network, and a Gaussian guidance is generated based on latent features extracted from the first network and the clean speech signal and provided to the first network as conditioning information to learn the speech restoration model; The method includes an inference process that restores a distorted voice signal and provides a clean voice signal by using a first network that uses Gaussian guidance sampled from a Gaussian distribution as conditioning information, excluding the second network in the learned voice restoration model, wherein the first network is trained to restore the clean voice signal using the clean voice signal and the distorted voice signal, and generates a clean voice signal with the distortion restored after training, and the second network provides a Gaussian distribution containing information about the clean voice to the first network as Gaussian guidance using latent features extracted from the first network and the clean voice signal.

[0008] Alternatively, the first network is composed of an input layer, downsampling, a bottleneck, upsampling, and an output layer.

[0009] Alternatively, the above learning process applies an adaptive time scale to estimate a time-dependent Gaussian distribution, scaling the Gaussian guidance according to the time step such that the conditioning information decreases as time elapses.

[0010] Alternatively, the first network is a score-based diffusion model, and the second network is a normalizing flow model.

[0011] Alternatively, the learning process comprises: a step of learning through a forward process in which a clean voice signal (x) with Gaussian noise added, a distorted voice signal (y), and a time-step (t) with a known time point are input to the first network, and the data distribution for the voice signal (x, y) is converted into a Gaussian distribution, and a reverse process in which noise is removed through sampling to generate voice sample data restored to a Gaussian distribution; a step in which the clean voice signal (x) and latent features (c) of the first network are input to the second network, and residual information between the clean voice signal (x) and latent features (c) is extracted and the residual information is modeled into a predefined Gaussian distribution (z); and a step in which the modeled Gaussian distribution (z) is provided to the first network as Gaussian guidance and conditioned.

[0012] A system for restoring distorted speech using Gaussian guidance according to a disclosed embodiment comprises: a memory for storing a plurality of neural network models including a speech restoration model in which a second network is combined based on a first network; A processor that trains the above neural network model and performs a speech restoration function for input data input through the trained neural network model; wherein the processor inputs a clean speech signal and a distorted speech signal to a speech restoration model combined with a second network based on a first network, generates a Gaussian guidance based on latent features extracted from the first network and the clean speech signal, and provides the Gaussian guidance to the first network as conditioning information to train the speech restoration model, excludes the second network from the trained speech restoration model, and restores the distorted speech signal using the first network which uses a Gaussian guidance sampled from a Gaussian distribution as conditioning information to provide a clean speech signal, wherein the first network is trained to restore the clean speech signal using the clean speech signal and the distorted speech signal, and generates a clean speech signal with the distortion restored after training, and the second network uses latent features extracted from the first network and the clean speech signal to provide a Gaussian distribution containing information about the clean speech to the first network It is provided as guidance.

[0013] A computer program stored on a computer-readable storage medium according to a disclosed embodiment, wherein the computer program, when executed on one or more processors, performs operations for restoring distorted speech using Gaussian guidance, and the operations include a learning operation in which a clean speech signal and a distorted speech signal are input to a speech restoration model combined with a second network based on a first network, and a Gaussian guidance is generated based on latent features extracted from the first network and the clean speech signal and provided to the first network as conditioning information to learn the speech restoration model; The method includes an inference operation that restores a distorted voice signal and provides a clean voice signal by using a first network that uses a Gaussian guidance sampled from a Gaussian distribution as conditioning information, excluding the second network from the learned voice restoration model; wherein the first network is trained to restore the clean voice signal using the clean voice signal and the distorted voice signal, and generates a clean voice signal with the distortion restored after training, and the second network provides a Gaussian distribution containing information about the clean voice to the first network as a Gaussian guidance using latent features extracted from the first network and the clean voice signal.

[0014] The method for restoring distorted speech using Gaussian guidance according to the disclosed embodiment and the system for executing the same can learn a speech restoration model combined with a normalized flow model based on a diffusion model using Gaussian guidance obtained from clean speech, and has the effect of effectively removing various speech distortions occurring in combination through the learned speech restoration model to restore high-quality clean speech.

[0015] In addition, the method for restoring distorted speech using Gaussian guidance according to the disclosed embodiment and the system for executing the same have the effect of improving speech restoration performance without increasing the complexity of the model by integrating Gaussian guidance into each block of the speech restoration model to enable fine-tuning, thereby inducing the generation of a Gaussian distribution closer to clean speech, and by removing the normalized flow model and using a score-based diffusion model during the inference process.

[0016] FIG. 1 is a control block diagram of a system for implementing a method to restore distorted speech using disclosed Gaussian guidance.

[0017] Figure 2 is a diagram illustrating the learning process of a speech enhancement model of a system that implements a method for restoring distorted speech using disclosed Gaussian guidance.

[0018] Figure 3 is a diagram illustrating the inference process of a speech enhancement model of a system that executes a method to restore distorted speech using disclosed Gaussian guidance.

[0019] FIG. 4 is a diagram illustrating the structure of a second network according to an embodiment of the present invention.

[0020] Figure 5 is a flowchart illustrating a method for restoring distorted speech using disclosed Gaussian guidance.

[0021] Throughout the specification, the same reference numerals refer to the same components. This specification does not describe all elements of the embodiments, and general content in the art to which the invention pertains or content that overlaps between embodiments is omitted.

[0022] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0023] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0024] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0025] In addition, terms such as "~part," "~unit," "~block," "~part," and "~module" may refer to a unit that processes at least one function or operation. For example, the above terms may refer to at least one piece of hardware such as an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit), at least one piece of software stored in memory, or at least one process processed by a processor.

[0026] The symbols attached to each step are used to identify each step and do not indicate the order of the steps relative to one another; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0027] Hereinafter, with reference to the attached drawings, embodiments relating to a method for restoring distorted speech using Gaussian guidance according to the disclosed embodiment and a system for executing the same will be described in detail.

[0028] FIG. 1 is a control block diagram of a system for implementing a method to restore distorted speech using disclosed Gaussian guidance.

[0029] A system (1) that executes a method for restoring distorted voice using Gaussian guidance can be implemented as a computer or portable terminal (hereinafter user terminal, 3) that can connect to a communication network such as the internet. Here, the computer includes, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser, and the portable terminal can be implemented as, for example, any type of handheld-based wireless communication device such as a smartphone, etc., as a wireless communication device that ensures portability and mobility.

[0030] Referring to FIG. 1, a user terminal (3) may include a communication interface (11) for receiving various data, such as a neural network model or training data required for speech recognition, from an external server (2); a memory (12) for storing the received training data and neural network model; an input unit (13) for receiving text to be converted into speech or other various user input commands; a processor (10) for controlling the overall system; and an output unit (14) for outputting the synthesized speech performed by the processor (10) as sound. However, since FIG. 1 is merely an example, the user terminal (3) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the user terminal (3).

[0031] Specifically, the communication interface (11) may include one or more components that enable communication with an external communication network, and may include, for example, at least one of a short-range communication module, a wired communication module, and a wireless communication module.

[0032] The short-range communication module may include various short-range communication modules that transmit and receive signals using a wireless communication network at short range, such as a Bluetooth module, an infrared communication module, an RFID (Radio Frequency Identification) communication module, a WLAN (Wireless Local Access Network) communication module, an NFC communication module, and a Zigbee communication module.

[0033] Wired communication modules may include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as various cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0034] In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, the wireless communication module may include wireless communication modules that support various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), and LTE (Long Term Evolution).

[0035] The memory (12) may store a neural network model required to implement a method for fine-tuning a voice restoration model for restoring a disclosed distorted voice, an algorithm required for the operation of the processor (10), or a program for implementing the algorithm. To this end, the memory (12) may be implemented as at least one of a non-volatile memory device such as a cache, ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and flash memory, a volatile memory device such as RAM (Random Access Memory), or a storage medium such as a hard disk drive (HDD) and a CD-ROM, but is not limited thereto.

[0036] The input unit (13) can receive various input commands. Specifically, the input unit (13) may include a microphone that receives voice that speaks into a video containing text as subtitles, in addition to receiving input text. Furthermore, the input unit (13) may include hardware devices such as various buttons, switches, pedals, keyboards, mice, trackballs, various levers, handles, or sticks for receiving execution commands required for the operation of the processor (10). Additionally, the input unit (13) may include a GUI (Graphical User Interface), i.e., a software device such as a touch pad, for user input commands. The touch pad may be implemented as a touch screen panel (TSP) and form a layered structure with the display of the output unit (14).

[0037] The output unit (14) may include a speaker that outputs the recognized voice as sound, as well as a display for outputting the learning results or inference results of a neural network model. The display may be provided as a Cathode Ray Tube (CRT), Digital Light Processing (DLP) panel, Plasma Display Panel, Liquid Crystal Display (LCD) panel, Electro Luminescence (EL) panel, Electrophoretic Display (EPD) panel, Electrochromic Display (ECD) panel, Light Emitting Diode (LED) panel, or Organic Light Emitting Diode (OLED) panel, but is not limited thereto.

[0038] The processor (10) can fine-tune or train a neural network model stored in memory (12) and perform speech recognition functions through the trained neural network model for input text.

[0039] Specifically, the processor (10) learns a speech restoration model that combines two generative models, a first network and a second network, and uses the learned speech restoration model to restore distorted speech. At this time, the processor (10) may use a diffusion model as the first network and a normalization flow model as the second network, and during inference, the second network is excluded from the speech restoration model to use the first network.

[0040] A specific method for the processor (10) to restore distorted speech using Gaussian guidance will be described later through other drawings below.

[0041] Meanwhile, the processor (10) may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of such data processing devices embedded in hardware may include, but are not limited to, processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a Graphics Processing Unit (GPU), and a neural Processing Unit (NPU). The processor (10) may be provided in multiple units.

[0042] The hardware configurations provided in the user terminal (10) can exchange data and signals via Network Termination (NT) of a digital network such as ISDN (Integrated Services Digital Network).

[0043] FIG. 2 is a diagram illustrating the learning process of a speech enhancement model of a system that executes a method for restoring distorted speech using the disclosed Gaussian guidance, and FIG. 3 is a diagram illustrating the inference process of a speech enhancement model of a system that executes a method for restoring distorted speech using the disclosed Gaussian guidance.

[0044] The first network (110) is a score-based diffusion model and consists of an input layer, downsampling, bottleneck, upsampling, and output layer. The architecture of the first network can use a U-net.

[0045] Score-based diffusion models are diffusion probabilistic models that use a Markov chain sampling strategy to generate a data distribution from a noise distribution. These diffusion models consist of a forward process and a reverse process. The forward process transforms the data distribution into a prior distribution (e.g., a Gaussian distribution), and the reverse process generates data from the prior distribution by progressively removing noise using a sampling method (e.g., Langevin dynamics).

[0046] During the learning process, the speech restoration model (100) is trained in a combined form of a first network (110) and a second network (120) for providing Gaussian guidance to the first network (110) as conditioning information.

[0047] The first network (110) is input with x_{t}, which is a form in which Gaussian noise is added to a clean voice signal, a distorted voice signal (y), and a time-step (t) that can determine the time point.

[0048] The second network (120) is a normalized flow model, into which a clean voice signal (x) and latent features (c) of the first network (110) are input, and which models a Gaussian distribution (z). The modeled Gaussian distribution (z) is conditioned by being added to the last two upsampling networks of the first network (110).

[0049] The normalized flow model aims to estimate the probabilistic path between x1 and x0 sampled from the data and prior distribution. To achieve this, it trains a neural network to estimate the probabilistic path and uses the optimal transport path to improve sampling efficiency.

[0050] The first network (110) can use a model called NCSN ++ (Noise Conditional Score Network) or SGMSE+ (score-based generative model for speech enhancement), and if SGMSE+ is extended to use as a flow model, it can be named FGMSE+ (Flow-based generative model for speech enhancement).

[0051] As illustrated in FIG. 3, in the inference process, the voice restoration model (100) does not use the second network (120), and Gaussian guidance, which is a feature sampled from a Gaussian distribution, is used as conditioning information, and can improve voice quality by restoring various distortions (noise, reverberation and bandwidth degradation) that occurred in the voice and generating a clean voice.

[0052] FIG. 4 is a diagram illustrating the structure of a second network according to an embodiment of the present invention.

[0053] As illustrated in FIG. 4, the second network (120) flips the clean voice signal (x), splits the clean voice signal into low-dimensional embeddings x1 and x2, applies a rational quadratic transform to perform parallel computation, and then connects z1 and z2 using a Concat function. During the conditioning process, the second network (120) adds a convolution block (121) to adjust the dimensions of the clean voice signal (x), latent feature (c), and Gaussian distribution (z), and the magnitude of the latent feature (c).

[0054] This second network (120) is a normalized flow model, and a clean voice signal (x) and latent features (c) extracted from the first network (110) are input. The latent features (c) serve as conditioning information for the second network (120), and the conditional data distribution (p(x|c)) is normalized to a prior distribution, i.e., a Gaussian distribution. Therefore, the log likelihood of the conditional data can be calculated as shown in Equation 1 below.

[0055]

[0056] In Equation 1, p(x|c) represents the probability distribution of x under condition c, p(z|c) represents the probability distribution of z under condition c, and det(∂f(x) / ∂x) is the Jacobian determinant of the function f(x), representing the change in probability density (scale adjustment) when x is transformed into z. In particular, det(∂f(x) / ∂x) can be used to map the distribution of data to a prior distribution in a normalizing flow transformation. Additionally, f(x) is an inversely transformable function that transforms data x into a latent variable z.

[0057] The negative log likelihood for learning the second network (120) is decomposed into Kullback-Leibler divergence and entropy as shown in Equation 2 below.

[0058]

[0059] In Equation 2, H(x|c) and q(z) are the constant data entropy and prior distribution, respectively. Based on Equation 1, c and z can be separated by minimizing KL(p(z|c)|q(z)). That is, the second network (120) can extract residual information between the clean voice signal (x) and the latent feature (c), and the residual information is modeled as a predefined Gaussian distribution (z).

[0060] Figure 5 is a flowchart illustrating a method for restoring distorted speech using disclosed Gaussian guidance.

[0061] A method for restoring distorted speech using Gaussian guidance performed by the disclosed system (1) involves training a speech restoration model (100) on a pre-configured architecture. Here, the pre-configured architecture may be a score-based diffusion model and a normalization flow model, which are types of generative models.

[0062] The first network (110) is input with a spectrogram of a clean voice signal and a spectrogram of a distorted voice signal, and is trained through a forward process and a reverse process, and the trained first network generates a voice sample with the distortion restored through several time steps (S10).

[0063] When a clean voice signal (x) and a latent feature (c) extracted from the first network (110) as conditioning information are input to the second network (120), it is converted into a Gaussian distribution (S20). At this time, the Gaussian distribution models information about the clean voice signal that the latent feature of the first network (110) does not possess.

[0064] In the learning process, a Gaussian distribution generated from the second network (120) is provided to the first network (110) as Gaussian guidance to perform a conditioning process, and the first network (110) receives a label for the target voice to be restored in the conditioning process as Gaussian guidance (S30).

[0065] At this time, the system (1) applies an adaptive time scale to estimate a time-dependent Gaussian distribution and scales the Gaussian guidance according to the time-step (t) so that the conditioning information decreases as time progresses.

[0066] The Gaussian guidance used as conditioning information reflects the gap between the clean voice signal and the latent features extracted from the first network (110), and has the characteristic of decreasing over time. Therefore, the system (1) scales the Gaussian guidance to (1-t) so that the conditioning information gradually decreases over time.

[0067] The voice restoration model (100) performs learning by combining the second network (120) based on the first network (110) (S40), and performs an inference process to restore distorted voice and generate clean voice by using only the first network (110) without using the second network (120) during the inference process (S50).

[0068] Accordingly, the disclosed speech restoration model (100) does not significantly increase the complexity of the model compared to the existing baseline model. As such, since the conditioning information generated from the normalized flow model during the learning process of the disclosed speech restoration model (100) is modeled as a Gaussian distribution, values ​​sampled from a random Gaussian distribution can be used as conditioning information in the diffusion model without the normalized flow model during the inference process.

[0069] The method for restoring distorted speech using the disclosed Gaussian guidance can be named FLOWER (FLOW-based Estimated Gaussian guidance for general speech Restoration).

[0070] Although speech in real-world environments is influenced by various factors, current speech restoration technologies have focused on restoring only a single distortion. However, the disclosed speech restoration model can restore distorted speech—where various distortions including noise, reverberation, and bandwidth degradation occur simultaneously—into clear speech. To this end, the disclosed speech restoration model can improve speech restoration performance by effectively utilizing conditioning information through Gaussian guidance obtained from clear speech (or a clear speech signal). Furthermore, the disclosed speech restoration model integrates Gaussian guidance into each block to enable fine-tuning, thereby inducing the generation of a distribution closer to clear speech, and efficiently utilizes Gaussian guidance by introducing an adaptive time scale. Additionally, the disclosed speech restoration model can improve performance without increasing model complexity because it eliminates the normalized flow model during the inference process and uses a score-based diffusion model. In addition, the disclosed speech restoration model improves sampling efficiency by combining a normalized flow model based on a score-based diffusion model and enables faster and more accurate sampling by using an optimal transport path, thereby demonstrating better performance in speech restoration tasks with various distortions.

[0071] Although FIG. 5 describes each process as being executed sequentially, this is merely an illustrative explanation of the technical concept of one embodiment of the present invention. In other words, a person skilled in the art to which one embodiment of the present invention belongs can modify and adapt it in various ways, such as changing the order described in each figure or executing one or more of the processes in parallel, without departing from the essential characteristics of one embodiment of the present invention; therefore, FIG. 5 is not limited to a chronological order.

[0072] Meanwhile, the processes illustrated in FIG. 5 can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. That is, a computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.). In addition, computer-readable recording media can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner.

Claims

1. A method for restoring distorted speech using Gaussian guidance, performed by a system comprising at least one processor, wherein A learning process in which a clean speech signal and a distorted speech signal are input to a speech restoration model combined with a second network based on a first network, and a Gaussian guidance is generated based on latent features extracted from the first network and the clean speech signal and provided to the first network as conditioning information to train the speech restoration model; and An inference process comprising: providing a clean voice signal by restoring a distorted voice signal using the first network, which uses Gaussian guidance sampled from a Gaussian distribution as conditioning information, excluding the second network from the learned voice restoration model; The first network is trained to restore the clean voice signal using the clean voice signal and the distorted voice signal, and after training, generates a clean voice signal with the distortion restored. The second network above provides the first network with a Gaussian distribution containing information about the clean voice as Gaussian guidance using latent features extracted from the first network and the clean voice signal. method.

2. In Paragraph 1, The above-mentioned first network is, It consists of an input layer, downsampling, bottleneck, upsampling, and output layer. method.

3. In Paragraph 1, The above learning process is, Scaling the Gaussian guidance according to a time step by applying an adaptive time scale to estimate a time-dependent Gaussian distribution so that conditioning information decreases as time progresses, method.

4. In Paragraph 1, The first network is a score-based diffusion model, and the second network is a normalizing flow model. method.

5. In Paragraph 4, The above learning process is, A step of learning through a forward process that converts the data distribution of the voice signal (x, y) into a Gaussian distribution and a reverse process that generates voice sample data restored to a Gaussian distribution by removing noise through sampling, wherein a clean voice signal (x), a distorted voice signal (y), and a time-step (t) with a known time point are input into the first network; A step of inputting the clean voice signal (x) and the latent feature (c) of the first network into the second network, extracting residual information between the clean voice signal (x) and the latent feature (c), and modeling the residual information as a predefined Gaussian distribution (z); and A step of providing the modeled Gaussian distribution (z) to the first network as Gaussian guidance and conditioning it; A method that includes 6. As a system for restoring distorted speech using Gaussian guidance, A memory for storing a plurality of neural network models, including a speech restoration model in which a second network is combined based on a first network; A processor that trains the above neural network model and performs a speech restoration function for input data input through the trained neural network model; The above processor is, A clean speech signal and a distorted speech signal are input to a speech restoration model combined with a second network based on a first network, and a Gaussian guidance is generated based on latent features extracted from the first network and the clean speech signal and provided to the first network as conditioning information to train the speech restoration model. Excluding the second network from the learned speech restoration model, the distorted speech signal is restored using the first network that uses Gaussian guidance sampled from a Gaussian distribution as conditioning information to provide a clean speech signal, The first network is trained to restore the clean voice signal using the clean voice signal and the distorted voice signal, and after training, generates a clean voice signal with the distortion restored. The second network above provides the first network with a Gaussian distribution containing information about the clean voice as Gaussian guidance using latent features extracted from the first network and the clean voice signal. System.

7. A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations to restore distorted speech using Gaussian guidance, and The above operations are, A learning operation in which a clean speech signal and a distorted speech signal are input to a speech restoration model combined with a second network based on a first network, and a Gaussian guidance is generated based on latent features extracted from the first network and the clean speech signal and provided to the first network as conditioning information to train the speech restoration model; and Including an inference operation that restores a distorted speech signal and provides a clean speech signal using the first network, which uses Gaussian guidance sampled from a Gaussian distribution as conditioning information, while excluding the second network from the learned speech restoration model; The first network is trained to restore the clean voice signal using the clean voice signal and the distorted voice signal, and after training, generates a clean voice signal with the distortion restored. The second network above provides the first network with a Gaussian distribution containing information about the clean voice as Gaussian guidance using latent features extracted from the first network and the clean voice signal. Computer program.

Images