Packet loss concealment system and training method for packet loss concealment system

Abstract

A packet loss concealment system according to one disclosed embodiment comprises: a memory storing an artificial intelligence model composed of a plurality of residual layers; and a processor for training the artificial intelligence model, wherein the processor is configured to: input, to the artificial intelligence model, speech data to which Gaussian noise has been added and noise level embedding; generate a first combined value by combining local conditioning information with the speech data and the noise level embedding through a first feature-wise linear modulation (FiLM) layer; generate a second combined value by combining packet loss embedding with the first combined value through a second FiLM layer; and output noise estimated at each time step by the residual layers through a reverse process.

Description

Packet Loss Hiding System and Learning Method for Packet Loss Hiding System

[0001] The present invention relates to a packet loss hiding system based on a diffusion model and a method for learning the packet loss hiding system.

[0002] Packet Loss Concealment (PLC) is a technology used to minimize voice quality degradation caused by packet loss during network transmission. In particular, PLC is primarily utilized in real-time voice streaming environments such as voice calls or VoIP (Voice over IP). PLC maintains natural voice quality by predicting or correcting voice data from lost packets.

[0003] Conventional PLC systems have evolved from classical methods that fill in missing parts with previous frames to models based on deep learning models. For example, prior paper 1 is a representative PLC system based on a predictive model rather than a generative model. It is a model that utilizes the structure of a speech recognition system and was introduced at ICASSP 2023. As another example, prior paper 2 is a PLC system that utilizes a GAN model, which is one of the generative models.

[0004] [Prior paper 1] Viet-Anh Nguyen, Anh HT Nguyen, and Andy WH Khong, "Improving performance of real-time full-band blind packet-loss concealment with predictive network," in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1-5.

[0005] [Previous paper 2] J. Wang, Y. Guan, C. Zheng, R. Peng, and X. Li, "A temporal-spectral generative adversarial network based end-to-end packet loss concealment for wideband speech transmission," The Journal of the Acoustical Society of America, vol. 150, no. 4, pp. 2577-2588, 2021.

[0006] A disclosed embodiment relates to a packet loss hiding system and a method for learning a packet loss hiding system that effectively restores packet loss compared to conventional technology by utilizing two types of conditioning information in a diffusion model to provide information about lost packets and proceeding with packet generation through this.

[0007] A packet loss hiding system according to a disclosed embodiment includes: a memory for storing an artificial intelligence model composed of a plurality of residual layers; and a processor for training the artificial intelligence model. The processor inputs voice data with Gaussian noise added and a noise level embedding to the artificial intelligence model, generates a first combined value by combining local conditioning information with the voice data and the noise level embedding through a first FiLM (Feature-wise Linear Modulation) layer, generates a second combined value by combining a packet loss embedding with the first combined value through a second FiLM layer, and outputs noise estimated by the residual layer at each time step through a reverse process.

[0008] The above processor can train the artificial intelligence model based on the difference between the estimated noise and the noise to be compared.

[0009] The processor can input the local conditioning information to a first 1x1 convolution layer, input a first output value output from the first 1x1 convolution layer to a Bi-DilConv layer, and input a second output value output from the Bi-DilConv layer to the first FiLM layer.

[0010] The processor can input the packet loss embedding to a second 1x1 convolution layer and input a third output value output from the second 1x1 convolution layer to the second FiLM layer.

[0011] A learning method for a packet loss hiding system according to another disclosed embodiment comprises: inputting voice data with Gaussian noise added and a noise level embedding to a residual layer; generating a first combined value by combining local conditioning information with the voice data and the noise level embedding through a first FiLM layer; generating a second combined value by combining a packet loss embedding with the first combined value through a second FiLM layer; and outputting the noise estimated by the residual layer at each time step through a reverse process.

[0012] It may further include training the artificial intelligence model based on the difference between the estimated noise and the noise to be compared.

[0013] Generating the first combined value may include inputting the local conditioning information into the first 1x1 convolution layer, inputting the first output value output from the first 1x1 convolution layer into the Bi-DilConv layer, and inputting the second output value output from the Bi-DilConv layer into the first FiLM layer.

[0014] Generating the second combined value may include inputting the packet loss embedding to the second 1x1 convolution layer and inputting the third output value output from the second 1x1 convolution layer to the second FiLM layer.

[0015] The disclosed computer program is a computer program stored on a computer-readable storage medium, wherein, when executed on one or more processors, the computer program performs operations for conditioning a diffusion model, the operations comprising: inputting voice data with Gaussian noise added and a noise level embedding to a residual layer; generating a first combined value by combining local conditioning information with the voice data and the noise level embedding through a first FiLM (Feature-wise Linear Modulation) layer; generating a second combined value by combining a packet loss embedding with the first combined value through a second FiLM layer; and outputting the noise estimated by the residual layer at each time step through a reverse process.

[0016] It may further include training the residual layer based on the difference between the estimated noise and the noise to be compared.

[0017] Generating the first combined value may include inputting the local conditioning information into the first 1x1 convolution layer, inputting the first output value output from the first 1x1 convolution layer into the Bi-DilConv layer, and inputting the second output value output from the Bi-DilConv layer into the first FiLM layer.

[0018] Generating the second combined value may include inputting the packet loss embedding to the second 1x1 convolution layer and inputting the third output value output from the second 1x1 convolution layer to the second FiLM layer.

[0019] A packet loss hiding system and a learning method for a packet loss hiding system according to a disclosed embodiment can effectively restore packet loss compared to conventional technology by utilizing two types of conditioning information in a diffusion model to provide information about lost packets and proceeding with packet generation through this.

[0020] FIG. 1 is a control block diagram of a packet loss hiding system according to a disclosed embodiment.

[0021] Figure 2 is a flowchart illustrating a learning method for a disclosed packet loss hiding system.

[0022] Figure 3 is a diagram to supplement the explanation of the learning method of Figure 2.

[0023] Throughout the specification, the same reference numerals refer to the same components. This specification does not describe all elements of the embodiments, and general content in the art to which the invention pertains or content that overlaps between embodiments is omitted.

[0024] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0025] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0026] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0027] In addition, terms such as "~part," "~unit," "~block," "~part," and "~module" may refer to a unit that processes at least one function or operation. For example, the above terms may refer to at least one piece of hardware such as an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit), at least one piece of software stored in memory, or at least one process processed by a processor.

[0028] The symbols attached to each step are used to identify each step and do not indicate the order of the steps relative to one another; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0029] Hereinafter, a packet loss hiding system and a learning method for the packet loss hiding system according to the disclosed embodiment will be described in detail with reference to the attached drawings.

[0030] FIG. 1 is a control block diagram of a packet loss hiding system according to a disclosed embodiment.

[0031] In the disclosed embodiment, the packet loss hiding system (1) can be implemented as a computer or portable terminal (hereinafter user terminal, 3) capable of connecting to a communication network such as the Internet. Here, the computer includes, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser, and the portable terminal can be implemented as, for example, any type of handheld-based wireless communication device such as a smartphone, etc., as a wireless communication device that ensures portability and mobility.

[0032] Referring to FIG. 1, a user terminal (3) may include a communication interface (11) for receiving voice data composed of packets and an artificial intelligence model required for learning a disclosed packet loss hiding system from an external server (2), a memory (12) for storing the received voice data and the artificial intelligence model, an input unit (13) for receiving operation commands for learning and inference of the artificial intelligence model from a user, a processor (10) for controlling the overall system, and an output unit (14) for displaying the learning results of the artificial intelligence model performed by the processor (10). However, since FIG. 1 is merely an example, the user terminal (3) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the user terminal (3).

[0033] Specifically, the communication interface (11) may include one or more components that enable communication with an external communication network, and may include, for example, at least one of a short-range communication module, a wired communication module, and a wireless communication module.

[0034] The short-range communication module may include various short-range communication modules that transmit and receive signals using a wireless communication network at short range, such as a Bluetooth module, an infrared communication module, an RFID (Radio Frequency Identification) communication module, a WLAN (Wireless Local Access Network) communication module, an NFC communication module, and a Zigbee communication module.

[0035] Wired communication modules may include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as various cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0036] In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, the wireless communication module may include wireless communication modules that support various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), and LTE (Long Term Evolution).

[0037] The memory (12) may store an artificial intelligence model required to implement the learning method of the disclosed packet hiding loss system, an algorithm required for the operation of the processor (10), or a program for implementing the algorithm. To this end, the memory (12) may be implemented as at least one of a non-volatile memory device such as a cache, ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and flash memory, a volatile memory device such as RAM (Random Access Memory), or a storage medium such as a hard disk drive (HDD) and CD-ROM, but is not limited thereto.

[0038] The input unit (13) can receive various input commands. Specifically, the input unit (13) may include hardware devices such as various buttons, switches, pedals, keyboards, mice, trackballs, various levers, handles, or sticks for receiving execution commands required for the operation of the processor (10). Additionally, the input unit (13) may include a GUI (Graphical User Interface), i.e., a software device such as a touch pad, for user input commands. The touch pad may be implemented as a touch screen panel (TSP) and form a layered structure with the display of the output unit (14).

[0039] The output unit (14) may include a speaker that outputs voice data restored based on lost packets, as well as a display for outputting the learning results or inference results of an artificial intelligence model. The display may be provided as a Cathode Ray Tube (CRT), Digital Light Processing (DLP) panel, Plasma Display Panel, Liquid Crystal Display (LCD) panel, Electro Luminescence (EL) panel, Electrophoretic Display (EPD) panel, Electrochromic Display (ECD) panel, Light Emitting Diode (LED) panel, or Organic Light Emitting Diode (OLED) panel, but is not limited thereto.

[0040] The processor (10) trains an artificial intelligence model based on Gaussian noise added to speech data (Diffused speech) received from an external source or a user, noise level embedding, local conditioning, and packet loss embedding. Specifically, the processor (10) recovers lost packets through a diffusion model composed of multiple residual layers, and trains the residual model through the training method of the diffusion model, namely the forward process and the reverse process. In particular, according to the disclosed example, the processor (10) performs local conditioning and packet loss embedding conditioning in the reverse process and trains a residual model that recovers lost packets differently from conventional methods. The specific method by which the processor (10) trains the residual model will be described later through the drawings below.

[0041] Meanwhile, the processor (10) may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of such data processing devices embedded in hardware may include, but are not limited to, processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a Graphics Processing Unit (GPU), and a neural Processing Unit (NPU). The processor (10) may be provided in multiple units.

[0042] The hardware configurations provided in the user terminal (10) can exchange data and signals via Network Termination (NT) of a digital network such as ISDN (Integrated Services Digital Network).

[0043] FIG. 2 is a flowchart illustrating a learning method for a disclosed packet loss hiding system. FIG. 3 is a diagram supplementing the explanation of the learning method of FIG. 2. To avoid redundant explanations, they are described together below.

[0044] Referring to FIGS. 2 and FIGS. 3 together, the packet loss hiding system (1) inputs voice data with Gaussian noise added and noise level embeddings into a residual layer (S10).

[0045] The artificial intelligence model disclosed herein may include multiple (e.g., 32) residual layers (110 in FIG. 3). The residual layer (110) is a neural network that learns the difference between the input and output values ​​of each layer, i.e., the residual, which is advantageous for solving the gradient loss problem that occurs as the layers become deeper during the process of learning a deep neural network. The residual layer (110) may be configured with a structure that performs skip connections and transmits the input value of each layer to the next layer.

[0046] The packet loss hiding system (1) inputs the noise-added voice data and noise level embeddings into a 1x1 convolution layer (Conv 1x1, 101 in FIG. 3) or a Fully Connected Layer (FC, 102, 103 in FIG. 3) before inputting them into the residual layer (110).

[0047] Here, the noise-added voice data (Diffused Speech of FIG. 3) is input by an external server (2) or a user in the form of Gaussian noise added to clean voice data without noise. The noise-added voice data passes through a 1x1 convolution layer (101) and is converted into latent features that can be utilized in the residual layer (110).

[0048] The noise level embedding (Noise Level Embedding of FIG. 3) is a value that indicates to the artificial intelligence model how much noise has been input into the voice data, and is information regarding the time step of the diffusion model. The noise level embedding is also input by an external server (2) or a user. The noise level embedding passes through multiple FCs (102, 103, 106) and is reduced to a size (dimension) that can be utilized in the residual layer (110).

[0049] The noise-added speech data and noise level embeddings output from the 1 x 1 convolution layer (101) and the final FC (106) are passed through the residual layer (110) and utilized as training data through the reverse process of the diffusion model. That is, the diffusion model composed of the residual layer (110) takes the noise-added speech data, the noise level embeddings, and the conditioning information described later as inputs, and outputs the noise (Noise in FIG. 3) to be removed at the time step (step t).

[0050] Meanwhile, the diffusion model is an artificial intelligence model based on diffusion probability that uses a Markov chain sampling strategy to generate a new data distribution from a noise distribution. The diffusion model consists of a forward process and a reverse process. In the forward process, clean speech data is gradually transformed into Gaussian noise, and in the reverse process, the noise added at each step (time step) is estimated and removed, thereby restoring the data to its original clean speech form free of noise. The entire process is defined by the product of Markov transition probabilities and can be defined as shown in the following Equation 1.

[0051] [Mathematical Formula 1]

[0052]

[0053] Here, q(x_{t}|x_{t-1}) represents the Markov transition probability, where t is the time step and x is the input speech data. In the forward process, Gaussian noise is gradually added at each time step, and latent variables x_{t}, t, and x_{T} are obtained from the speech data x{0}. The backward process aims to restore the initial clean signal from the Gaussian noise, and iteratively recovers x_{t-1} at each step t. The initiated AI model including the residual layer is trained in a direction that minimizes the difference between the generated sample (estimated noise) and the data distribution (noise to be compared).

[0054] The packet loss hiding system (1) generates a first combined value by combining local conditioning information with voice data and noise level embeddings through the first FiLM layer (113) (S20).

[0055] The disclosed packet loss detection system (1) uses two pieces of information from the diffusion model as conditioning features.

[0056] First, the packet loss detection system (1) utilizes local conditioning information (local conditioning in FIG. 3). The local conditioning information provides the information necessary for the diffusion model to restore the voice waveform based on the lost signal. The local conditioning information converts the lost signal into latent features through a 1x1 convolution layer (104) and transmits the local conditioning information to the first FiLM layer (113) through a Bi-DilConv layer (112) having a wide acceptance area.

[0057] Here, the Bi-DiConv (Bidirectional Dilated Convolution) layer (111, 112) is a neural network used to effectively capture more information contained in speech data through bidirectional dilated convolution. By processing information bidirectionally, such as with a bidirectional LSTM or a bidirectional GRU, the Bi-DiConv layer (111, 112) enables local conditioning information containing various information to be well reflected in the first FiLM layer (113). In the disclosed example, the Bi-DiConv layer (111, 112) may be set to a kernel size of 3, but is not limited thereto.

[0058] The packet loss detection system (1) combines local conditioning information with noise-added voice data and noise level embeddings through the first FiLM layer (113). The first FiLM layer (113) provides a normalization effect, thereby enabling the disclosed artificial intelligence model to dynamically adjust the modeling. The process of combining local conditioning information by film conditioning to produce a first combined value is as shown in Equation 2.

[0059] [Mathematical Formula 2]

[0060]

[0061] Here, / hat{x}_{ / bar{ / alpha}} is a feature formed by combining local conditioning information / dot{x}_{l} with the input / dot{x}_{ / bar{ / alpha}} of the disclosed artificial intelligence model. r_{i} and h_{i} represent the i-th layer.

[0062] The disclosed first FiLM layer may consist of at least two linear layers and extracts weights and biases.

[0063] The packet loss recognition system (1) generates a second combined value by combining the packet loss embedding with the first combined value through the second FiLM layer (114) (S30).

[0064] The packet loss detection system (1) utilizes two types of information as conditioning information, among which packet loss embedding information is utilized as conditioning information. Packet loss embedding information is used to provide information about packets of lost voice data and consists of binary values ​​of 0 and 1. The value of the packet loss embedding passes through a 1x1 convolution layer (105) and is then conditioned to a first combined value output from the first FiLM layer (113) through the second FiLM layer (114).

[0065] The second FiLM layer (114) may also be composed of at least two linear layers.

[0066] The second FiLM layer (114) provides a normalization effect, thereby enabling the disclosed artificial intelligence model to dynamically consider lost packet information through learning. The process of adding the packet loss embeddings to the first combined value by film conditioning to calculate the second combined value is as shown in Equation 3.

[0067] [Mathematical Formula 3]

[0068]

[0069] Equation 3 represents the process of combining the input feature / hat{x}_{ / bar{ / alpha}} and the packet loss embedding feature / hat{x}_{e}. Similarly, r and h represent the linear layer in each layer, which takes the packet loss embedding feature as input and outputs weights and biases. The output weights and biases are computed by multiplying and adding them, respectively, to the features output by the residual layer.

[0070] Through a reverse process, the residual layer (110) outputs the estimated noise at each time step (S40).

[0071] As illustrated in FIG. 3, the disclosed artificial intelligence model inputs weights and biases (σ) output through the second FiLM layer (114) into a 1x1 convolution layer (115) and generates output values. The packet loss detection system (1) combines multiple output values ​​output from multiple residual layers (110) or finally outputs noise through a 1x1 convolution layer (116).

[0072] The packet loss detection system (1) performs training of an artificial intelligence model based on the difference between the estimated noise and the noise to be compared (S50).

[0073] As described above, the disclosed packet loss detection system (1) may include a plurality of residual layers (110), a plurality of 1x1 convolution layers (101, 104, 105, 116), and FCs (102, 103). The disclosed packet loss detection system (1) compares the noise output from the bidirectional process with the noise contained in the actual voice data, and while restoring the output noise to the voice data without initial noise, adjusts the parameters of all layers included in the artificial intelligence model. Through this, it is trained as an artificial intelligence model that recovers packets lost in a communication system.

[0074] Although FIGS. 2 and FIGS. 3 describe each process as being executed sequentially, this is merely an illustrative explanation of the technical concept of one embodiment of the present invention. In other words, a person skilled in the art to which one embodiment of the present invention belongs can modify and adapt the process in various ways, such as changing the order described in each figure or executing one or more of the processes in parallel, without departing from the essential characteristics of one embodiment of the present invention; therefore, FIGS. 2 and FIGS. 3 are not limited to a chronological order.

[0075] Meanwhile, the processes illustrated in FIGS. 2 and 3 can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. That is, a computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.). In addition, the computer-readable recording medium can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner.

Claims

1. Memory for storing an artificial intelligence model composed of multiple residual layers; A processor for training the above artificial intelligence model; comprising, The above processor is, Voice data with added Gaussian noise and noise level embeddings are input into the above artificial intelligence model, and A first combined value is generated by combining local conditioning information with the voice data and the noise level embedding through a first FiLM (Feature-wise Linear Modulation) layer, and A second combined value is generated by combining the packet loss embedding with the first combined value through the second FiLM layer, and A packet loss hiding system in which the residual layer outputs estimated noise at each time step through a reverse process.

2. In Paragraph 1, The above processor is, A packet loss recognition system that trains the artificial intelligence model based on the difference between the estimated noise and the noise to be compared.

3. In Paragraph 1, The above processor is, The above local conditioning information is input into the first 1x1 convolution layer, and The first output value output from the first 1x1 convolution layer is input to the Bi-DilConv layer, and A packet loss hiding system that inputs a second output value output from the above Bi-DilConv layer to the above first FiLM layer.

4. In Paragraph 1, The above processor is, The above packet loss embedding is input to the second 1x1 convolution layer, and A packet loss hiding system that inputs a third output value output from the second 1x1 convolution layer to the second FiLM layer.

5. Input the speech data with Gaussian noise added and the noise level embeddings into the residual layer; Generating a first combined value by combining local conditioning information with the voice data and the noise level embedding through the first FiLM layer; A second combined value is generated by combining the packet loss embedding with the first combined value through the second FiLM layer; A learning method for a packet loss recognition system comprising: the residual layer outputting estimated noise at each time step through a reverse process.

6. In Paragraph 5, A method for learning a packet loss recognition system, further comprising: learning the artificial intelligence model based on the difference between the estimated noise and the noise to be compared.

7. In Paragraph 5, Generating the above first combination value is, The above local conditioning information is input into the first 1x1 convolution layer, and The first output value output from the first 1x1 convolution layer is input to the Bi-DilConv layer, and A learning method for a packet loss recognition system comprising inputting a second output value output from the above Bi-DilConv layer to the above first FiLM layer.

8. In Paragraph 5, Generating the above second combined value is, The above packet loss embedding is input to the second 1x1 convolution layer, and A learning method for a packet loss recognition system comprising inputting a third output value output from the second 1x1 convolution layer to the second FiLM layer.

9. A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for conditioning to a diffusion model, and The above operations are, Input speech data with Gaussian noise added and noise level embeddings into the residual layer; Generating a first combined value by combining local conditioning information with the voice data and the noise level embedding through a first FiLM (Feature-wise Linear Modulation) layer; A second combined value is generated by combining the packet loss embedding with the first combined value through the second FiLM layer; A computer program comprising: the residual layer outputting estimated noise at each time step through a reverse process.

10. In Paragraph 9, A program further comprising: training the residual layer based on the difference between the estimated noise and the noise to be compared.

11. In Paragraph 9, Generating the above first combination value is, The above local conditioning information is input into the first 1x1 convolution layer, and The first output value output from the first 1x1 convolution layer is input to the Bi-DilConv layer, and A program comprising inputting a second output value output from the above Bi-DilConv layer into the above first FiLM layer.

12. In Paragraph 9, Generating the above second combined value is, The above packet loss embedding is input to the second 1x1 convolution layer, and A program comprising inputting a third output value output from the second 1x1 convolution layer to the second FiLM layer.

Images