Speech recognition system, control method thereof, and training method of speech recognition system

Abstract

The present invention relates to a speech recognition system, a control method thereof, and a training method of the speech recognition system. More specifically, the present invention relates to a training method of a speech recognition system, the method comprising: obtaining a first latent vector by inputting a target speech signal to an ASR encoder; and training an acoustic tokenizer with a cluster vector generated by clustering the first latent vector; and training a noise removal model by using knowledge distillation from the trained tokenizer and an ASR model, and relates to a control method of a speech recognition system, comprising: receiving speech from a user; processing noise with a noise removal model trained on the basis of knowledge distillation; inputting speech generated after the noise processing to an ASR encoder and an ASR decoder to extract a latent vector; and generating a final text sequence by using the extracted latent vector.

Description

Speech recognition system and control method thereof, and learning method of speech recognition system

[0001] The present invention relates to a speech recognition system, a method for controlling the same, and a method for learning the speech recognition system. More specifically, the present invention relates to a method for learning a noise removal model of a speech recognition system.

[0002]

[0003] Recent advancements in artificial intelligence technology have demonstrated consistent achievements not only in image and natural language processing but also in audio and speech signal processing. Speech processing encompasses the comprehensive analysis, synthesis, and recognition of speech, including speaker authentication, speech segmentation, speech enhancement, speech synthesis, and speech recognition.

[0004] Automatic Speech Recognition (ASR) is a technology that converts human speech into text and is utilized in various fields, such as voice assistants, voice command automation, meeting recording, and real-time caption generation.

[0005] However, in real-world environments, speech signals are susceptible to distortion by ambient noise, which affects the performance of speech recognition systems. In particular, since most speech recognition models are trained under the premise of a noise-free environment, their performance can be significantly degraded by background noise or acoustic disturbances in actual settings.

[0006] To address these issues, active research has recently been conducted on noise-robust speech recognition systems that enable accurate speech recognition even in noisy environments. This research includes methods that improve recognition performance in noisy situations by using noise removal (SE) models and speech recognition models together.

[0007] However, simultaneously training (or optimizing) speech recognition and noise reduction models can lead to conflicting objectives, and frame mismatch issues may arise because each model has a different frame structure. In other words, while this approach aims to improve both speech quality and recognition performance, it has limitations such as conflicting gradients between the loss functions of the two models with different purposes, as well as increased training time and cost.

[0008] Accordingly, the present invention proposes a learning method that combines a speech recognition model and a noise removal model based on Knowledge Distillation (KD).

[0009]

[0010] The present invention is intended to provide a speech recognition system capable of maintaining high speech recognition performance in various noisy environments, a method for controlling the same, and a method for learning the speech recognition system.

[0011] More specifically, the present invention aims to provide a noise removal model that can flexibly respond to various noise environments and be universally utilized in various noise environments.

[0012] Furthermore, the present invention aims to provide a method for training a noise removal model that eliminates (or suppresses) unnecessary noise and improves speech recognition performance even in various noisy environments.

[0013] Furthermore, the present invention aims to solve the problem of performance degradation caused by frame mismatch between a speech recognition model and a noise removal model, and to provide a speech recognition system that provides high speech recognition accuracy in various noisy environments, a method for controlling the same, and a method for learning the speech recognition system.

[0014]

[0015] To solve the problem described above, a learning method for a speech recognition system comprising an Acoustic Tokenizer module according to the present invention, the Acoustic Tokenizer module and the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, and the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, and the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, and the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, and the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, and the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, and the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Tokenizer module, the Acoustic Token

[0016] Furthermore, the step of training the noise removal model comprises: generating a voice signal containing noise using the target voice signal and inputting the voice signal containing noise into the noise removal model; obtaining a generated voice signal from the noise removal model that is generated by processing the noise included in the voice signal; inputting the generated voice signal into the encoder; obtaining a second latent vector for the generated voice signal from the encoder; inputting the second latent vector for the generated voice signal into the trained acoustic tokenizer module; obtaining a second acoustic token embedding vector corresponding to the second latent vector from the trained acoustic tokenizer module; and training the noise removal model using at least one of the outputs of the encoder and the trained acoustic tokenizer module, wherein the weights of the encoder and the trained acoustic tokenizer module may exist in a frozen state.

[0017] Furthermore, in the step of learning the acoustic tokenizer module, learning of the acoustic tokenizer module can be performed using the first latent vector and the cluster vector.

[0018] Furthermore, training of the acoustic tokenizer module based on the cluster vector can be performed to generate a first acoustic token embedding vector corresponding to the cluster vector using the first latent vector.

[0019] Furthermore, the step of learning the acoustic tokenizer module includes the step of defining a tokenizer loss function to be used for learning the acoustic tokenizer module using the first latent vector and the cluster vector, and the step of performing learning on the acoustic tokenizer using the tokenizer loss function, wherein the tokenizer loss function can be defined using the probabilistic similarity between the first latent vector and the cluster vector.

[0020] Furthermore, the step of training the noise removal model includes the step of defining at least one loss function to be used for training the noise removal model, and in the step of defining the loss function, the loss function can be defined using the input and output of the speech recognition model and the generated speech signal generated by the noise removal model.

[0021] Furthermore, the loss function may include a first loss function defined using the probabilistic similarity between the cluster vector and the second acoustic token embedding vector, a second loss function defined by the difference between the target speech signal and the generated speech signal, and a third loss function defined by the difference between the first latent vector and the second latent vector.

[0022] Furthermore, in the step of defining the loss function, different weights are multiplied to each of the first loss function, the second loss function, and the third loss function, and a final loss function is defined by combining the first loss function, the second loss function, and the third loss function multiplied by the different weights, and in the step of training the noise removal model, training of the noise removal model can be performed using the final loss function.

[0023] Meanwhile, a speech recognition method of a speech recognition system according to the present invention may include: receiving a speech signal to be converted from a user terminal; processing the speech signal as an input to a noise removal model (SE) learned using knowledge distillation of a speech recognition model; obtaining a generated speech signal from the learned noise removal model by processing the noise included in the speech signal; processing the generated speech signal as an input to an encoder of a speech recognition model (ASR); obtaining a latent vector for the generated speech signal as an output of the encoder; processing the latent vector as an input to a decoder; and obtaining a text sequence corresponding to the speech signal using the latent vector in the decoder.

[0024] Meanwhile, a speech recognition system comprising an Acoustic Tokenizer module according to the present invention, a speech recognition model (ASR) composed of a Teacher Model, and a noise removal model (SE) composed of a Student Model, can input a target speech signal corresponding to correct answer data into an encoder of the speech recognition model, obtain a first latent vector for the target speech signal from the encoder, perform clustering on the first latent vector to generate a cluster vector, train the acoustic tokenizer module using the cluster vector, and train the noise removal model using the learned acoustic tokenizer module and knowledge distillation of the speech recognition model.

[0025] Meanwhile, the program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored on a computer-readable recording medium, and may include instructions for performing the steps of: inputting a target voice signal corresponding to correct answer data into an encoder of a voice recognition model; obtaining a first latent vector for the target voice signal from the encoder; performing clustering on the first latent vector to generate a cluster vector; learning an acoustic tokenizer module using the cluster vector; and learning a noise removal model using the learned acoustic tokenizer module and knowledge distillation of the voice recognition model.

[0026]

[0027] As described above, according to the speech recognition system and control method thereof and the learning method of the speech recognition system of the present invention, by utilizing a knowledge distillation-based learning method to transfer linguistic features of the speech recognition (ASR) model to the noise removal (SE) model, it is possible to enable speech recognition with high accuracy even in various noisy environments.

[0028] In particular, according to the speech recognition system and the control method thereof and the learning method of the speech recognition system of the present invention, by allowing the noise removal model to be trained to meet the requirements of the speech recognition model through a knowledge distillation-based learning method, the noise removal model can effectively remove noise while minimizing distortion of language information, thereby improving overall speech recognition performance.

[0029] Furthermore, according to the speech recognition system, the control method thereof, and the learning method of the speech recognition system of the present invention, a speech recognition model and a noise removal model having different learning objectives can be effectively combined using an acoustic tokenizer module. Through this, the present invention reduces the complexity of the learning process and reduces the time and cost required for learning, thereby enabling high performance to be maintained with minimal resources.

[0030] In this way, the present invention performs knowledge distillation-based learning using an acoustic tokenizer module, thereby significantly reducing computational costs and time consumption during the learning process, enabling fast and efficient learning of large-scale datasets, and mitigating frame mismatch issues between the speech recognition model and the noise removal model, thereby simultaneously improving speech quality and speech recognition accuracy.

[0031] Furthermore, according to the speech recognition system, the control method thereof, and the learning method of the speech recognition system of the present invention, high speech recognition accuracy can be provided even in various noisy environments, so it can be universally utilized in various fields such as industrial sites, traffic control systems, and medical fields.

[0032] In other words, the present invention can effectively improve speech recognition performance in noisy environments by optimizing mutual learning between a speech recognition model and a noise removal model through the application of a knowledge distillation-based learning method.

[0033]

[0034] FIGS. 1 and FIGS. 2 are conceptual diagrams for explaining a voice recognition system according to the present invention.

[0035] FIG. 3 is a flowchart illustrating a learning method of a speech recognition system according to the present invention.

[0036] FIGS. 4 and FIGS. 5 are conceptual diagrams for explaining a learning method of a speech recognition system according to the present invention.

[0037] Figure 6 is a set of formulas for explaining an algorithm related to the learning of a noise removal model according to the present invention.

[0038] FIGS. 7 and 8 are graphs and tables showing an example of the learning results of a model learned using the knowledge distillation-based learning method according to the present invention.

[0039] FIG. 9 is a flowchart illustrating a speech recognition method of a speech recognition system according to the present invention.

[0040]

[0041] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not have distinct meanings or roles in themselves. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0042] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0043] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0044] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0045] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0046] The present invention relates to a speech recognition system, a method for controlling the same, and a method for learning the speech recognition system. The speech recognition system according to the present invention may be a speech recognition system capable of maintaining high speech recognition performance in various noisy environments. Furthermore, the speech recognition system according to the present invention may be a system that solves the problem of performance degradation caused by frame mismatch between an Automatic Speech Recognition (ARS) model and a Speech Enhancement (SE) model, and provides high speech recognition accuracy in various noisy environments.

[0047] The speech recognition system according to the present invention includes a noise removal model, and the present invention aims to provide a noise removal model that can flexibly respond to various noise environments and be universally utilized in various noise environments.

[0048] Hereinafter, the present invention will be examined in more detail with reference to the attached drawings. FIGS. 1 and 2 are conceptual diagrams for explaining a speech recognition system according to the present invention. FIG. 3 is a flowchart for explaining a learning method of a speech recognition system according to the present invention, FIGS. 4 and 5 are conceptual diagrams for explaining a learning method of a speech recognition system according to the present invention, and FIG. 6 is a set of formulas for explaining an algorithm related to the learning of a noise removal model according to the present invention. In addition, FIGS. 7 and 8 are graphs and tables showing an example of the learning results of a model learned using a knowledge distillation-based learning method according to the present invention. Furthermore, FIG. 9 is a flowchart for explaining a speech recognition method of a speech recognition system according to the present invention.

[0049] Referring to FIG. 1(a), a pipeline of a conventional speech recognition system consisting of a noise removal model (SE) and an autospeech recognition model (ASR) can be observed. As illustrated in FIG. 1(a), the conventional learning method suffers from voice quality loss ( ) and ASR( After combining the losses, the noise reduction model and the speech recognition model are trained together (or synchronously or asynchronously). However, training (or optimizing) the speech recognition model and the noise reduction model simultaneously can lead to conflicting objectives between the two models, and frame mismatch issues may occur because each model has a frame structure configured differently. For example, the speech recognition model primarily predicts language information (text) based on long frames, while the noise reduction model performs the noise reduction process using short frames. This difference can arise because the frame lengths and frame time windows set for the two models differ to suit their respective optimal objectives.

[0050] In other words, conventional learning methods aimed to simultaneously improve speech quality and recognition performance, but they have limitations such as conflicting gradients between the loss functions of two models with different objectives and increased training time and cost.

[0051] On the other hand, as illustrated in Fig. 1(b), the present invention enables a noise removal model (SE) corresponding to a Student Model (or equivalent) to receive knowledge generated (or extracted) from an ASR model corresponding to a Teacher Model based on Knowledge Distillation (KD) and perform learning.

[0052] In order to account for the different goals of a noise removal model and a speech recognition model (e.g., the noise removal model predicts clean speech, and the speech recognition model predicts text), the present invention first performs training on the speech recognition model using a speech training dataset containing noise, and then ensures that the weights of the speech recognition model (or the ASR encoder and ASR decoder of the speech recognition model) are frozen.

[0053] When the weights of the speech recognition model are fixed, the present invention can construct a teacher model by connecting the encoder (ASR encoder) and the acoustic tokenizer module of the speech recognition model. In the present invention, by configuring the speech recognition model as a teacher model, language-related information possessed by the speech recognition model can be transmitted (or sent) to the noise reduction model. That is, in the present invention, the speech recognition model, acting as a teacher model, can transmit important language information and speech features to the noise reduction model.

[0054] Here, the acoustic tokenizer module can be understood as an intermediate module for connecting linguistic information extracted from a speech recognition model corresponding to a teacher model with a noise reduction model corresponding to a student model. Since the output of the encoder of the speech recognition model corresponds to an acoustic token for the input frame, the acoustic tokenizer module can serve as a substitute model for a decoder (ASR decoder) that extracts linguistic information on a frame-by-frame basis. That is, in the present invention, the acoustic tokenizer module is responsible for the adaptation between the speech domain and the language domain, and can perform the role of transmitting important features extracted from the speech signal (or speech data) to the noise reduction model. In this case, the acoustic tokenizer module is a new module that does not exist in conventional speech recognition models and can be understood as a module additionally configured for model training in the knowledge distillation-based learning process according to the present invention.

[0055] Furthermore, in the present invention, the noise reduction model can be configured as a Student Model. As discussed above, the noise reduction model can learn the linguistic information and speech features possessed by the speech recognition model. Through this, the noise reduction model can be trained to exhibit robust performance even in noisy environments.

[0056] That is, as illustrated in Fig. 1(b), in the present invention, a speech recognition model is trained first, and then the language information and speech features extracted through the training of the speech recognition model can be utilized in the training of the noise removal model.

[0057] In this regard, as illustrated in FIG. 2, the voice recognition system (1000) according to the present invention may include at least one of an input unit (100), a storage unit (200), an acoustic tokenizer module (300), a noise removal model (400), and a voice recognition model (500).

[0058] Although not illustrated, the voice recognition system (1000) may include one or more processors, and such processors may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), neural network processing units (NPUs), application integrated circuits, application semiconductors (ASICs), etc.). One or more processors may be configured to execute instructions, computer-readable instructions, and / or other instructions described herein that are stored (or included) in the storage unit (200). The voice recognition system and method according to the present invention may perform data processing described below in cooperation with memory and at least one processor. The processor may perform a series of operations and data processing using data and information stored in memory. In this case, memory may be a component of the storage unit (200).

[0059] Meanwhile, the input unit (100) can be configured in various ways as a means of data input. For example, the input unit (100) can be configured to receive user input. The input unit (100) can be configured to receive user input from a user terminal. Here, “receiving input” may mean receiving an input signal (or selection signal) corresponding to the user’s input based on input being made by the user through the configuration of the input unit provided in the user terminal. In the present invention, the input unit (100) does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user.

[0060] The input unit (100) may also be referred to as a user interface module. The input unit (100) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of input unit (100).

[0061] Here, user input may include documents, text, images (or videos), voice, etc. In this case, the voice recognition system (1000) may further include a module that converts voice into text.

[0062] Next, the storage unit (200) serves to store various data related to the present invention and may include one or more non-transient computer-readable storage media that can be read and / or accessed by at least one of one or more processors.

[0063] One or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage unit (200) may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage device), whereas in other examples, the storage unit (200) may be implemented using two or more physical devices.

[0064] The storage unit (200) may include computer-readable instructions and additional data. The storage unit (200) may include a storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functions of the device and network.

[0065] Furthermore, at least a portion of the storage unit (200) may be a cloud storage or a cloud server. At least a portion of the data corresponding to user input and training data received from the input unit (100) may be stored in the storage unit (200).

[0066] That is, the storage unit (200) is sufficient as a space where information necessary for the operation of the voice recognition system (1000) according to the present invention is stored, and it can be understood that there are no restrictions on the physical space.

[0067] Next, the acoustic tokenizer module (300) may be configured to serve the role of connecting the noise removal model (400) and the speech recognition model (500). The acoustic tokenizer module (300) may be understood as a channel for connecting the language-related information extracted from the speech recognition model (500) and the noise removal model (400).

[0068] Since the output of the encoder (510) of the speech recognition model (500) corresponds to an acoustic token for the input speech signal (or speech data), the acoustic tokenizer module (300) can be used (or utilized) as an alternative configuration of the decoder (520) that extracts language information on a frame-by-frame basis.

[0069] That is, the acoustic tokenizer module (300) is responsible for the adaptation between the speech domain and the language domain, and can transmit important features extracted from the speech signal to the noise removal model (400). More specific details regarding the acoustic tokenizer module (300) will be described later.

[0070] Next, the noise removal model (400) may be configured to function as a Student Model, and when a voice signal containing noise is input, to process (or remove) the noise contained in the voice signal to generate a voice signal with the noise processed. For example, when the noise removal model (400) receives a voice signal containing noise, it may convert the received voice signal into a voice signal with the noise removed.

[0071] In this regard, in the learning process according to the present invention, the noise removal model (400) can receive knowledge from the speech recognition model (500) and perform learning to convert a voice signal containing noise into a voice signal from which noise has been removed. More specifically, the noise removal model (400) can receive language-related information extracted as a result of learning from the speech recognition model (500) as input and learn the language information and voice features possessed by the speech recognition model (500).

[0072] In one embodiment, the noise removal model (400) according to the present invention processes speech signals using deep learning technology and can learn language information and speech features based on a DCCRN network (Deep Complex Convolutional Recurrent Network) structure. However, in addition to the artificial intelligence model (or network) mentioned above, various models may be applied (or utilized) in the noise removal model (400) according to the present invention.

[0073] Next, the speech recognition model (500) is a teacher model, and the speech signal ( Generate a latent vector (i.e., a vector containing language information) for , 10), and use the latent vector to generate the final text (or text sequence, for the speech signal, It can be configured to perform the role of predicting (or outputting, generating, etc.) , 20). The text sequence (20) is in the form of multiple words arranged in chronological order according to the flow of speech, and may be in the form of a sequence containing words or sentences rather than a single text.

[0074] The speech recognition model (500) may be configured to include an encoder (510) and a decoder (520). The encoder (510) may perform the role of extracting a latent vector for a speech signal (10). Additionally, the decoder (520) may generate (or extract) a text sequence (20) for a speech signal (10) using the latent vector extracted from the encoder (510).

[0075] In this regard, in the learning process according to the present invention, the voice recognition system (1000) can train a voice recognition model (500) using a learning data set including at least one voice signal stored in a storage unit (200).

[0076] And, when training for the speech recognition model (500) is completed, the speech recognition system (1000) can fix the weights of the encoder (510) and decoder (520) of the speech recognition model (500) and connect the encoder (510) of the speech recognition model (500) with the acoustic tokenizer module (300) to form a teacher model.

[0077] Furthermore, the speech recognition system (1000) can transmit language information and speech features extracted as a result of learning from the speech recognition model (500) to the noise removal model (400) so that the noise removal model (400) learns language-related information extracted from the speech recognition model (500). In this way, the noise removal model (400) according to the present invention can be learned through knowledge distillation of the speech recognition model (500).

[0078] In one embodiment, the speech recognition model (500) according to the present invention can process speech signals based on a conformer-transducer (or encoder-decoder) structure. However, in addition to the artificial intelligence model (or network) mentioned above, various models may be applied (or utilized) in the speech recognition model (500) according to the present invention.

[0079] Meanwhile, the present invention is intended to provide a noise removal model that can flexibly respond to various noise environments and be universally utilized in various noise environments. More specifically, the present invention is intended to provide a training method for a noise removal model that can remove (or suppress) unnecessary noise and improve speech recognition performance even in various noise environments. Below, we will examine the training method of the noise removal model (400) in more detail.

[0080] First, in the present invention, a target voice signal corresponding to the correct answer data is input into an encoder of a voice recognition model (S310, see FIG. 3), and a process of obtaining a first latent vector for the target voice signal from the encoder may be performed (S302, see FIG. 3).

[0081] As illustrated in FIGS. 4 and 5, the voice recognition system (1000) has a target voice signal ( , 410) can be input into the encoder (510) of the speech recognition model (500) composed of the teacher model. As seen above, the weights of the encoder (510) and decoder (520) of the speech recognition model (500) composed of the teacher model are fixed, so it can be understood that no learning is taking place.

[0082] Here, the target speech signal (410) refers to a noise-removed speech signal (Clean Speech) and can be utilized (or used) to extract language-related information (or language information features) in a speech recognition model (500).

[0083] For example, a voice recognition system (1000) may sample a target voice signal (410) at 16 kHz and divide it into consecutive frames with an overlap of 16 ms at intervals of 25 ms.

[0084] And, the speech recognition system (1000) can input the divided frames into the encoder (510) of the speech recognition model (500). The encoder (510) has a first latent vector (Latent Vector, for the target speech signal (410). ..., 411) can be generated. For example, the encoder (510) can extract speech features for each frame of the target speech signal (410) to extract a first latent vector (411) for the target speech signal (410). In this case, the first latent vector (411) may have a value of a specific dimension (e.g., 144 dimensions), and subsampling may be applied to the encoder (510) to increase processing speed.

[0085] Next, in the present invention, clustering is performed on the first potential vector to generate a cluster vector (S330, see FIG. 3), and a process of learning an acoustic tokenizer module using the cluster vector may be carried out (S340, see FIG. 3).

[0086] The speech recognition system (1000) can perform clustering on the first latent vector (411). In this case, the present invention can perform clustering on the latent vector using various clustering techniques such as K-means clustering, hierarchical clustering, and K-medoids clustering. However, it is obvious that various clustering techniques other than those described above can be utilized.

[0087] In the present invention, the first latent vector (411) is converted (or encoded) into a cluster vector (or one-hot cluster vector), which may represent a specific cluster to which the frame belongs. Since the speech recognition model (500) according to the present invention is trained to include a large number of language units generated through a specific algorithm (e.g., a Unigram Language Model algorithm), the number of clusters may be set to 1.5k. These units may include not only language information but also non-verbal noise (e.g., short pauses, breathing sounds, coughing sounds, etc.).

[0088] Here, a cluster vector is, for example, when multiple latent vectors exist, K-means clustering is applied to cluster vectors with similar features into a specific group. At this time, a unique number (cluster label) is assigned to each group through clustering, and such a label can be called a “cluster vector.” The cluster vector serves as a “label” indicating which specific group (or cluster) each latent vector belongs to, and can provide reference information necessary for the acoustic tokenizer module (300) to predict (or learn) which specific group a latent vector belongs to.

[0089] That is, the cluster vector acts as a reference label and can provide structural information during the process in which the acoustic tokenizer module (300) converts the latent vector into an acoustic token vector (or acoustic token embedding vector).

[0090] The speech recognition system (1000) applies K-means clustering to the first latent vector (411) output from the encoder (510) to train the acoustic tokenizer module (300) to obtain a cluster vector ( , 412, or any label or pseudo-label can be generated. For example, through K-means clustering, the latent vector corresponding to the silent frame in the first latent vector (411) can be removed, and clustering can be performed on the remaining speech utterances. Here, a silent frame may mean a frame representing a state in which there is almost no speech (or audio) sound (e.g., only background noise without human voice or music, or a completely silent state).

[0091] Furthermore, the voice recognition system (1000) can perform learning on the acoustic tokenizer module (300) using the cluster vector (412).

[0092] More specifically, the speech recognition system (1000) can train the acoustic tokenizer module (300) using the first latent vector (411) and the cluster vector (412). The cluster vector (412) describes an important pattern of the speech signal (410), and the acoustic tokenizer module (300) can be trained to perform learning based on the cluster vector (412) and to perform predictions thereon. That is, the speech recognition system (1000) can train the acoustic tokenizer module (300) so that the acoustic tokenizer module (300) learns the cluster vector (412) and predicts the probability (or probability value) that the corresponding latent vector belongs to a specific cluster.

[0093] Accordingly, the acoustic tokenizer module (300) uses the first potential vector (411) to obtain a first acoustic token embedding vector (412) corresponding to the cluster vector (412). Learning can be performed based on the cluster vector (412) to generate (or extract) an Acoustic Token Embedding Vector (413).

[0094] In one embodiment, the acoustic tokenizer module (300) can generate a first acoustic token embedding vector (413) by converting the first potential vector (411) into a probability of belonging to each cluster based on the cluster vector (412). Then, the speech recognition system (1000) can obtain the first acoustic token embedding vector (413) generated from the acoustic tokenizer module (300).

[0095] Finally, as seen above, the cluster vector (412) is an acoustic token embedding vector (or logit vector ( It is tokenized into ), and this is “ It can be expressed as follows. The acoustic tokenizer module (300) can be configured using a time-distributed layer, and can represent the number of clusters. To train the acoustic tokenizer module (300), the speech recognition system (1000) uses a logit vector ( , or acoustic token embedding vector) and cluster vector( A loss function can be defined using ). This loss function is a “tokenizer loss function ( It can be named as )” and can be expressed as [Mathematical Formula 1] below.

[0096] [Mathematical Formula 1]

[0097]

[0098] In the above [Mathematical Formula 1], Is of As the nth element, This applies when, (=0.5) may represent a temperature parameter (or parameter). This loss function is defined as a method of calculating the probabilistic similarity between a logit vector and a cluster vector, and can be calculated in a manner similar to a softmax function. For example, a speech recognition system (1000) converts a first acoustic token embedding vector (413) into a probability distribution to measure the match with a cluster vector (412), and the temperature parameter ( The smoothing of the probability distribution can be adjusted (or controlled) through ).

[0099] That is, in the present invention, the degree to which an acoustic token embedding vector generated from an acoustic tokenizer module (300) matches a specific cluster vector is calculated probabilistically, and the output of the acoustic tokenizer module (300) ) and cluster vector( Learning of the acoustic tokenizer module (300) can be performed in a direction that minimizes the difference between ).

[0100] In this way, the learned acoustic tokenizer module (300) can determine which cluster a new potential vector belongs to when a new potential vector is input. The acoustic token embedding vector generated from the acoustic tokenizer module (300) contains information about the cluster predicted for each potential vector (i.e., the probability of which group each potential vector corresponds well to), and can be utilized (or used) as an important vector that reflects the characteristics of the voice data.

[0101] Meanwhile, in the present invention, a process of training a noise removal model using knowledge distillation of a speech recognition model may be performed (S350, see FIG. 3).

[0102] As illustrated in FIGS. 4 and 5, a speech recognition system (1000) first, in order to train a noise removal model (400) composed of a student model based on knowledge distillation using information about a speech recognition model (500, or an encoder of a speech recognition model), ( Using , 410), a voice signal containing noise (Noisy Speech, It can generate , 420).

[0103] Here, can mean noise, and a voice signal containing noise ( , 420) is the target voice signal ( ) and noise( It can be generated by combining voices containing ).

[0104] A speech recognition system (1000) can process a voice signal (420) containing noise as input to a noise removal model (400). The noise removal model (400) can process the noise contained in the voice signal (420) containing noise and generate a predicted (or estimated) voice signal (hereinafter, generated voice signal) as a result of the noise processing. The speech recognition system (1000) can, from the noise removal model (400), the noise ( Generated speech signal (Enhanced Speech, generated by processing ) You can obtain , 430).

[0105] And, the voice recognition system (1000) obtains a generated voice signal (from a noise removal model (400) , 430) can be input into the encoder (510) of the speech recognition model (500). The encoder (510) has a latent vector ( , 431, or a second latent vector can be generated.

[0106] Furthermore, the voice recognition system (1000) has a second potential vector for the generated voice signal (430). , 431) can be input into the learned acoustic tokenizer module (300). The learned acoustic tokenizer module (300) can generate an acoustic token embedding vector corresponding to the latent vector.

[0107] More specifically, the acoustic tokenizer module (300), which is trained based on the cluster vector (412) examined above, when the second latent vector (431) among the multiple latent vectors is input, uses the first latent vector (411) to produce a second acoustic token embedding vector ( , 433) can be generated. In this case, the weights of the encoder (510) and the learned acoustic tokenizer module (300) may remain in a fixed state.

[0108] Meanwhile, in the present invention, at least one loss function to be used for training a noise removal model (400) can be defined.

[0109] Specifically, the speech recognition system (1000) can define a loss function using the input and output of the speech recognition model (500) and the generated speech signal (430) generated from the noise removal model (400).

[0110] As illustrated in FIG. 5(b), the input and output of the speech recognition model (500) include a target speech signal (e.g., a noise-free speech signal) corresponding to the correct answer data. ), the above target voice signal ( Information extracted (or output) from a speech recognition model (500, or encoder (510)) for ) (e.g., a first latent vector ( ), cluster vector( ), first acoustic token embedding vector( It may include )). In this case, the first acoustic token embedding vector (413) may also be understood as being extracted from an acoustic tokenizer module (300) connected to a speech recognition model (500).

[0111] In addition, the input and output of the speech recognition model (500) include a generated speech signal ( generated from the noise removal model (400) , 430), information output from a speech recognition model (500, or encoder (510)) for the generated speech signal (430) (e.g., a second latent vector ( ), second acoustic token embedding vector( It may include )). In this case, the second acoustic token embedding vector (433) may also be understood as being extracted from an acoustic tokenizer module (300) connected to a speech recognition model (500).

[0112] In this regard, the voice recognition system (1000) can first define a loss (or tokenizer loss) for the acoustic tokenizer module (300). Tokenizer loss ( ) is calculated according to the cluster vector (412) and may also be referred to as the “first loss function” in the present invention. This second loss function can be expressed as shown in the mathematical formula in FIG. 6 (a) or as in [Equation 2] below.

[0113] [Mathematical Formula 2]

[0114]

[0115] As such, the first loss function is the cluster vector ( , 412) and the second acoustic token embedding vector( It can be defined using the probabilistic similarity between , 433). Here, Is of As the nth element, This applies when [Equation 1] is examined above and can be calculated in the same way. That is, the first loss function can be used (or utilized) to enable the noise removal model (400) to learn the knowledge of the speech recognition model (500) and to train it to show higher performance even in a noisy environment.

[0116] Next, the speech recognition system (1000) may define two different loss functions to improve the sound quality and the performance of the encoder (510) of the speech recognition model (500) (e.g., voice quality improvement). Sound quality loss (or sound quality signal-to-noise ratio loss) target speech signal ( , 410) and generated speech signal( It is defined as a negative (SNR(NSNR)) loss calculated by comparing the sound quality between , 430), and in the present invention, it may also be named a “second loss function.” This second loss function can be expressed as shown in the mathematical formula in FIG. 6 (b) or as in [Equation 3] below.

[0117] [Mathematical Formula 3]

[0118]

[0119] The second loss function is the target speech signal ( , 410) and generated speech signal( Sound quality is compared through the difference between , 430). The greater the difference between the two signals, the higher the loss value, which may imply lower sound quality. The second loss function is for each frame ( ) and the corresponding The difference between them can be calculated on a logarithmic scale to measure the loss in sound quality. That is, the second loss function can be used to train the noise removal model (400) to remove noise and generate (or estimate) a more accurate (clean) speech.

[0120] Next, the voice recognition system (1000) can first define a loss for the encoder (510). In the present invention, the encoder (510) loss ( ) may also be named a “third loss function,” and this third loss function can be expressed as shown in (c) of FIG. 6 or as in [Equation 4] below.

[0121] [Mathematical Formula 4]

[0122]

[0123] Referring to [Equation 4] above, the third loss function is a first latent vector ( for each of the target speech signal (410) and the generated speech signal (430) ,411) and the second potential vector ( , 431) between sequences It is defined as a norm, which may be defined as the difference between the first potential vector (411) and the second potential vector (431). Through the third loss function, the noise removal model (400) [is] the first potential vector ( , 411) and the second potential vector ( It can be trained to minimize the difference between , 431). That is, the third loss function can be used to train the noise removal model (400) to recover speech signals more robustly in a noisy environment.

[0124] Furthermore, the speech recognition system (1000) can define a final loss function (or joint loss function) to be used for training the noise removal model (400). The speech recognition system (1000) can define a final loss function by multiplying each of the first loss function, second loss function, and third loss function described above by different weights, and by combining the first loss function, second loss function, and third loss function multiplied by different weights. In the present invention, the final loss function can be expressed as shown in the mathematical formula (d) of FIG. 6.

[0125] As shown in Fig. 6(d), in the final loss function , , Each represents the weights of the first loss function, the second loss function, and the third loss function. In one embodiment, the speech recognition system (1000) has weights , , Instead of performing an inclusive search to determine the value of, a grid search can be utilized. First, different weights (first weight ( ), second weight ( ), third weight ( Two of the weights are fixed at 1.0, and the remaining weight is adjusted from 0.1 to 1.0 in increments of 0.1 to measure the Word Error Rate (WER) for each combination. The combination with the lowest WER is identified; since two weights in the identified combination are fixed at 1.0, a grid search is performed again to identify one of the two weights that have not yet been determined. Subsequently, the weight with the lowest WER is selected, and the remaining weight is left undetermined. Finally, the two previously determined weights are fixed, and a grid search is performed while adjusting the remaining weight from 0.1 to 1.0. As a result, different weights = 0.3, = 0.7, It can be set as = 1.0. The speech recognition system (1000) can train a noise removal model (400) using a weighted final loss function through this process.

[0126] In this regard, referring to FIGS. 7 and FIGS. 8, an example of the learning result of a model learned using the knowledge distillation-based learning method according to the present invention can be observed.

[0127] First, looking at the graph shown in Fig. 7, it can be intuitively observed that the conflicting problem is effectively resolved when examining the learning loss curve of a model trained using a final loss function that combines the first, second, and third loss functions in the present invention. It can be confirmed that the learning method according to the present invention (e.g., Proposed) converges faster than conventional learning methods (e.g., Joint-Straight, Joint-ASO, Joint-Grad, etc.) and has a shorter processing time through the efficient combination of loss functions. In other words, the learning method of the present invention provides an efficient and fast convergence speed, demonstrating its advantage in reducing the training time of the model.

[0128] Next, as shown in FIG. 8(a), a model trained using the knowledge distillation-based learning method of the present invention described above (e.g., Joint-KD) records the lowest WER compared to models trained using conventional learning methods and shows excellent performance even in noisy environments.

[0129] In addition, as shown in Fig. 8(b), a model trained using the knowledge distillation-based learning method according to the present invention (e.g., Joint-KD) records more accurate and higher performance than other models in other speech recognition network structures (e.g., Attention-based Encoder-Decoder (AED) and Connectionist Temporal Classification (CTC), etc.) and demonstrates that it can operate efficiently even in real noisy environments.

[0130] Furthermore, as illustrated in FIG. 8(c), a model (e.g., Joint-KD) trained using the knowledge distillation-based learning method according to the present invention was applied to other network structures (e.g., HuBERT and Whisper, etc.) and demonstrated superior performance compared to conventional models. In particular, it showed excellent performance even in noisy environments and demonstrated high recognition accuracy in real-world environments. These learning results demonstrate that knowledge distillation-based learning can significantly improve speech recognition and noise reduction performance.

[0131] Meanwhile, referring again to FIG. 1(b), in the inference process of the speech recognition method according to the present invention, the noise removal model and the speech recognition model may be connected to each other to perform the speech recognition process, excluding the acoustic tokenizer module. In this case, a speech signal without noise is processed using only the speech recognition model, and a speech signal with noise can be processed using both the noise removal model and the speech recognition model.

[0132] According to the learning method described above, the noise removal model and the speech recognition model cooperate with each other to generate (or predict) text for the voice signal input by the user through the following steps in the inference process: receiving a voice signal to be converted from a user terminal (S910); processing the voice signal as input to the noise removal model (SE) learned using the knowledge distillation of the speech recognition model (S920); obtaining a generated voice signal created by processing noise included in the voice signal from the learned noise removal model (S930); processing the generated voice signal as input to the encoder of the speech recognition model (ASR) (S940); obtaining a latent vector for the generated voice signal as output of the encoder (S950); processing the latent vector as input to the decoder (S960); and obtaining a text sequence corresponding to the voice signal using the latent vector in the decoder (S970).

[0133] As described above, according to the speech recognition system and control method thereof and the learning method of the speech recognition system of the present invention, by utilizing a knowledge distillation-based learning method to transfer linguistic features of the speech recognition (ASR) model to the noise removal (SE) model, it is possible to enable speech recognition with high accuracy even in various noisy environments.

[0134] In particular, according to the speech recognition system and the control method thereof and the learning method of the speech recognition system of the present invention, by allowing the noise removal model to be trained to meet the requirements of the speech recognition model through a knowledge distillation-based learning method, the noise removal model can effectively remove noise while minimizing distortion of language information, thereby improving overall speech recognition performance.

[0135] Furthermore, according to the speech recognition system, the control method thereof, and the learning method of the speech recognition system of the present invention, a speech recognition model and a noise removal model having different learning objectives can be effectively combined using an acoustic tokenizer module. Through this, the present invention reduces the complexity of the learning process and reduces the time and cost required for learning, thereby enabling high performance to be maintained with minimal resources.

[0136] In this way, the present invention performs knowledge distillation-based learning using an acoustic tokenizer module, thereby significantly reducing computational costs and time consumption during the learning process, enabling fast and efficient learning of large-scale datasets, and mitigating frame mismatch issues between the speech recognition model and the noise removal model, thereby simultaneously improving speech quality and speech recognition accuracy.

[0137] Furthermore, according to the speech recognition system, the control method thereof, and the learning method of the speech recognition system of the present invention, high speech recognition accuracy can be provided even in various noisy environments, so it can be universally utilized in various fields such as industrial sites, traffic control systems, and medical fields.

[0138] In other words, the present invention can effectively improve speech recognition performance in noisy environments by optimizing mutual learning between a speech recognition model and a noise removal model through the application of a knowledge distillation-based learning method.

[0139] Meanwhile, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a computer-readable medium (or recording medium).

[0140] Furthermore, the present invention described above can be implemented as computer-readable code or instructions on a medium on which a program is recorded. That is, the present invention can be provided in the form of a program.

[0141] Meanwhile, computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0142] Furthermore, the computer-readable medium may be a server or cloud storage that includes a storage and is accessible to an electronic device via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage via wired or wireless communication.

[0143] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, namely a CPU (Central Processing Unit), and no special limitations are placed on its type.

[0144] Meanwhile, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

1. A learning method for a speech recognition system comprising an ASR model composed of a Teacher Model and a SE model composed of a Student Model by connecting an Acoustic Tokenizer module, A step of inputting a target voice signal corresponding to the correct answer data into the encoder of the voice recognition model; A step of obtaining a first latent vector for the target voice signal from the encoder; A step of generating a cluster vector by performing clustering on the first potential vector; A step of learning the acoustic tokenizer module using the cluster vector; and A method for learning a speech recognition system characterized by including the step of learning the noise removal model using the learned acoustic tokenizer module and the knowledge distillation of the speech recognition model.

2. In Paragraph 1, The step of training the above noise removal model is, A step of generating a voice signal containing noise using the above target voice signal, and inputting the voice signal containing noise into the noise removal model; A step of obtaining a generated speech signal by processing the noise included in the speech signal from the above noise removal model; A step of inputting the generated voice signal into the encoder; A step of obtaining a second latent vector for the generated voice signal from the encoder; A step of inputting the second latent vector for the generated voice signal into the learned acoustic tokenizer module; A step of obtaining a second acoustic token embedding vector corresponding to the second latent vector from the above-mentioned learned acoustic tokenizer module; and The method includes the step of learning the noise removal model using at least one of the outputs of the encoder and the learned acoustic tokenizer module. A learning method for a speech recognition system characterized in that the weights of the encoder and the learned acoustic tokenizer module exist in a frozen state.

3. In Paragraph 1, In the step of learning the above acoustic tokenizer module, A learning method for a speech recognition system characterized by performing learning on the acoustic tokenizer module using the first latent vector and the cluster vector.

4. In Paragraph 3, In the step of learning the above acoustic tokenizer module, A learning method for a speech recognition system characterized by performing learning on the acoustic tokenizer module based on the cluster vector to generate a first acoustic token embedding vector corresponding to the cluster vector using the first latent vector.

5. In Paragraph 4, The step of learning the above acoustic tokenizer module is, A step of defining a tokenizer loss function to be used for learning the acoustic tokenizer module using the first potential vector and the cluster vector; and The method includes the step of performing learning on the acoustic tokenizer using the above tokenizer loss function, The above tokenizer loss function is, A learning method for a speech recognition system characterized by being defined using the probabilistic similarity between the first latent vector and the cluster vector.

6. In Paragraph 2, The step of training the above noise removal model is, The method includes the step of defining at least one loss function to be used for training the noise removal model, and In the step of defining the above loss function, A learning method for a speech recognition system characterized by defining the loss function using the input and output of the speech recognition model and the generated speech signal generated by the noise removal model.

7. In Paragraph 4, The above loss function is, A first loss function defined using the probabilistic similarity between the cluster vector and the second acoustic token embedding vector. A second loss function defined as the difference between the target voice signal and the generated voice signal, and A learning method for a speech recognition system characterized by including a third loss function defined as the difference between the first latent vector and the second latent vector.

8. In Paragraph 5, In the step of defining the above loss function, Multiplying each of the first loss function, the second loss function, and the third loss function by different weights, A final loss function is defined by combining the first loss function, the second loss function, and the third loss function, which are multiplied by the above different weights. In the step of training the above noise removal model, A learning method for a speech recognition system characterized by performing learning on the noise removal model using the above final loss function.

9. A method for controlling a speech recognition system robust in a noisy environment, A step of receiving a voice signal to be converted from a user terminal; A step of processing the above voice signal as input to a noise removal model (SE) learned using knowledge distillation of a voice recognition model; A step of obtaining a generated speech signal by processing the noise included in the speech signal from the above-mentioned learned noise removal model; A step of processing the above-mentioned generated voice signal as the input to an encoder of a speech recognition model (ASR); A step of obtaining a latent vector for the generated voice signal as the output of the encoder; A step of processing the above latent vector as input to a decoder; and A control method for a speech recognition system characterized by including the step of obtaining a text sequence corresponding to the speech signal using the latent vector in the above decoder.

10. A speech recognition system comprising an Acoustic Tokenizer module connected to a speech recognition model (ASR) composed of a Teacher Model and a noise removal model (SE) composed of a Student Model, The above voice recognition system is, A target voice signal corresponding to the correct answer data is input into the encoder of the above-mentioned voice recognition model, and A first latent vector for the target voice signal is obtained from the encoder above, and Clustering (K-means Clustering) is performed on the above first latent vector to generate a cluster vector, and The acoustic tokenizer module is trained using the cluster vector above, and A speech recognition system characterized by training the noise removal model using the above-mentioned learned acoustic tokenizer module and the knowledge distillation of the above-mentioned speech recognition model.

11. A program that is executed by one or more processes in an electronic device and stored on a computer-readable recording medium, The above program is, A step of inputting a target voice signal corresponding to the correct answer data into the encoder of a speech recognition model; A step of obtaining a first latent vector for the target voice signal from the encoder; A step of generating a cluster vector by performing clustering (K-means Clustering) on ​​the first potential vector; A step of training an acoustic tokenizer module using the above cluster vector; and A program stored on a computer-readable recording medium characterized by including instructions for performing a step of learning a noise removal model using the above-mentioned learned acoustic tokenizer module and the knowledge distillation of the above-mentioned speech recognition model.

Images