Training device, estimation device, training method, estimation method, and program
By using a single adapter with a learnable gating mechanism for multiple languages, the method addresses the challenge of knowledge sharing and interference in multilingual speech recognition, achieving reduced training costs and improved performance.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- NT T INC
- Filing Date
- 2024-11-29
- Publication Date
- 2026-06-04
AI Technical Summary
Existing multilingual speech recognition models face issues with knowledge sharing and complementary performance improvement due to separate adapters for each language, leading to increased training costs and interference between languages.
Implementing a single adapter with a learnable gating mechanism that allows multiple residual adapters to be shared among languages, using a distribution mechanism to select the appropriate adapter based on language similarity, reducing the number of adapters needed.
This approach reduces training costs and avoids interference while enabling complementary performance improvement by sharing knowledge between languages.
Smart Images
Figure JP2024042362_04062026_PF_FP_ABST
Abstract
Description
Learning device, estimation device, learning method, estimation method, and program
[0001] The present invention relates to a speech recognition technology capable of supporting multiple languages.
[0002] As the demand for communication between different languages increases and rich datasets become available, attention has been focused on the development of large-scale multilingual speech recognition models.
[0003] With the emergence of large-scale pre-trained multilingual speech recognition models such as Whisper and XLS-R (hereinafter also referred to as "large-scale pre-trained models"), it has become possible to construct a speech recognition model customized for a specific language by fine-tuning these models.
[0004] As a method for customizing a large-scale pre-trained model, Non-Patent Document 1 is known, in which a small neural network (hereinafter also referred to as an "adapter") is individually prepared for each language and the adapter is inserted into each layer of the large-scale pre-trained model.
[0005] Junwen Bai et al., "Efficient Adapter Finetuning for Tail Languages In Streaming Multilingual ASR", 2021.
[0006] However, since parameters are not shared between different data in the prior art, it is not possible to aim for the effect that knowledge is shared between similar languages and performance is complementarily improved.
[0007] An object of the present invention is to provide a learning device, an estimation device, methods thereof, and a program that can be expected to share knowledge and complementarily improve performance by using the same adapter between similar languages.
[0008] To solve the above problems, according to one aspect of the present invention, the learning device includes a model application unit that uses a multi-language speech recognition model consisting of a neural network model with multiple layers to perform speech recognition processing on information based on training audio data and output a speech recognition result, and an update unit that updates the parameters used in the multi-language speech recognition model to minimize the error between the speech recognition result and the correct label. Each layer of the multi-language speech recognition model performs speech processing on information based on training audio data or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of multiple language-dependent models, and performs speech processing on the first processing result using a language-dependent model that performs language-dependent speech processing to obtain a second processing result.
[0009] To solve the above problems, according to another aspect of the present invention, the estimation device includes a model application unit that uses a trained multi-language speech recognition model consisting of a neural network model comprising multiple layers to perform speech recognition processing on information based on speech data to be recognized and outputs a speech recognition result. Each layer of the multi-language speech recognition model performs speech processing on information based on speech data to be recognized or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result. The first processing result is then distributed to one of multiple language-dependent models, and speech processing is performed on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result.
[0010] According to the present invention, when training a speech recognition model that can handle multiple languages, it is expected that knowledge can be shared between languages using the same adapter, thereby improving performance in a complementary manner.
[0011] A diagram illustrating the configuration of the estimation system according to the first embodiment. A functional block diagram of the learning device included in the estimation system according to the first embodiment. A diagram showing an example of the processing flow of the learning device included in the estimation system according to the first embodiment. A diagram showing an example of the configuration of the k-th layer of the multi-language speech recognition model used in the model application unit. A functional block diagram of the estimation device included in the estimation system according to the first embodiment. A diagram showing an example of the processing flow of the estimation device included in the estimation system according to the first embodiment. A diagram showing an example of the configuration of the k-th layer of the multi-language speech recognition model used in the model application unit. A diagram showing an example of the configuration of a computer to which this method is applied.
[0012] Embodiments of the present invention will be described below. In the drawings used in the following description, components with the same function or steps that perform the same processing will be denoted by the same reference numerals, and redundant explanations will be omitted.
[0013] <Key Points of the First Embodiment> Simply fine-tuning a large, pre-trained model for a new, specific language improves recognition performance in that language, but leads to a problem of "catastrophic forgetting," where performance in other languages deteriorates (see Reference 1).
[0014] (Reference 1) Luca Della Liber et al., "CL-MASR: A Continual Learning Benchmark for Multilingual ASR", 2023. On the other hand, when a large-scale pre-trained model is trained simultaneously in multiple languages, interference occurs between languages with different systems, leading to a decrease in the performance of each language (see Reference 2).
[0015] (Reference 2) Oliver Adams et al., "Massively Multilingual Adversarial Speech Recognition", 2019. Because large-scale pre-trained models have a large number of parameters, fine-tuning the overall model takes time to train.
[0016] Therefore, Non-Patent Document 1 proposes preparing a small neural network (adapter) for each language and inserting it into each layer of a large-scale pre-trained model. In this case, as mentioned above, an adapter is required for each language, and it is not possible to aim for the effect of knowledge sharing between similar languages and complementary performance improvement.
[0017] To solve this problem, one approach is to train a single adapter with multiple languages. In this case, it becomes unnecessary to prepare parameters for each language, reducing the training cost, but interference between languages with different systems can still occur, leading to a decrease in performance.
[0018] Therefore, in this embodiment, we propose a method in which languages with similar systems share a small neural network (also called a residual adapter), insert multiple residual adapters into each layer of a large-scale pre-trained model, and select an appropriate residual adapter for each input using a learnable gating mechanism.
[0019] By providing multiple residual adaptation sections, interference between languages with different systems can be avoided while preventing performance degradation.
[0020] Furthermore, since the number of adapters can be reduced to less than the number of target languages, it is expected that performance will improve complementarily through the sharing of knowledge between languages, without having to prepare a separate fitting unit for each language. This is achieved by using a classification mechanism to train languages with similar systems using the same residual fitting unit.
[0021] During training, multiple small residual fitting units are inserted into each layer of the large-scale pre-trained model, and a learnable distribution mechanism is used to select the appropriate residual fitting unit for each input, thus enabling training.
[0022] The number of residual fit sections selected during training for each frame (or utterance) of speech data is arbitrary for each layer. If only one residual fit section is selected, the training cost is the same as a method using a single adapter.
[0023] During inference, a distribution mechanism is used to select a residual fit at each layer based on the trained model parameters. Here again, if only one residual fit is selected, the inference cost is the same as a method using a single adapter.
[0024] <First Embodiment> Figure 1 shows a diagram of the estimation system according to the first embodiment.
[0025] The estimation system according to the first embodiment includes a learning device 100 and an estimation device 200.
[0026] The learning device 100 and the estimation device 200 are special devices configured by loading a special program onto a known or dedicated computer having, for example, a central processing unit (CPU) and main memory (RAM). The learning device 100 and the estimation device 200 execute their respective processes under the control of, for example, the central processing unit. Data input to the learning device 100 and the estimation device 200 and data obtained from each process are stored, for example, in main memory, and the data stored in main memory is read to the central processing unit as needed and used for other processes. Each processing unit of the learning device 100 and the estimation device 200 may be configured, at least in part, with hardware such as integrated circuits. Each storage unit of the learning device 100 and the estimation device 200 can be configured with, for example, main memory such as RAM (Random Access Memory), or middleware such as a relational database or key-value store. However, each storage unit does not necessarily need to have the learning device 100 and estimation device 200 inside it. It may be configured as an auxiliary storage device consisting of a hard disk, optical disk, or semiconductor memory element such as flash memory, and may be provided outside the learning device 100 and estimation device 200.
[0027] Prior to training the multi-language speech recognition model, the learning device 100 receives a large-scale pre-trained model.
[0028] The learning device 100 takes information based on training audio data and correct labels as input, trains a multilingual speech recognition model, and outputs the trained multilingual speech recognition model. The information based on audio data may be, for example, the audio data itself (speech waveform), or it may be features extracted from the audio data (for example, acoustic features such as Mel-frequency cepstrum coefficients).
[0029] Prior to speech recognition processing, the estimation device 200 receives a pre-trained multi-language speech recognition model. The estimation device 200 takes information based on the speech data to be recognized as input, performs speech recognition processing, and outputs the speech recognition result.
[0030] First, let me explain the learning stages.
[0031] <Learning device 100> Figure 2 shows a functional block diagram of the learning device 100 included in the estimation system according to the first embodiment, and Figure 3 shows an example of its processing flow.
[0032] The learning device 100 includes a model application unit 110 that performs speech recognition processing using a large-scale pre-trained model, and an update unit 120.
[0033] Here, the large-scale pre-trained model consists of an i-layer neural network. For example, it includes one input layer, one or more hidden layers, and one output layer, where i is any integer greater than or equal to 3. However, the large-scale pre-trained model may also include only one input layer and one output layer, without any hidden layers, so i=2. The large-scale pre-trained model is a large-scale pre-trained model that includes a neural network trained using conventional speech recognition technology, and each layer of the neural network includes a speech processing unit that performs language-independent speech processing. This large-scale pre-trained model is not a speech recognition model customized for a specific language, but rather, for example, a large-scale pre-trained model before inserting the adapter described in Non-Patent Literature 1, or a large-scale pre-trained model such as Whisper or XLS-R.
[0034] <Model Application Unit 110> The model application unit 110 takes information based on training audio data as input and uses a multi-language speech recognition model consisting of a neural network model with multiple layers to perform speech recognition processing on the information based on training audio data (S110) and output the speech recognition result. Each layer of the multi-language speech recognition model performs speech processing on the information based on training audio data or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of the multiple language-dependent models, performs speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result, and outputs it.
[0035] The multilingual speech recognition model used in the model application unit 110 consists of i layers, similar to the large-scale pre-trained model described above. Figure 4 shows an example of the configuration of the k-th layer of the multilingual speech recognition model used in the model application unit 110. Let k = 1, 2, ..., i. The k-th layer of the neural network includes the speech processing unit 111 included in the k-th layer of the large-scale pre-trained model, a gating mechanism 113, N residual adapters 115, and an additive unit 117.
[0036] <Speech Processing Unit 111> The speech processing unit 111 takes the output values from the (k-1)th layer as input, performs speech processing using a known large-scale pre-trained model (corresponding to the aforementioned "trained language-independent model that performs language-independent speech processing") (S111), obtains the first processing result, and outputs it. Note that the parameters of the speech processing unit 111 are not updated by the update processing of the update unit 120. Here, the speech processing unit 111 of the first layer does not have a (k-1)th layer, so it takes information based on training speech data as input. The speech processing unit 111 performs language-independent speech processing and uses the same parameters regardless of the language.
[0037] <Distribution Unit 113> The distribution unit 113 takes the first processing result as input, selects n residual fitting units 115 from among the N residual fitting units 115 from the first processing result (S113), and outputs the processing result of the speech processing unit 111 to the selected residual fitting units 115. The structure of the distribution unit 113 is arbitrary; for example, it may be a simple linear layer, or if distribution is done on an utterance basis, an LSTM (Long Short Term Memory) may be used. For example, the prior art described in Reference 3 can be used.
[0038] (Reference 3) Yoohwan Kwon et al., "MoLE: MIXTURE OF LANGUAGE EXPERTS FOR MULTI-LINGUAL AUTOMATIC SPEECH RECOGNITION", 2023 For example, the distribution unit 113 outputs the processing results of the speech processing unit 111 to the top n residual fitting units 115, starting with the largest values obtained by the softmax function. Here, n is one of the values 1, 2, ..., N. Alternatively, the configuration may be set to n=1, and the processing results of the speech processing unit 111 are output only to the residual fitting unit 115 that is considered most appropriate. Note that since the residual fitting unit 115 and the language-dependent model used within the residual fitting unit 115 correspond to each other, it can be said that the distribution unit 113 is selecting a language-dependent model.
[0039] The distribution unit 113 obtains and outputs weights to be used in the addition unit 117, which will be described later. For example, the values obtained by the softmax function may be used as weights, or n weights may be set such that the larger the value obtained by the softmax function, the larger the weight.
[0040] The parameters of the distribution unit 113 are updated by the update process of the update unit 120. As a result, the parameters of the distribution unit 113 are updated so that the first processing results corresponding to speech data that would be more appropriately recognized using the same residual fitting unit 115 are distributed to the same residual fitting unit 115. Languages corresponding to the first processing results distributed to the same residual fitting unit 115 are considered to have similar language systems.
[0041] <N Residual Adaptation Units 115> Among the N residual adaptation units 115, the n residual adaptation units 115 selected by the distribution unit 113 receive the first processing result and perform language-dependent speech processing on the first processing result using a language-dependent model that executes language-dependent speech processing (S115), and output the processing result. The speech processing performed here depends on the language and uses different parameters for each residual adaptation unit 115.
[0042] Note that the parameters of the language-dependent model are updated by the update process of the update unit 120.
[0043] For example, the language-dependent model includes a down projection that compresses dimensions, a non-linear function, and an up projection that restores the dimensions. For example, it has the same structure as the language-dependent adapter (LDA) in Non-Patent Document 1.
[0044] Regardless of the number N of residual adaptation units 115, the distribution unit 113 can identify languages that can pass through the same residual adaptation unit and those that cannot. The number N can be appropriately set according to the combination of languages and the speech data. As the number N of residual adaptation units increases, it becomes difficult to aim for complementary performance improvement. It is better for the number of residual adaptation units to be less than the number of languages to be recognized. However, for example, even when the number of residual adaptation units is greater than the number of languages to be recognized, there may be cases where multiple languages are assigned to a single residual adaptation unit, and complementary performance improvement may be achieved. Also, not all of the N residual adaptation units 115 are necessarily used, and there may be residual adaptation units 115 that are not used in the estimation stage. The number N of residual adaptation units should be 2 or more. From the perspective of suppressing an increase in the number of parameters, it is desirable to make it not more than the number of target languages. However, even if it is prepared to exceed the number of target languages, there may be cases where multiple languages are assigned to a single residual adaptation unit, and complementary performance improvement can be expected. However, not all of the prepared residual adaptation units are necessarily used.
[0045] <Addition unit 117> The addition unit 117 takes as inputs the processing result of the voice processing unit 111, the processing results of the n residual adaptation units 115 selected by the distribution unit 113, and weights, and performs weighted addition (S117), and outputs the calculation result as the output value of the k-th layer.
[0046] The above processes S111 to S117 are repeated from the first layer to the i-th layer.
[0047] <Update unit 120> The update unit 120 takes as inputs the output value (estimated speech recognition result) from the last layer (i-th layer) of the neural network and the correct label (correct speech recognition result), and updates the parameters used in the distribution unit 113 and the N residual adaptation units 115 so as to minimize the error between the output value from the last layer of the neural network and the correct label (S120). As a method for updating the parameters, for example, the gradient descent method, the error backpropagation method, etc. can be considered.
[0048] When a predetermined condition is satisfied (YES in S121), the update unit 120 outputs the model including the parameters at that time as a learned multi-language compatible speech recognition model.
[0049] When the predetermined condition is not satisfied (NO in S121), the update unit 120 outputs the updated parameters to the distribution unit 113 and the N residual adaptation units 115, and controls each unit so as to repeat the processes S111 to S120.
[0050] As the predetermined condition, a condition for determining whether or not the update of the parameters has converged may be set. For example, has the learning been repeated a certain number of times (e.g., several times)? Is the difference between the parameters before and after the update less than a predetermined threshold? Is the loss less than a predetermined threshold? etc. Such conditions can be used.
[0051] Next, the estimation stage will be described.
[0052] <Estimation device 200> FIG. 5 is a functional block diagram of the estimation device 200 included in the estimation system according to the first embodiment, and FIG. 6 shows an example of its processing flow.
[0053] The estimation device 200 receives a learned multi-language compatible speech recognition model from the learning device 100 prior to the estimation process.
[0054] The estimation device 200 includes a model application unit 210 that performs speech recognition processing using a pre-trained multi-language speech recognition model. The model application unit 210 takes information based on speech data to be recognized as input instead of information based on training speech data, and performs the same processing as the model application unit 110.
[0055] <Model Application Unit 210> The model application unit 210 takes information based on the speech data to be recognized as input and uses a trained multi-language speech recognition model consisting of a neural network model with multiple layers to perform speech recognition processing on the information based on the speech data to be recognized (S210), and outputs the speech recognition result. As described above, each layer of the multi-language speech recognition model performs speech processing on the information based on the training speech data or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of the multiple language-dependent models, performs speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result, and outputs it.
[0056] Figure 7 shows an example of the configuration of the k-th layer of the multi-language speech recognition model used in the model application unit 210. The k-th layer of the neural network includes a speech processing unit 211, a distribution unit 213, N residual fitting units 215, and an addition unit 117, which correspond to the speech processing unit 111, distribution unit 213, N residual fitting units 215, and addition unit 217, respectively, and performs S211, S213, S215, and S217 in Figure 6.
[0057] <Effects> With the above configuration, when training a speech recognition model that can handle multiple languages, the number of adapters can be reduced to less than the number of target languages. It is also expected that knowledge can be shared among languages using the same adapter, and performance can be improved in a complementary manner.
[0058] In Non-Patent Document 1, a small neural network (adapter) was prepared individually for each language, and a language ID indicating which language the audio data belonged to was assigned to the information based on the training audio data or the information based on the audio data to be recognized. However, in this embodiment, since the distribution unit determines which residual fitting unit to distribute the data to, it is not necessary to assign a separate language ID.
[0059] <Modification> As explained above in <N residual fitting units 115>, some residual fitting units 115 may not be used in the estimation stage among the N residual fitting units 115. In this case, the unused residual fitting units 115 can be identified from the output, and the identified residual fitting units 115 can be deleted in the estimation stage, resulting in a configuration that does not use them. This configuration can reduce the number of parameters in the estimation stage. However, since the parameters of the distribution unit 113 are updated in the update process of the update unit 120, it is desirable to use the same number of residual fitting units 215 without changing the conditions from the learning stage to avoid unexpected problems.
[0060] A new language may be added to the multilingual speech recognition model trained using the learning method of the first embodiment. In this case, a new residual adaptation unit 115 may or may not be added. Even if a new residual adaptation unit 115 is added, as a result of learning, the distribution unit 113 may assign the new language to the existing residual adaptation unit 115, and the new residual adaptation unit 115 may not be used. Also, adding a new language may update the parameters of the distribution unit 113, changing the relationship between the language and the residual adaptation unit.
[0061] Furthermore, although this embodiment describes a multi-language speech recognition model that includes N residual fitting units 115 in each layer, the configuration may include a different number of residual fitting units 115 in each layer. Also, it is sufficient to include multiple residual fitting units 115 in at least one layer, and the multi-language speech recognition model may include layers that do not contain residual fitting units 115.
[0062] <Modification> Furthermore, the present invention may also include a device (terminal) for using the apparatus, system, or method of the present invention via a network (telecommunication line). The "device (terminal) for use" may be equipped with functions necessary to obtain the effects of implementing the apparatus, system, or method of the present invention (for example, control functions, decoding functions, restoration functions, input / output functions, etc.). Note that a configuration including a device (terminal) for using the apparatus or method of the present invention via a network (telecommunication line) may also be called an estimation system.
[0063] The following additional information is disclosed regarding the embodiments described above.
[0064] (Note 1) A learning device comprising: memory; and at least one processor connected to the memory, wherein the processor performs speech recognition processing on information based on training audio data using a multilingual speech recognition model consisting of a neural network model comprising multiple layers, outputs a speech recognition result, updates the parameters used in the multilingual speech recognition model to minimize the error between the speech recognition result and the correct label, and in each layer of the multilingual speech recognition model, performs speech processing on the information based on the training audio data or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of a plurality of language-dependent models, and performs speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result.
[0065] (Appendix 2) A non-temporary storage medium storing a program executable by a computer to perform a learning process, wherein the learning process involves: performing speech recognition processing on information based on training audio data using a multi-language speech recognition model consisting of a neural network model including multiple layers, outputting a speech recognition result; updating parameters used in the multi-language speech recognition model to minimize the error between the speech recognition result and the correct label; and in each layer of the multi-language speech recognition model, performing speech processing on the information based on the training audio data or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result; distributing the first processing result to one of a plurality of language-dependent models; and performing speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result.
[0066] (Appendix 3) Estimation device comprising: memory; and at least one processor connected to the memory, wherein the processor performs speech recognition processing on information based on speech data to be recognized using a trained multilingual speech recognition model consisting of a neural network model comprising multiple layers, and outputs a speech recognition result; and in each layer of the multilingual speech recognition model, performs speech processing on the information based on the speech data to be recognized or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of a plurality of language-dependent models, and performs speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result.
[0067] (Appendix 4) A non-temporary storage medium storing a program executable by a computer to perform an estimation process, wherein the estimation process uses a trained multilingual speech recognition model consisting of a neural network model comprising multiple layers to perform speech recognition processing on information based on speech data to be recognized and outputs a speech recognition result, and each layer of the multilingual speech recognition model performs speech processing on the information based on the speech data to be recognized or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of a plurality of language-dependent models, and performs speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result.
[0068] <Hardware, Programs and Recording Media> The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including a general-purpose processor, an application-specific processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), a CPU (a Central Processing Unit), conventional circuits, and / or a combination thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory.
[0069] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.
[0070] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.
[0071] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 8, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.
[0072] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.
[0073] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.
[0074] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed using a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer.In addition, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. Furthermore, the term "program" in this form includes information used for processing by an electronic computer that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the processing of the computer).
[0075] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.
[0076] <Other Modifications> The present invention is not limited to the embodiments and modifications described above. For example, the various processes described above may not only be performed sequentially according to the description, but may also be performed in parallel or individually as needed, depending on the processing capacity of the device performing the processes. Other modifications can be made as appropriate without departing from the spirit of the present invention.
Claims
1. A learning device comprising: a model application unit that uses a multilingual speech recognition model consisting of a neural network model with multiple layers to perform speech recognition processing on information based on training audio data and output a speech recognition result; and an update unit that updates the parameters used in the multilingual speech recognition model to minimize the error between the speech recognition result and the correct label, wherein each layer of the multilingual speech recognition model performs speech processing on the information based on the training audio data or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of a plurality of language-dependent models, and performs speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result.
2. A learning device according to claim 1, wherein one layer of the multilingual speech recognition model includes: a speech processing unit that performs speech processing using the language-independent model and obtains the first processing result; a distribution unit that selects a predetermined number of language-dependent models from the multilingual model based on the first processing result and outputs the first processing result to the selected language-dependent models; a residual fitting unit that performs speech processing on the first processing result using the language-dependent model and obtains the second processing result; and an addition unit that weights and adds the first processing result and the second processing result, wherein the update unit updates the language-dependent model and the parameters used in the distribution unit.
3. An estimation device comprising a model application unit that uses a pre-trained multilingual speech recognition model consisting of a neural network model with multiple layers to perform speech recognition processing on information based on speech data to be recognized and outputs a speech recognition result, wherein each layer of the multilingual speech recognition model performs speech processing on the information based on the speech data to be recognized or the output value of the previous layer using a pre-trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of multiple language-dependent models, and performs speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result.
4. A learning method comprising: a model application step of performing speech recognition processing on information based on training audio data using a multilingual speech recognition model consisting of a neural network model including multiple layers, and outputting a speech recognition result; and an update step of updating the parameters used in the multilingual speech recognition model to minimize the error between the speech recognition result and the correct label, wherein each layer of the multilingual speech recognition model performs speech processing on the information based on the training audio data or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of a plurality of language-dependent models, and performs speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result.
5. An estimation method comprising a model application step of performing speech recognition processing on information based on speech data to be recognized using a trained multilingual speech recognition model consisting of a neural network model with multiple layers, and outputting a speech recognition result, wherein each layer of the multilingual speech recognition model performs speech processing on the information based on the speech data to be recognized or the output value of the previous layer using a trained language-independent model that performs language-independent speech processing to obtain a first processing result, distributes the first processing result to one of multiple language-dependent models, and performs speech processing on the first processing result using the language-dependent model that performs language-dependent speech processing to obtain a second processing result.
6. A program for causing a computer to function as a learning device according to claim 1 or claim 2, or as an estimation device according to claim 3.