Speech recognition system, computer program, and speech recognition method

Abstract

A speech recognition system 100 for providing a speech recognition system capable of performing speech recognition with high accuracy at low cost even for a speech signal of an out-of-domain utterance comprises: a speech encoder 112 and an adapter 113 that perform speech recognition on a speech signal 110 and convert the speech signal into a speech token sequence; a text LLM 124 that is trained to generate an output having a predetermined relationship to an input; a DB access unit 136 that extracts related data of an output of a large-scale language model from an embedding vector DB 134; and a context addition unit 116 that extends the input to the text LLM 124 using the speech token sequence, the output of the text LLM 124, and the related data extracted by the DB access unit 136, and re-inputs the expanded input to the text LLM 124. An output 132 of a branch unit 126 for the re-input becomes a speech recognition result for the speech signal.

Description

Voice Recognition System, Computer Program, and Voice Recognition Method

[0001] This invention relates to voice recognition technology, and particularly to voice recognition technology that can achieve high recognition accuracy for data in a domain different from the domain of the learning data. This application claims priority based on Japanese Patent Application No. 2025-009202 filed on January 22, 2025, and the entire description of the Japanese patent application is hereby incorporated by reference.

[0002] In a voice recognition system, a model trained using pre-prepared learning data, such as a neural network, is often used. Particularly, with the development of so-called deep learning, the accuracy of voice recognition has also become higher compared to the past.

[0003] Until deep learning became widely used, a language model was used in voice recognition. In voice recognition, a language model consisting of a neural network was also used. However, recently, neural networks have been used for voice recognition itself, and the use of independent language models has become less frequent.

[0004] On the other hand, as a technology that also uses neural networks, there is a large language model (Large Language Model, hereinafter referred to as "LLM"). Particularly recently, the scaling up and performance improvement of LLMs have been progressing. Therefore, using an LLM to improve the accuracy of voice recognition has been proposed in Non-Patent Document 1 below.

[0005] Referring to Figure 1, the speech recognition system 50 proposed in Non-Patent Literature 1 includes a speech encoder 66 that recognizes an input speech signal and outputs a sequence of speech vectors representing the result. The speech recognition system 50 further includes a text LLM 74, which is a general LLM; a learnable adapter 68 consisting of a neural network that adjusts the number of dimensions and the number of vectors constituting each vector in the sequence of speech vectors output by the speech encoder 66 and outputs it as a sequence of speech tokens; and a memory 70 that stores the output of the adapter 68 and inputs it into the text LLM 74. The speech recognition system 50 further includes an associated text creation unit 72.

[0006] The related text generation unit 72 behaves differently during the training and inference phases of the adapter 68. Specifically, during inference, the related text generation unit 72 sequentially inputs the token sequence output by the text LLM 74 into the text LLM 74 as related text related to the next utterance input into the text LLM 74. During learning, the related text generation unit 72 inputs a tokenized transcript of an utterance into the text LLM 74 as related text for that utterance.

[0007] The training data 60 for training the adapter 68 of the speech recognition system 50 includes multiple training data items. Each training data item includes multiple speech signals 62 representing utterances and a transcript 64 corresponding to the speech signals 62.

[0008] During training, the speech recognition system 50 basically operates as follows. First, the adapter 68 is in its initial state. An adapter 68 that has already undergone basic training may be used. One of the speech signals 62 is given to the speech encoder 66. The speech vector sequence output by the speech encoder 66 is input to the adapter 68. The adapter 68 adjusts the number of dimensions of each vector in this speech vector sequence and the number of vector sequences to be suitable for input to the text LLM 74, and outputs it to the memory 70 as a speech token sequence. This speech token sequence is stored in the memory 70 in predetermined units (e.g., one sentence). Meanwhile, the related text creation unit 72 reads the transcript 64 corresponding to the speech signal input to the speech encoder 66, tokenizes it, adds it to the speech token sequence output by the adapter 68 stored in the memory 70, and inputs it to the text LLM 74. At this time, the tokens output by the related text creation unit 72 are selected from the transcript 64 up to the character that the text LLM 74 should infer, and provided to the related text creation unit 72.

[0009] The inference result 76 output by the text LLM 74 is compared with the corresponding transcript 64, and the parameters of the adapter 68 are adjusted so that the difference between them is minimized.

[0010] This learning process is performed on all of the training data stored in training data 60, and the training of the adapter 68 is completed. Note that the above explanation is simplified, and various techniques are employed to speed up the learning process.

[0011] During inference, the speech encoder 66 converts the audio signal into an audio vector sequence. The adapter 68 converts this audio vector sequence into an audio token sequence by changing the number of dimensions and the number of vectors in each vector according to the learning results, and stores it in memory 70 in predetermined units (e.g., each sentence). For each predetermined unit (e.g., one sentence), the accumulated audio token sequence is read from memory 70 and input to text LLM 74. In response to this audio token sequence, the tokens generated and output by text LLM 74 are sequentially provided to the related text creation unit 72 as output 80. Each time the related text creation unit 72 receives a token from text LLM 74 in output 80, it adds the received token sequence to the output of memory 70 and inputs it back into text LLM 74. This process is repeated until text LLM 74 receives a token indicating the end of output. All the tokens output by text LLM 74 are concatenated and output as the inference result 76.

[0012] Mingqiu Wang, et al., SLM: Bridge the thin gap between speech and text foundation models, [online], Cornell University, Arxiv, September 2023, [Retrieved November 20, 2020], Internet<URL: https: / / arxiv.org / abs / 2310.00230> .

[0013] Non-patent document 1 states that the accuracy of speech recognition results can be improved by LLM using the above technology. However, there is still room for improvement in this technology. One of the problems is that only limited training data can be used for LLM training. Typically, the training data for LLM is limited to data up to a certain point in the past. Therefore, even if there is newer information than that point in time, it will not be reflected in the speech recognition results. When there is input from a field other than the data used for training, the improvement of speech recognition results is not successful. Furthermore, confidential information within a company is not included in the training data, so it is difficult to improve speech recognition in this regard as well. These problems can be summarized as the inability to improve the recognition accuracy for out-of-domain utterances.

[0014] One method used to improve these problems is to create a dictionary of technical terms and refer to it during inference. However, it is difficult to use dictionaries in so-called end-to-end generative models, such as recent LLMs. Even if dictionaries could be used, it would only be at the word level, and it would be difficult to improve the accuracy of speech recognition for the entire utterance.

[0015] Another approach is to collect out-of-domain data and use it to fine-tune or further train the LLM. However, this method cannot handle sensitive information or other data that is difficult to collect. Furthermore, it requires updating the model, which can be costly, especially for further training of the LLM.

[0016] Therefore, the object of this invention is to provide a speech recognition system, computer program, and speech recognition method that can accurately recognize speech signals from out-of-domain speech at low cost.

[0017] The speech recognition system according to the first aspect of this invention includes: a speech token sequence output unit that receives a speech signal, performs speech recognition on the speech signal and converts it into a speech token sequence; a large-scale language model trained to generate an output having a predetermined relationship with an input; a database access unit that accesses a predetermined database and retrieves related data having a predetermined relationship with the output of the large-scale language model; and an input extension unit that uses the output of the speech token sequence output unit, the output of the large-scale language model which receives the output from the speech token sequence output unit as input, and the related data retrieved by the database access unit which receives the output of the large-scale language model to extend the input to the large-scale language model and re-input it to the large-scale language model, wherein the output of the large-scale language model to the re-input becomes the speech recognition result for the speech signal.

[0018] Preferably, the input extension unit includes a related data addition unit that adds related data to the end of the output of the voice token sequence output unit.

[0019] More preferably, the input extension unit further includes a prompt addition unit that adds a prompt for a large-scale language model to the beginning of the output of the speech token sequence output unit.

[0020] More preferably, the prompt instructs the large-scale language model to modify the output of the speech token sequence output unit included in the augmented input, with reference to the relevant data.

[0021] A computer program relating to the second aspect of this invention is a computer program that causes a computer to function as an input extension unit that extends the input to the large-scale language model and re-inputs it to the large-scale language model using the output of the voice token sequence output unit, the output of the large-scale language model that has received the output of the voice token sequence output unit, and the related data output by the database access unit that has received the output of the large-scale language model, wherein the output of the large-scale language model in response to the re-input becomes the speech recognition result for the voice signal.

[0022] A third aspect of the present invention relates to a speech recognition method comprising the steps of: a computer receiving a speech signal, performing speech recognition on the speech signal and converting it into a sequence of speech tokens; a computer inputting the sequence of speech tokens into a large-scale language model trained to generate outputs having a predetermined relationship with an input; a computer accessing a predetermined database and retrieving related data having a predetermined relationship with the output of the large-scale language model; a computer using the sequence of speech tokens, the output of the large-scale language model, and the related data to expand the input to the large-scale language model and re-inputting it into the large-scale language model; and outputting the output of the large-scale language model for the re-input as a speech recognition result for the speech signal.

[0023] As described above, this invention provides a speech recognition system, computer program, and speech recognition method that can accurately recognize speech signals from out-of-domain speech at low cost.

[0024] Figure 1 is a block diagram illustrating the schematic configuration of a speech recognition system described in Non-Patent Document 1. Figure 2 is a block diagram illustrating the schematic configuration of a speech recognition system according to the first embodiment of this invention. Figure 3 is a block diagram illustrating the process of creating an embedded vector DB (Data Base) used by the speech recognition system shown in Figure 2. Figure 4 is a block diagram illustrating the functional configuration of the DB access unit shown in Figure 2. Figure 5 is a block diagram illustrating the functional configuration of the context addition unit shown in Figure 2. Figure 6 is a block diagram illustrating the functional configuration of the prompt addition unit shown in Figure 2. Figure 7 is a block diagram illustrating the functional configuration of a training device for training the speech recognition system shown in Figure 2. Figure 8 is a flowchart illustrating the procedure for training the speech recognition system using the training device shown in Figure 7. Figure 9 is a flowchart illustrating the control structure of a program for the step of creating the learning data shown in Figure 8. Figure 10 is a flowchart illustrating the control structure of a program for the step of training the speech encoder shown in Figure 8. Figure 11 is a flowchart illustrating the control structure of a program for the step of training the text LLM shown in Figure 8. Figure 12 is a flowchart illustrating the control structure of a program that performs speech recognition using the speech recognition system according to the first embodiment of this invention. Figure 13 is a flowchart showing the control structure of a program for processing the first stage of inference (speech recognition) by the speech recognition system shown in Figure 2. Figure 14 is a flowchart showing the control structure of a program for processing the second stage of inference by the speech recognition system shown in Figure 2. Figure 15 is a flowchart showing the control structure of a program for preparing learning data for the speech recognition system according to the second embodiment of this invention. Figure 16 is a block diagram showing the functional configuration of a training device for preparing learning data for the speech recognition system according to the third embodiment of this invention. Figure 17 is a flowchart showing the control structure of a program that realizes the training device shown in Figure 16. Figure 18 is a diagram showing experimental results for the speech recognition system according to an embodiment of this invention, in comparison with experimental results from the prior art, in tabular form. Figure 19 is a diagram showing experimental results for the speech recognition system according to an embodiment of this invention, in tabular form, to confirm the effect of using prompts.Figure 20 shows the external appearance of a computer that implements each embodiment of the speech recognition system and training device according to the present invention. Figure 21 is a block diagram showing the hardware configuration of the computer shown in Figure 20.

[0025] In the following descriptions and diagrams, identical parts are assigned the same reference number. Therefore, detailed explanations of them will not be repeated. Also, in the following descriptions, each data item included in the training data is referred to as a training data item. The same applies to the training preparation data described later.

[0026] First Embodiment (Figures 2 to 14) 1. Configuration A. Overall Configuration (Figure 2) Referring to Figure 2, the speech recognition system 100 according to the first embodiment of the present invention includes a speech encoder 112 that converts a speech signal into a sequence of speech vectors, a learnable adapter 113 consisting of a neural network that converts the sequence of speech vectors output by the speech encoder 112 into a sequence of speech tokens of a predetermined number of dimensions and a predetermined format, and a memory 114 that stores the sequence of speech tokens output by the adapter 113.

[0027] The speech recognition system 100 further includes a selector 122. The selector 122 has a first input, a second input, and a third input. The first input is given a speech token sequence 138 from memory 114. The second input is given a token sequence 142. The source of the token sequence 142 will be described later. The third input is given a token sequence 146, which is the final speech recognition result for the preceding input. During training, the selector 122 selects the token sequence 142 and outputs it with the token sequence 146 appended. During estimation, the selector 122 selects the speech token sequence 138 and outputs it with the token sequence 146 appended to it. The speech recognition system 100 further includes a text LLM 124 connected to receive the token sequence 144 output by the selector 122, which predicts subsequent tokens in order based on the input token sequence and outputs them as a token sequence.

[0028] The speech encoder 112, adapter 113, and text LLM 124 all include neural networks and are trained using training data as described later.

[0029] The speech recognition system 100 further includes a branching unit 126 that, in response to a control signal, branches the token sequence output by the text LLM 124 to two output destinations. One branch of the branching unit 126 is output externally as the token sequence 132 of the speech recognition result. The token sequence 146 branched from the token sequence 132 is input to the selector 122 as a third input. The speech recognition system 100 further includes a memory 130 that stores the token sequence which is the other branch from the branching unit 126.

[0030] The speech recognition system 100 further includes a memory 130 and a DB access unit 136 connected to an external embedding vector DB 134. The embedding vector DB 134 stores pre-prepared texts that have been concatenated and then chunked. That is, the embedding vector DB 134 stores a text corpus consisting of chunks of text. An embedding vector is generated from each chunk, and when an embedding vector is input from the outside, a predetermined number of chunks corresponding to embedding vectors in the vicinity of the input vector are retrieved and output in the embedding vector space. The retrieved chunks correspond to a predetermined number of higher-order embedding vectors closest to the input vector.

[0031] The DB access unit 136 reads at least a portion of the token sequence stored in memory 130, accesses the embedding vector DB 134, and receives a predetermined number of chunks of text from the embedding vector DB 134.

[0032] The speech recognition system 100 further includes a context-adding unit 116 connected to a memory 114 and a DB access unit 136. The context-adding unit 116 reads a speech token sequence 140 from the memory 114. The context-adding unit 116 also receives chunks of text from the DB access unit 136. The context-adding unit 116 tokenizes and concatenates the chunks received from the DB access unit 136 to the end of the speech token sequence 140 and outputs it.

[0033] The speech recognition system 100 further includes a prompt addition unit 118 that receives a token sequence concatenated by a context addition unit 116. The prompt addition unit 118 concatenates a token sequence of a prompt string, which is an instruction for the text LLM 124, to the beginning of the token sequence from the context addition unit 116, and provides it as a token sequence 142 to the second input of the selector 122. The speech recognition system 100 further includes a prompt storage unit 120 connected to the prompt addition unit 118, which pre-stores prompt strings (or their token sequences) to be used by the prompt addition unit 118. The prompt string instructs the text LLM 124 on how to process the input. Details of an example of a prompt string will be described later.

[0034] The context addition unit 116 and the prompt addition unit 118 implement the function of expanding the token sequence output from the speech encoder 112 and input to the text LLM 124 after conversion by the adapter 113. Specifically, the prompt addition unit 118 expands the token sequence input to the text LLM 124 by adding a predetermined prompt to a chunk of text retrieved by the DB access unit 136 from the embedded vector DB 134 based on the output of the text LLM 124. This expanded input is then provided back to the text LLM 124 via the selector 122.

[0035] The speech recognition system 100 further includes a control unit 128 that controls various parts of the speech recognition system 100, including the selector 122 and the branching unit 126, to execute the processing described later.

[0036] B. Preparation of the Embedded Vector DB 134 (Figure 3) Referring to Figure 3, in this embodiment, the embedded vector DB 134 is assumed to be located outside the speech recognition system 100. However, the invention is not limited to such embodiments. The embedded vector DB 134 may be included inside the speech recognition system 100. The embedded vector DB 134 is prepared in advance according to the domain that the speech recognition system 100 targets, such as confidential information of a certain company or text newly collected from the web after learning the text LLM 124. When dealing with confidential information of a certain company, it is desirable to provide the embedded vector DB 134 inside the speech recognition system 100, or to place the embedded vector DB 134 in a space different from the speech recognition system 100 and connect the two by secure communication means. For example, in the case of text collected from the web after learning the text LLM 124, the embedded vector DB 134 may be open access, or it may be connected to the network in a closed format that can only be accessed by users who have made a predetermined subscription.

[0037] The embedding vector DB 134 is constructed, for example, by a DB construction unit 150 having the following configuration. That is, the DB construction unit 150 in this example includes a text corpus 160 that stores pre-collected text from a specific domain. The DB construction unit 150 further includes a chunking unit 162 that concatenates all the text stored in the text corpus 160 and then divides it into chunks of a predetermined length, and an embedding unit 164 that vectorizes each of the multiple chunks output by the chunking unit 162 into an embedding vector of a predetermined number of dimensions.

[0038] The embedding vector DB 134 stores the embedding vectors output by the embedding unit 164, associating each embedding vector with a corresponding chunk. The embedding vectors stored in the embedding vector DB 134 form a vector space of a predetermined dimension. When an embedding vector to be searched is input to the embedding vector DB 134, it identifies a predetermined number of higher-order embedding vectors in the vector space that have the closest defined distance to the embedding vector to be searched. The embedding vector DB 134 has the function of outputting the chunk associated with each of the embedding vectors thus identified. Note that the "distance" referred to here can be any definition that satisfies the properties of "distance" in a vector space. Typically, the square root of the sum of the squares of the differences of each element of two vectors is used as the distance between those two vectors.

[0039] C. DB Access Unit 136 (Figure 4) Referring to Figure 4, the DB access unit 136 includes an embedding unit 200 and a chunk search unit 202. If the number of token sequences stored in memory 130 exceeds a predetermined number, the embedding unit 200 reads a predetermined number of token sequences from the beginning and vectorizes them into an embedding vector of the same dimension as the embedding vector in embedding vector DB 134. The chunk search unit 202 receives the embedding vector from the embedding unit 200 and provides the embedding vector to embedding vector DB 134, thereby receiving a predetermined number (let's call this K chunks) of chunks from embedding vector DB 134. The chunk search unit 202 provides these K chunks to the context addition unit 116.

[0040] D. Context Addition Unit 116 (Figure 5) Referring to Figure 5, the context addition unit 116 includes a tokenization unit 230 that receives chunks from the DB access unit 136 and tokenizes them, and a memory 232 that stores a context token sequence 250 consisting of a speech token sequence 140 from the memory 114 and tokens output by the tokenization unit 230. In the memory 232, the context token sequence 250 is concatenated to the end of the speech token sequence 140. The speech token sequence 140 and the context token sequence 250 form a new token sequence 234. This token sequence 234 is provided to the prompt addition unit 118.

[0041] E. Prompt Addition Unit 118 (Figure 6) Referring to Figure 6, the prompt addition unit 118 includes a prompt reading unit 280 that reads the prompt text from the prompt storage unit 120 and tokenizes it, and a memory 282 that stores the token sequence 234 from the context addition unit 116. In the memory 282, the prompt token sequence 300 from the prompt reading unit 280 is concatenated before the voice token sequence 140. In this embodiment, the prompt token sequence 300, the voice token sequence 140, and the context token sequence 250 form a prompted voice token sequence concatenated in this order. This token sequence is provided to the second input of the selector 122, as shown in Figure 2, as the token sequence 142 shown in Figure 2.

[0042] Furthermore, the prompts used serve as a guide for the text LLM124, instructing how to effectively incorporate and use the domain-specific content obtained through the search process described above into the speech recognition process. For example, in the experiment described later, the following prompt was used. It is desirable to use a language in which the text LLM124 is expected to perform at its best as this prompt. "Transcribe the audio strictly following its content. Use context to verify technical terms and domain-specific vocabulary when uncertain. Ensure the transcription reflects exactly what is spoken, with context aiding in clarifying domain-related ambiguities."

[0043] F. Learning Device (Figures 7 to 11) a. Overall Configuration (Figure 7) Figure 7 shows a schematic configuration of the training device 330, which trains the speech encoder 112 and text LLM 124 of the speech recognition system 100, in block diagram form. Referring to Figure 7, the training device 330 includes a learning data creation unit 342 that creates learning data used in common for both training the text LLM 124 and training the speech encoder 112 from learning preparation data 340, which is the basis for creating learning data. The training device 330 further includes a learning data storage unit 344 that stores the learning data created by the learning data creation unit 342, an LLM learning unit 346 that trains the text LLM 124 using the learning data stored in the learning data storage unit 344, and a speech encoder / adapter learning unit 348 that trains the speech encoder 112 and adapter 113 using the learning data also stored in the learning data storage unit 344.

[0044] The learning preparation data 340 includes a plurality of learning preparation data items. Each learning preparation data item includes an audio signal 350 representing an utterance and associated text 352 consisting of a transcription of that utterance.

[0045] The learning data creation unit 342 has a configuration that is partially common to the configuration of the speech recognition system 100 shown in FIG. 2. More specifically, the learning data creation unit 342 includes a speech encoder 112 that receives as input the audio signal 350 of each learning data item of the learning preparation data 340, and an adapter 113 that outputs a sequence of audio tokens by converting each vector of the sequence of audio vectors output by the speech encoder 112 into an audio token of a predetermined number of dimensions and a predetermined format, and a memory 114 that stores the sequence of audio tokens output by the adapter 113. Note that the speech encoder 112 is depicted at two locations in FIG. 7, but these represent the same thing. The same applies to the adapter 113.

[0046] The learning data creation unit 342 further includes a context addition unit 116 that is connected to the memory 114 and tokenizes the associated text 352 stored in the associated text 352 corresponding to the audio signal 350 that is the target of speech recognition at the end of the sequence of audio tokens stored in the memory 114 and concatenates it as context information of the utterance. The learning data creation unit 342 further includes a prompt addition unit 118 that concatenates the sequence of tokens obtained by tokenizing the prompt stored in the prompt storage unit 120 and the sequence of tokens stored in the context addition unit 116 and stores them in the learning data storage unit 344 in association with the corresponding learning preparation data item in the learning preparation data 340. Note that the audio signal 350 and the associated text 352 are associated with each other, and this association is maintained hereafter.

[0047] As a result, each learning data item stored in the learning data storage unit 344 includes the audio signal 350 and a sequence of tokens obtained by concatenating (1) a sequence of prompt tokens, (2) a sequence of audio tokens obtained by the speech encoder 112 and the adapter 113, and (3) a sequence of tokens representing the associated text 352.

[0048] Regarding the configuration of the program that realizes the LLM learning unit 346 and the speech encoder adapter learning unit 348, it will be described by the flowchart shown in FIG. 8 and below.

[0049] b. Training process (FIG. 8) The training processes of the speech encoder 112, the adapter 113, and the text LLM 124 are shown in FIG. 8. Referring to FIG. 8, the training process includes a step 380 of preparing the learning preparation data 340 shown in FIG. 8, and a step 382 of creating learning data using the learning preparation data 340 and storing it in the learning data storage unit 344. The training process further includes a step 384 of training the speech encoder 112 and the adapter 113 using the learning data stored in the learning data storage unit 344, and a step 386 of training the text LLM 124 using the learning data stored in the learning data storage unit 344.

[0050] This learning is performed using the speech recognition system 100 shown in FIG. 2. Note that the embedding vector DB 134 used during learning and the embedding vector DB 134 used during inference do not necessarily have to match.

[0051] In this embodiment, the speech encoder 112, the adapter 113, and the text LLM 124 are trained using common learning data. However, they may be trained using separate learning data. Furthermore, each step in FIG. 8 represents a procedure, and it is not necessarily required to continuously execute this entire process by a computer program. It is also possible to execute each step separately. It is desirable to execute step 384 and step 386 in this order. However, the order of the two may be reversed, or the two may be executed in parallel.

[0052] c. Creation of learning data (FIG. 9) Referring to FIG. 9, the control structure of the program that realizes step 382 shown in FIG. 8 is as follows. That is, this program includes a step 400 of repeating step 402 for each learning preparation data item in the learning preparation data 340.

[0053] Step 402 includes reading a learning preparation data item in step 420 and tokenizing the associated text contained in the learning preparation data item in step 422. Step 402 further includes inputting the audio signal contained in the learning data item to be processed into the speech encoder 112 in step 424, outputting an audio token sequence by converting each vector in the audio vector sequence output by the speech encoder 112 in response to the audio signal into an audio token of a predetermined number of dimensions and a predetermined format using the adapter 113, and storing the audio token sequence in the memory 114 in step 426.

[0054] Step 402 further includes step 428, which appends a sequence of tokens for the associated text to the end of the sequence of speech tokens stored in memory 114, and step 430, which appends a sequence of tokens for the prompt to the beginning of the sequence of tokens stored in memory 114. In this embodiment, a sequence of tokens for the prompt is added to the training data item. However, the invention is not limited to such an embodiment. If the prompt tokens are appended to the training data item when training the speech encoder 112, adapter 113, and text LLM 124 using this training data item, step 430 is unnecessary. In this case, the prompt storage unit 120 and the prompt appending unit 118 shown in Figure 7 are unnecessary in the training device 330.

[0055] Finally, step 402 includes step 432, which stores the token sequence obtained through the processing up to step 430 as a learning data item in the learning data storage unit 344 shown in Figure 7, thereby ending step 402.

[0056] d. Optimization of speech encoder 112 and adapter 113 (Figure 10) Figure 10 shows the control structure of a program (step 384 shown in Figure 8) for realizing a speech encoder / adapter learning unit 348 that optimizes the speech encoder 112 and adapter 113 shown in Figure 7. In this example, for the sake of clarity in illustration and explanation, the speech encoder 112 and adapter 113 are optimized for each learning data item. However, this invention is not limited to such embodiments. For example, the speech encoder 112 and adapter 113 may be optimized by using mini-batches. In practice, the process shown in Figure 10 is considered as one epoch, and processing is performed for a predetermined number of epochs, but Figure 10 simplifies this and shows the process ending in only one epoch. The same applies to Figure 11, which will be described later.

[0057] Referring to Figure 10, the program that implements step 384 includes step 450, which performs the processing of step 452 for all training data items.

[0058] Step 452 includes step 460 of receiving (reading out) an audio signal included in the learning data item to be processed, and step 462 of inputting this audio signal to the speech encoder 112. Step 452 further includes step 463 of converting the audio vector sequence output by the speech encoder 112 into an audio token sequence using the adapter 113 in response to the audio signal input to the speech encoder 112 in step 462, and step 464 of storing the audio token sequence converted in step 463 in memory.

[0059] Step 452 further includes step 466, which generates an input to the text LLM 124 using the training data items and the output of the adapter 113, and inputs it into the text LLM 124. Finally, step 452 includes step 468, which optimizes the parameters of the speech encoder 112 and the adapter 113 using the token sequence output by the text LLM 124 in response to the input in step 466, and terminates step 452. In step 468, the parameters of the text LLM 124 are fixed, and only the parameters of the speech encoder 112 and the adapter 113 are optimized so that the difference between the token sequence output by the text LLM 124 and the associated text (transcription) contained in the training data items is small.

[0060] This process is performed for each training data item, completing one epoch of training. In reality, this process is performed for multiple epochs.

[0061] e. Optimization of text LLM124 (Figure 11) Figure 11 shows the control structure of a program (step 386 shown in Figure 8) for realizing the function of the LLM learning unit 346 that optimizes the text LLM124 shown in Figure 7 using a computer, in flowchart form.

[0062] Referring to Figure 11, step 386 includes step 480, which terminates step 386 by performing step 482, described below, for all training data items.

[0063] Step 482 includes step 490 of receiving (reading out) an audio signal from a learning data item, and step 492 of inputting this audio signal to the speech encoder 112. Step 482 further includes step 493 of inputting a sequence of audio vectors obtained from the speech encoder 112 in response to the processing in step 492 to the adapter 113, and step 494 of storing the sequence of audio tokens output by the adapter 113 in step 493 in memory.

[0064] Step 482 further includes step 496, which generates an input to text LLM 124 using the prompt and associated text of the learning data item being processed and the output from adapter 113 held in memory, and inputs it into text LLM 124. Finally, step 482 includes step 498, which optimizes the parameters of text LLM 124 so that the difference between the token sequence output by text LLM 124 in response to the processing in step 496 and the associated text is small, and then terminates step 482. In this embodiment, only text LLM 124 is optimized in step 498.

[0065] Once these processes are performed for each training data item, one epoch of training is completed.

[0066] In this embodiment, the speech encoder 112 and adapter 113 and the text LLM 124 are optimized separately. However, the invention is not limited to such embodiments. The speech encoder 112 and adapter 113 and the text LLM 124 may be optimized together.

[0067] G. Configuration for Inference a. Overall Configuration of Inference The functional configuration of the speech recognition system 100 is as shown in Figure 2. However, each function of the speech recognition system 100 is realized through the cooperation of computer hardware and other computer hardware.

[0068] Figure 12 shows the processing flow of the speech recognition system 100 during inference. Referring to Figure 12, the program executed by the computer to realize the speech recognition system 100 during inference includes a step 520 of receiving an audio signal to be recognized, and a step 522 of performing a first-stage processing to generate a second input to the text LLM 124 using the speech encoder 112, adapter 113, text LLM 124, DB access unit 136, context addition unit 116, and prompt addition unit 118 on the audio signal received in step 520. Specifically, in step 522, the computer generates a second input to the text LLM 124 by adding contextual information related to the output of the text LLM 124 to the output of the text LLM 124, and also by adding a prompt.

[0069] The program further includes step 524, which performs a second stage of processing in which speech recognition is performed again on the audio signal using the information obtained from the embedding vector DB134 as a reference, by inputting the input obtained as a result of the first stage processing in step 522 back into the text LLM124. The output of step 524 is output as the speech recognition result for the input audio signal.

[0070] The selector 122 (Figure 2) selects the first input in the first stage and the second input in the second stage during both training and inference. The text LLM 124 (Figure 2) provides its output to the memory 130 in the first stage and outputs it externally in the second stage during both training and inference. These controls are performed by the control unit 128 (Figure 2). In both stages, the selector 122 adds the third input, token sequence 146, to the selected token sequence and outputs it.

[0071] As mentioned above, the embedding vector DB134 used during training and the embedding vector DB134 used during inference do not need to be the same. For example, even if it is necessary to handle confidential information during inference, it is sufficient to use an embedding vector DB134 obtained from publicly available information during training, and it is not necessary to perform training using confidential information. Similarly, it is acceptable to use an embedding vector DB134 used during training that only contains data up to the time of its construction, and to use an embedding vector DB134 used during inference that contains data added between the time of construction and the time of inference.

[0072] b. In the configuration diagram 13 for the processing of the first stage, the control structure of the program for realizing step 522 shown in Figure 12 by a computer is shown in flowchart form.

[0073] Referring to Figures 13 and 2, step 522 includes step 542, which inputs the audio signal obtained in step 520 (Figure 12) to the speech encoder 112 (Figure 2). The program further includes step 543, which inputs the audio vector sequence output from the speech encoder 112 as a result of the processing in step 542 to the adapter 113, and step 544, which stores the audio token sequence output by the adapter 113 as a result of the processing in step 543 in memory 114 (Figure 2).

[0074] Step 522 further includes step 546, in response to step 544, when the output of the adapter 113 is completed and a predetermined unit of audio token sequence is stored in the memory 114, inputting the audio token sequence stored in the memory 114 to the text LLM 124 via the selector 122. Step 522 also includes step 548, which repeatedly executes step 550, in which the tokens output sequentially by the text LLM 124 are sequentially stored in the memory 130 (Figure 2), until a predetermined condition is met. The predetermined condition may be any of the following: when a predetermined number of tokens are stored in the memory 130, when the output of the text LLM 124 is completed, when a predetermined time has elapsed since the token sequence was input to the text LLM 124, or when any combination of these conditions is met.

[0075] Step 522 further includes step 552, which, after step 548 is completed, embeds the contents of memory 130 into an embedding vector. The processing in step 552 corresponds to the function of the embedding unit 200 shown in Figure 4. Step 522 further includes step 554, which searches for a predetermined number of embedding vectors in the vector space to which those vectors belong that are closest to the embedding vectors obtained in step 552, and retrieves them. Step 522 further includes step 556, which reads and outputs text chunks corresponding to the embedding vectors retrieved in step 554 from the built-in text corpus. Note that the embedding vector DB 134 used during inference may be different from the embedding vector DB 134 used during training. In that case, the text corpus built into the embedding vector DB 134 is different from the one used during training. For example, a publicly known text can be used as the text corpus during training, and a text belonging to confidential information can be used during inference.

[0076] The processing in steps 554 and 556 corresponds to the function of the chunk search unit 202 shown in Figure 4. The chunks obtained in this way are considered to be text related to the sequence of speech tokens obtained as a result of the speech recognition in the first stage. In this specification, such text is referred to as the context for the speech recognition result, and the information they represent is referred to as contextual information.

[0077] Step 522 further includes step 558, which tokenizes a predetermined number of chunks extracted in step 556. This process corresponds to the function of the tokenization unit 230 shown in Figure 5.

[0078] Step 522 further includes step 560, which adds a predetermined number of token sequences obtained by the processing in step 558 as a context token sequence 250 to the end of the speech token sequence 140 held in memory (memory 232 in Figure 5). When performing this processing, it is necessary to copy the speech token sequence, which is the output of the adapter 113 and was stored in memory 114, to the memory 232 of the context addition unit 116 prior to this processing. Of course, the context token sequence 250 may be stored in memory 232 first, and then the speech token sequence 140 may be read from memory 114 and added before the output of the speech encoder 112. By concatenating the speech token sequence 140 and the context token sequence 250 in this way, a new token sequence 234 is obtained, consisting of contextual information attached to the output of the speech encoder 112.

[0079] Finally, step 522 includes step 562, which generates a new token sequence by adding a token sequence obtained by tokenizing the prompt sequence read from the prompt storage unit 120 (Figure 2) to the beginning of the new token sequence obtained in step 560, thereby ending the execution of step 522. The processing in step 562 corresponds to the function of the prompt addition unit 118 shown in Figure 6. In particular, the prompt reading unit 280 shown in Figure 6 has the function of reading prompts from the prompt storage unit 120 and tokenizing them. The memory 282 of the prompt addition unit 118 stores the voice token sequence 140 and the context token sequence 250 received from the context addition unit 116. By adding the prompt token sequence 300 output by the prompt reading unit 280 to the beginning of the voice token sequence 140, a token sequence 142, which is a prompt-attached voice recognition token sequence, is obtained and provided to the second input of the selector 122.

[0080] c. Diagram 14 for the second stage of processing shows the details of step 524 in Figure 12. The processing up to step 522 shown in Figure 12 has resulted in a token sequence 142 (see Figure 6), which is a sequence of voice tokens with prompts. Step 524 includes step 580, in which the context addition unit 116 inputs this token sequence 142 to the text LLM 124. At this time, the token sequence output by the text LLM 124 in the previous processing is added to the token sequence 142. Step 524 further includes step 582, in which the token sequence output by the text LLM 124 in response to the input in step 580 is output as a token sequence 132 (see Figure 2) representing the speech recognition result, and step 524 ends. The branching unit 126 shown in Figure 2 outputs the output of the text LLM 124 as the token sequence 132. By converting the token sequence 132 back to text, the speech recognition result for the input is obtained. Furthermore, this token sequence 132 is branched off as token sequence 146 and provided to selector 122 as a third input when processing the next audio.

[0081] 2. Operation A. Overview of Operation The speech recognition system 100 (Figure 2) and the training device 330 (Figure 7) operate as follows. Below, the operation of the speech recognition system 100 and the training device 330 during learning will be explained first, followed by the operation of the speech recognition system 100 during inference.

[0082] B. Learning (Figures 2, 7 to 11) a. Creation of learning data (Figures 7 to 9) Referring to Figure 7, learning preparation data is prepared in learning preparation data 340. The learning data creation unit 342 reads each of the learning preparation data items included in this learning preparation data (step 420 in Figure 9) and tokenizes the associated text (step 422). The learning data creation unit 342 further inputs the audio signals 350 of the learning preparation data items to the speech encoder 112 (step 424). The audio vector sequence output by the speech encoder 112 is input to the adapter 113 (step 425), and the adapter 113 converts the number of dimensions and format of the vector so that it is suitable for input to the text LLM 124, and outputs it as an audio token sequence. After this audio token sequence is stored in memory 114 (step 426), the context addition unit 116 concatenates it with the token sequence of the associated text 352 corresponding to the audio signal 350 for each predetermined unit (e.g., one sentence) (step 428). The token sequence of the related text 352 becomes contextual data for the output of the adapter 113. The prompt addition unit 118 adds a prompt token sequence to the output of the context addition unit 116 (step 430) and stores it in the learning data storage unit 344 as a learning data item (step 432). In this way, when the processing in step 402 is completed for each learning preparation data included in the learning preparation data 340 (when step 400 is completed), the learning data storage unit 344 stores the learning data for the text LLM 124 and the speech encoder 112.

[0083] b. Optimization of speech encoder 112 and adapter 113 (Figure 10) In this embodiment, the speech encoder 112 and adapter 113 are first optimized as follows. The speech encoder / adapter learning unit 348 shown in Figure 7 is optimized by the functional units of the speech recognition system 100 shown in Figure 2 and the program (step 384) whose control structure is shown in Figure 10.

[0084] Referring to Figure 10, the speech encoder adapter learning unit 348 performs the following processing for all learning data items. In step 460 (Figure 10), the speech encoder adapter learning unit 348 receives (reads) the audio signal 110 (Figure 2) included in the target learning data item and inputs that audio signal to the speech encoder 112 (Figure 2) (step 462 in Figure 10). The speech encoder adapter learning unit 348 further inputs the audio vector sequence output by the speech encoder 112 in response to this input to the adapter 113 (step 463). The adapter 113 converts each text in this audio vector sequence into an audio token suitable for input to the text LLM 124 and outputs it as an audio token sequence. This audio token sequence is stored in the memory 114 (Figure 10) (step 464 in Figure 10).

[0085] The speech encoder / adapter learning unit 348 further tokenizes the prompt and associated text contained in the learning data item being processed, and appends them to the beginning and end of the speech token sequence held in memory 114, respectively. This process generates input to text LLM 124. The speech encoder / adapter learning unit 348 inputs the token sequence thus generated to text LLM 124 (step 466 in Figure 10). In response to this input, text LLM 124 outputs the token sequence. The speech encoder / adapter learning unit 348 optimizes the parameters of speech encoder 112 and adapter 113 so that the difference between the text indicated by this token sequence and the associated text contained in the learning data item being processed approaches zero. At this time, the parameters of text LLM 124 are kept fixed.

[0086] The learning of the speech encoder 112 and adapter 113 is completed by repeating this process for all learning data items until a predetermined termination condition is met. Termination conditions can include the completion of a predetermined number of learning cycles, or the absolute value of the total change in the parameters of the speech encoder 112 and adapter 113 falling below a predetermined value.

[0087] c. Optimization of Text LLM124 (Figures 2 and 11) The optimization of Text LLM124 is also performed by having a computer execute the speech recognition system 100 shown in Figure 2 and the program whose control structure is shown in Figure 11.

[0088] Referring to Figure 11, the LLM learning unit 346 (Figure 2) performs step 480, which repeats step 482 for all learning data items. In step 482, the following processing is performed.

[0089] First, in step 490, the LLM learning unit 346 receives (reads out) the audio signal of the learning data item to be processed. The LLM learning unit 346 inputs this audio signal to the speech encoder 112 (Figure 2) (step 492 in Figure 11). The LLM learning unit 346 inputs the audio vector sequence output by the speech encoder 112 in response to this input to the adapter 113 (step 493). The adapter 113 outputs an audio token sequence by converting each vector in this audio vector sequence into an audio token of a predetermined number of dimensions and format. The memory 114 (Figure 2) stores this audio token sequence (step 494).

[0090] The LLM learning unit 346 tokenizes the prompt and associated text included in the learning data item and generates a token sequence, which is the input to the text LLM 124, by adding it to the beginning and end of the token sequence held in memory 114, respectively (step 496). The LLM learning unit 346 inputs the thus generated token sequence to the text LLM 124 (step 496). When the text LLM 124 outputs tokens in order in response to this input, the LLM learning unit 346 holds the token sequence consisting of these tokens in memory. The LLM learning unit 346 updates the parameters of the text LLM 124 so that the difference between the text version of this token sequence and the associated text in the learning data item being processed approaches zero. At this time, the parameters of the speech encoder 112 and the adapter 113 are kept fixed.

[0091] The learning of the text LLM 124 is completed by repeating this process for all learning data items until a predetermined termination condition is met. Various conditions can be used as the termination condition, similar to the termination conditions for learning the speech encoder 112 and adapter 113.

[0092] Once the speech encoder 112, adapter 113, and text LLM 124 have finished learning as described above, the speech recognition system 100 becomes available.

[0093] C. Inference (Figures 2 to 6, Figures 12 to 14) a. Overall flow (Figure 12) The embedding vector DB134 (Figure 2) used in inference does not need to match the embedding vector DB134 used in training. As mentioned above, the embedding vector DB134 may be one created from confidential information or one that incorporates newer information than that used in training.

[0094] Referring to Figure 12, the speech recognition system 100 (Figure 2) receives the audio signal to be processed in step 520. The speech recognition system 100 performs step 522 on this audio signal, converting the audio vector sequence output by the speech encoder 112 (Figure 2) into an audio token sequence using the adapter 113. The speech recognition system 100 further inputs this audio token sequence into the text LLM 124 to obtain the token sequence output by the text LLM 124. By adding a prompt token sequence and a contextual information token sequence obtained from the embedding vector DB 134 based on the token sequence output by the text LLM 124, respectively, before and after this token sequence, a new token sequence, which is the input to the text LLM 124 (Figure 2), is generated. In this process, as described above, the text LLM 124, memory 130, DB access unit 136, context addition unit 116, and prompt addition unit 118 (all in Figure 2) are used. Details of the processing in step 522 will be described later.

[0095] Following step 522, the speech recognition system 100 executes step 524. In step 524, the token sequence obtained in step 522 is input again into the text LLM 124. The token sequence output by the text LLM 124 in response to this input is converted back into text and output as the result of speech recognition of the audio signal received in step 520. Details of the processing in step 524 will be described later.

[0096] b. First Stage (Figure 13) Referring to Figure 13, in the first processing of step 522 (step 542), the speech recognition system 100 (Figure 2) inputs the audio signal received in step 520 to the speech encoder 112 (Figure 2) (step 543). The speech encoder 112 outputs an audio vector sequence for the audio signal. In step 543, this audio vector sequence is input to the adapter 113. The adapter 113 converts this audio vector sequence into an audio token sequence of a predetermined number of dimensions and format and stores it in the memory 114 (step 544). When the audio token sequence stored in the memory 114 reaches a predetermined unit (e.g., a sentence), the stored content is input to the text LLM 124 (step 546). In response to this input, the text LLM 124 sequentially outputs tokens indicating the output text.

[0097] In step 548, step 550 is repeatedly executed, in which the tokens output by the text LLM 124 are stored in the memory 130 (Figure 2) until a predetermined termination condition is met. In this process, the token sequence following the input is output one by one from the text LLM 124 and stored in the memory 130 in order. As described above, when the termination condition is met, step 548 ends, and the contents of the memory 130 are provided to the DB access unit 136 (Figures 2 and 4).

[0098] In step 552, the embedding unit 200 of the DB access unit 136 shown in Figure 4 embeds the contents of the memory 130 into an embedding vector. The embedding unit 200 provides this embedding vector to the chunk search unit 202. The chunk search unit 202 searches the space formed by the embedding vectors in the embedding vector DB 134 and retrieves a predetermined number of embedding vectors that are closest to the vector received from the embedding unit 200 (the vectors with the smallest distance from the vector received from the embedding unit 200) (step 554). The chunk search unit 202 retrieves the text of the chunks corresponding to each of these retrieved embedding vectors from the text corpus built into the embedding vector DB 134 (step 556). The chunk search unit 202 provides the chunks obtained in this way to the context addition unit 116 (Figures 2 and 5).

[0099] The tokenization unit 230 (Figure 5) of the context addition unit 116 (Figures 2 and 5) tokenizes each of the extracted chunks (step 558). The memory 232 of the context addition unit 116 has the contents of memory 114 (the token sequence output by the speech encoder 112) pre-stored therein. The tokenization unit 230 appends the token sequence tokenized in step 558 to the end of the speech token sequence 140 stored inside memory 232 (step 560). The speech token sequence 140 and the context token sequence 250 form a new token sequence 234. The token sequence 234 is provided from the context addition unit 116 to the prompt addition unit 118 shown in Figure 6.

[0100] Referring to Figure 6, the token sequence 234 from the context addition unit 116 is stored in the memory 282 inside the prompt addition unit 118. The prompt reading unit 280 of the prompt addition unit 118 reads the prompt from the prompt storage unit 120 and tokenizes it. The prompt reading unit 280 adds the prompt token sequence 300, consisting of the tokenized prompt, to the beginning of the voice token sequence 140 (step 562). The prompt token sequence 300, the voice token sequence 140, and the context token sequence 250 form a new token sequence 284. The prompt addition unit 118 provides this token sequence 284 as token sequence 142 to the second input of the selector 122 (Figure 2).

[0101] c. Second Stage (Figure 14) In the second stage, the selector 122 selects the token sequence 142 in the second input and inputs it to the text LLM 124 (step 560 in Figure 11). In response to this input, the text LLM 124 generates and outputs a token sequence following the input. The branching section 126 in Figure 2 outputs the token sequence generated a second time by the text LLM 124 to the outside of the speech recognition system 100. By converting this token sequence back into text, the speech recognition result for the speech signal 110 is obtained. This token sequence is then added to the next input to the text LLM 124 during the next speech recognition and input to the text LLM 124 again.

[0102] 3. Effects As described above, according to this embodiment, the speech recognition result for the audio signal 110 reflects the information obtained from the embedding vector DB 134 as context. For example, if a DB 134 containing confidential information is used as the embedding vector DB 134, a speech recognition result that reflects the confidential information can be obtained compared to when only embedding vectors obtained from publicly known data are used. This increases the likelihood of correctly performing speech recognition on audio containing confidential information in company meetings, etc. Furthermore, if a DB 134 containing the latest information is used as the embedding vector DB 134, new information that was not available when training the speech encoder 112 and text LLM 124 can also be used as contextual information for speech recognition. As a result, the likelihood of obtaining a speech recognition result that reflects the latest information increases. Including these factors, by using a DB 134 containing information that was not used in training the speech encoder 112, adapter 113, and text LLM 124, speech recognition can be performed with high accuracy even on information that was not included in the training data, i.e., out-of-domain information.

[0103] Second Embodiment (Figure 15) 1. Configuration In the first embodiment described above, the transcription of the audio signal is used as the associated text for the learning data items. However, this invention is not limited to such embodiments. In addition to, or instead of, the transcription of the audio signal may be embedded into vectors, and a predetermined number of embedding vectors close to that embedding vector may be extracted from the embedding vector DB 134. For each of these embedding vectors, the corresponding chunk may be added to the associated text and used as a learning data item. In that case, the entire text may be used as the associated text, or separate learning data may be created using each chunk.

[0104] Furthermore, related data may be created by combining multiple related texts in various ways. This second embodiment is an example of creating separate training data using various combinations of these multiple related texts. The learning process and inference process by the speech recognition system 100 are the same as in the first embodiment. The training preparation data items used are also the same as those in the first embodiment.

[0105] Referring to Figure 15, the program 650 for realizing the learning data according to this second embodiment includes a step 670 which repeats the process of executing step 672 for each learning preparation data item.

[0106] Referring to Figure 15, step 672 includes steps 420, 422, 424, and 426, similar to those shown in Figure 9. Step 672 further includes step 700, which creates a combination of related texts and performs step 702 for each of the combinations.

[0107] Step 702 includes step 710, which adds a token sequence obtained from the combination of related text being processed to the end of the speech token sequence output by the adapter 113 by converting the output of the speech encoder 112. Step 702 further includes step 712, which adds a prompt token sequence to the beginning of the token sequence obtained in step 710, and step 714, which stores the token sequence thus obtained as a learning data item in the learning data storage unit 344 (Figure 7) and terminates step 702.

[0108] According to this second embodiment, a variety of training data items can be created using various combinations of related data. By training the speech encoder 112, adapter 113, and text LLM 124 using these training data items, the likelihood of appropriately reflecting contextual data in the speech recognition result increases, regardless of the type of embedding vector DB 134 used during inference. As a result, it becomes possible to perform speech recognition with high accuracy even for out-of-domain information.

[0109] Third Embodiment 1. Configuration In the first embodiment described above, the transcription included in the training preparation data is used as the relevant text when training the speech encoder 112, adapter 113, and text LLM 124. However, as described in the description of the second embodiment above, the relevant text can also be extracted from the embedded vector DB. This third embodiment is an embodiment relating to the creation of such training data. The operation of the speech encoder 112, adapter 113, and text LLM 124 during training and inference is the same as in the first embodiment.

[0110] A. Training device 750 (Figure 16) a. Configuration (Figure 16) Figure 16 shows a schematic configuration of a training device 750 for training the speech encoder 112, adapter 113 and text LLM 124 in this third embodiment, in block diagram form. Referring to Figure 16, this training device 750 includes a learning data creation unit 770 for creating learning data for the speech encoder 112, adapter 113 and text LLM 124 from learning preparation data 340 similar to that of the first embodiment and chunks of text searchable in the embedded vector DB 752, and a learning data storage unit 344 for storing the learning data created by the learning data creation unit 770.

[0111] The training device 750 further includes an LLM learning unit 346 for training the text LLM 124 using the learning data stored in the learning data storage unit 344, and a speech encoder / adapter learning unit 348 for training the speech encoder 112 and adapter 113 using the learning data stored in the learning data storage unit 344.

[0112] The configuration of the embedded vector DB752 used in this embodiment is the same as that of the embedded vector DB134 shown in the first embodiment. However, its contents do not need to be the same as those shown in the first embodiment.

[0113] The learning data creation unit 770 includes a speech encoder 112, an adapter 113, a memory 114, a context addition unit 116, a prompt addition unit 118, and a prompt storage unit 120, similar to those included in the learning data creation unit 342 shown in Figure 7. The learning data creation unit 770 further includes a DB access unit 780. The DB access unit 780 accesses the embedding vector DB 752 using the related text 352. The DB access unit 780 thereby retrieves a predetermined number of chunks of text close to the related text 352 from the embedding vector DB 752 (i.e., chunks of text corresponding to a predetermined number of higher-level embedding vectors close to the embedding vector of the related text 352) and provides them to the context addition unit 116.

[0114] Based on the related text 352, multiple related texts can be retrieved from the embedding vector DB 752. As a result, the number of training data items included in the training data can be increased. The operation of the speech recognition system 100 during training and inference is the same as in the first embodiment.

[0115] b. Program for creating training data (Figure 17) Figure 17 shows the control structure of the program for creating training data in this third embodiment.

[0116] Referring to Figure 17, this program includes step 800, which performs the processing in step 802 for each learning preparation data item.

[0117] Step 802 includes step 820 of reading a learning preparation data item, and step 822 of tokenizing the relevant text in the learning preparation data item read in step 820. Step 802 further includes step 824 of inputting the audio signal of the learning preparation data item being processed to the speech encoder 112 (Figure 2), step 825 of inputting the audio vector sequence output by the speech encoder 112 in response to this input to the adapter 113, and step 826 of storing the audio token sequence output by the adapter 113 as a result of the processing in step 825 into memory.

[0118] Step 802 further includes, following step 826, step 828, which searches the embedding vector DB 134 using the relevant data in the learning preparation data item being processed as a key, and receives a predetermined number of top-level embedding vectors from the embedding vector DB 134 that are closest to the embedding vector of the relevant data in the vector space. Step 802 further includes step 830, which retrieves chunks of text corresponding to the predetermined number of embedding vectors received in step 828 from the text corpus contained within the embedding vector DB 134.

[0119] Step 802 further includes step 832, which tokenizes each of the text chunks extracted in step 830, and step 834, which appends the token sequence obtained in step 832 to the end of the audio token sequence output by the adapter 113, which was stored in memory in step 826. Step 802 further includes step 836, which appends a prompt token sequence to the beginning of the token sequence obtained in step 834 (a concatenation of the output of the adapter 113 and the token sequence obtained from the text chunks).

[0120] Finally, step 826 includes step 838, which stores the token sequence obtained in step 836 as a learning data item in the learning data storage unit 344 (Figure 2) and terminates step 802.

[0121] 2. Operation (Figures 16, 17) a. Preparation of training data (Figures 16, 17) In preparing the training data, first, training preparation data 340 is prepared. The training preparation data 340 includes multiple training data items. Each training data item includes an audio signal 350 and related text 352 associated with the utterance represented by the audio signal 350. In this example, the related text 352 is a transcription of the audio signal 350.

[0122] During training, learning preparation data items are read sequentially from the learning preparation data 340. The audio signal 350 of the learning preparation data item to be processed is input to the speech encoder 112. The speech encoder 112 performs speech recognition processing on this audio signal 350 and outputs an audio vector sequence representing the result. This audio vector sequence is input to the adapter 113. The adapter 113 converts each vector included in the audio vector into an audio token of a predetermined number of dimensions and format and outputs it as an audio token sequence. This audio token sequence is stored in the memory 114.

[0123] Meanwhile, the related text 352 is provided to the DB access unit 780. The DB access unit 780 converts this related text into an embedding vector and accesses the embedding vector DB 752 using this embedding vector as a key. The embedding vector DB 752 searches for a predetermined number of top-level embedding vectors with small distances from the key embedding vector in the vector space of embedding vectors stored in the embedding vector DB 752. The embedding vector DB 752 then reads chunks of text corresponding to these predetermined number of embedding vectors from the text corpus built into the embedding vector DB 752 and returns them to the DB access unit 780. The DB access unit 780 provides these chunks to the context addition unit 116.

[0124] The context addition unit 116 tokenizes and concatenates each chunk in the sequence received from the DB access unit 780, and appends it to the end of the token sequence stored in memory 114. The context addition unit 116 provides the resulting token sequence to the prompt addition unit 118.

[0125] The prompt addition unit 118 tokenizes the prompts stored in the prompt storage unit 120 and adds them to the beginning of the token sequence received from the context addition unit 116. The prompt addition unit 118 stores the token sequence thus obtained as a learning data item in the learning data storage unit 344.

[0126] Once the above processing is completed for all learning preparation data items stored in the learning preparation data 340, the learning data storage unit 344 will have the learning data for the text LLM 124 and speech encoder 112 ready.

[0127] b. Learning and Inference The training method for the speech encoder 112, adapter 113, and text LLM 124 performed by the LLM learning unit 346 and the speech encoder / adapter learning unit 348 using the learning data prepared by the method according to this third embodiment is the same as the training method according to the first embodiment. Furthermore, the operation of the speech recognition system during inference using the speech encoder 112, adapter 113, and text LLM 124 obtained in this way is also the same as the operation of the speech recognition system during inference according to the first embodiment. Therefore, a detailed explanation of these will not be repeated here.

[0128] Section 4 Experimental Results An experiment was conducted on the performance of the speech recognition system according to the first embodiment described above, under the following conditions.

[0129] The experiment used the Corpus of Spontaneous Japanese (CSJ) (National Institute for Japanese Language and Linguistics (2006) "Method for Constructing a Corpus of Spontaneous Japanese" National Institute for Japanese Language and Linguistics. https: / / doi.org / 10.15084 / 00001357). The CSJ database is "a database for spoken language research that collects a large amount of spontaneous Japanese speech and adds a lot of research information, and was jointly developed by the National Institute for Japanese Language and Linguistics, the National Institute of Information and Communications Technology (formerly the Communications Research Laboratory), and Tokyo Institute of Technology, and is a spoken language database of the highest quality and quantity in the world" (excerpted from the description at https: / / clrd.ninjal.ac.jp / csj / ).

[0130] The CSJ (Chemical Science Journal) contains 660 hours of spontaneous audio recordings of lectures, primarily academic in nature. These lectures typically range in length from 10 to 25 minutes. A key feature of the CSJ is its inclusion of domain-specific texts across various fields, including science, engineering, humanities, and social sciences. Therefore, the CSJ data allows for testing using both in-domain and out-of-domain data.

[0131] For training data, we used 610 hours of lecture data obtained by removing lectures that mentioned "speech recognition," "language models," and "acoustic models" from the lecture data included in CSJ. From the removed lecture data, 20 lectures were selected as in-domain test data. In this selection, the domain of the test data was not included in the training data. In addition to this test data, officially prepared test data was used as a reference for the baseline of the speech recognition system according to this embodiment.

[0132] In the experiment, speech data was clipped to 30 seconds or less to mitigate the impact of long texts. The Character Error Rate (CER) was used as the evaluation metric. A lower CER indicates higher accuracy in speech recognition.

[0133] As described above, the first embodiment includes a first stage for training the speech encoder 112 and a second stage for training the text LLM 124. In the experiment, the first stage of training was performed for 10 epochs, and the second stage of training was performed for 20 epochs. Furthermore, for comparison, a model in which only the first stage of training was performed for 10 epochs and a model in which only the first stage of training was performed for 30 epochs were also prepared.

[0134] The parameters of the speech encoder 112 were initialized using the Whisper-large-v2 model (Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023). The text LLM 124 was initialized using the ELYZA-Japanese-Llama-2-7b model (Akira Sasaki, Masato Hirakawa, Shintaro Horie, and Tomoaki Nakamura, “ELYZA-japanese-Llama-2-7b,” 2023). The Whisper-large-v2 model is a 32-layer transformer architecture including two convolutional downsampling layers. The ELYZA-Japanese-Llama-2-7b model is a 32-layer transformer decoder with a hidden size of 4096 and a total of 7 billion parameters. The adapter 113 consists of a one-dimensional average pooling layer and a fully connected layer. The average pooling layer compresses the input features by half in the time direction. The fully connected layer projects the pooled features onto the target dimension (from 1280 dimensions to 4096 dimensions).

[0135] Furthermore, we investigated the results of searching for related data using only a portion of the output of text LLM124 during inference (for example, only the first 30 characters or only the first 100 characters) and when searching for related data using the entire output of text LLM124. Cosine distance was used as the distance between embedding vectors during the search in embedding vector DB134. The number of chunks extracted from embedding vector DB134 was set to 2. However, chunks belonging to the same utterance as the utterance being targeted for speech recognition were excluded.

[0136] The models were trained using the Adam optimizer, global mini-batch size = 128, and initial learning rate 5e-5, with a warm restart learning rate scheduler adjusted by a cosine annealing schedule (Ilya Loshchilov and Frank Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. ICLR, 2017.) for both the speech encoder 112 and the text LLM 124. The results are shown in Figures 18 and 19.

[0137] Figure 18 compares the results obtained using the speech recognition system according to the present invention with those obtained using the prior art speech recognition system.

[0138] The first and second rows of Figure 18 show the results obtained using the techniques described in Reference 1 (Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai, “ESPnet: End-to-end speech processing toolkit,” in Proc. Interspeech, 2018).

[0139] The third to fifth rows of Figure 18 show the results using the technique described in Reference 2 (Yukiya Hono, Koh Mitsuda, Tianyu Zhao, Kentaro Mitsui, Toshiaki Wakatsuki, and Kei Sawada, “Integrating pre-trained speech and language models for end-to-end speech recognition,” in Findings of ACL, 2024).

[0140] The sixth row of Figure 18 shows the results of the voice recognition system in the present invention after performing only the first stage of training for 10 epochs. The seventh row shows the results of the same invention after performing only the first stage of training for 30 epochs. The eighth row shows the results of the same invention after performing the first stage of training for 10 epochs and then the second stage for 20 epochs.

[0141] Referring to Figure 18, for example, the models shown in the third and fourth rows have high CERs even on in-domain test data, despite being trained on a very large amount of training data (more than 5 million hours in the case of v3 in the fourth row). On the other hand, the speech recognition system according to the present invention shows performance equivalent to (seventh and eighth rows) or better than (ninth row) the results of the prior art.

[0142] Regarding out-of-domain test data, a direct comparison with the present invention is not possible because results are not available for the prior art. However, the accuracy of the results for out-of-domain data should generally be lower than that for in-domain data.

[0143] In the present invention, as shown in line 6, if only the speech encoder 112 is trained, the accuracy for out-of-domain data decreases. On the other hand, in the case of line 7, the accuracy for out-of-domain data is equivalent to the accuracy for in-domain data. Therefore, in this respect, it can be seen that when inference is performed in two stages as in the present invention, the recognition accuracy for out-of-domain data can be increased compared to the prior art.

[0144] Furthermore, when training is divided into two stages, and inference is also divided into two stages, as shown in the eighth line, the accuracy on out-of-domain test data is higher than the accuracy on in-domain test data (the CER value is smaller). Therefore, from these experimental results, it can be seen that, according to the present invention, a speech recognition system capable of performing speech recognition with high accuracy even on out-of-domain data can be obtained.

[0145] Figure 19 shows the results of an experiment conducted to investigate whether using prompts in input to the text LLM124 contributes to improving speech recognition accuracy. In Figure 19, the leftmost column indicates whether prompts were used or not. Columns 2 through 6 show what kind of related data was used. Column 7 shows the CER obtained for the test data based on the combination of whether or not instructions were used (as shown in column 1) and the type of related data (as shown in row 1).

[0146] In the first column, a checkmark indicates when a prompt was used. A blank space indicates that a prompt was not used.

[0147] In the second column, "D:Rand" indicates that a randomly selected text chunk was used as the related data. In the third column, "D:30" indicates that the first 30 characters of the output of text LLM124 were used for the search in the embedding vector DB134. In the fourth column, "D:100" indicates that the first 100 characters were used, and in the fifth column, "D:Full" indicates that the entire output of text LLM124 was used. In the sixth column, "D:Trans" indicates that the transcription accompanying the audio data was used as the related data.

[0148] In this table, the difference between using and not using prompts can be seen in the fourth column, "D:100," and the fifth column, "D:Full." In the case of "D:100," the CER was 3.97 when prompts were not used, while it became 3.82 when prompts were used. In other words, speech recognition accuracy improved. Also in the case of "D:100," the CER was 3.86 when prompts were not used, while it decreased to 3.73 when prompts were used. In this case as well, using prompts improved speech recognition accuracy.

[0149] From the above results, it was confirmed that, as in the present invention, by performing two stages of training and processing related data as context to improve the speech recognition results, the accuracy for out-of-domain data is improved, and that the accuracy for out-of-domain data is further improved by providing prompts when providing related data as context to the text LLM124. The prompts used in this experiment are the same as those already described.

[0150] Section 5 Implementation by Computer (Figures 20 and 21) Figure 20 shows the appearance of a computer that runs a computer program to implement the speech recognition system and its training method according to the above embodiment, and Figure 21 shows an example of its hardware configuration. Figures 20 and 21 are merely examples, and the above speech recognition system and its training method can be implemented using a computer different from those shown in Figures 20 and 21.

[0151] Referring to Figure 20, this computer system 1030 includes a computer 1050 and a keyboard 1054, a mouse 1056, and a monitor 1052, all connected to the computer 1050, for user interaction. These are just one example of a configuration for when user interaction is required, and any general hardware and software available for user interaction (e.g., touch panels, voice input, pointing devices in general) can be used. When operating the computer 1050 remotely, it is not necessary for the computer 1050 to be connected to such hardware.

[0152] Referring to Figure 21, the computer 1050 includes a bus 1110 and, all connected to the bus 1110, a CPU (Central Processing Unit) 1100, a GPU (Graphics Processing Unit) 1102, a ROM (Read-Only Memory) 1104 for storing the computer 1050's boot-up program, etc., a RAM (Random Access Memory) 1106 for storing instructions, system programs, and work data that constitute the program when the program is executed, and an SSD (Solid State Drive) 1108, which is a non-volatile memory connected to the bus 1110. SSD 1108 is for storing programs executed by CPU 1100 and GPU 1102, as well as data used by programs executed by CPU 1100 and GPU 1102. These programs include programs for implementing the speech recognition system described above. Any of the functions shown in the above embodiment, such as the speech encoder 112, adapter 113, and text LLM 124, are realized by computer hardware and programs executed on it.

[0153] The computer 1050 further includes a network interface 1118 that provides connectivity to the network 1060, and a USB port 1116 that allows for the insertion and removal of a USB memory 1058 and provides communication between the USB memory 1058 and various parts within the computer 1050. In this embodiment, the network 1060 is the internet.

[0154] The computer 1050 further includes an input / output interface 1114 connected to the bus 1110 for inputting input from the keyboard 1054 and mouse 1056 to the CPU 1100, and a display control unit 1112 connected to the bus 1110 for controlling the display on the monitor 1052 according to the control from the CPU 1100.

[0155] The computer program for operating the computer system 1030 to perform each of the functions of the voice recognition system and each step of its training method described above is stored, for example, in a USB memory 1058. The USB memory 1058 is plugged into the USB port 1116, and the program is transferred to the SSD 1108. Alternatively, the program may be transmitted to the computer 1050 via the network 1060 and stored in the SSD 1108. The program is loaded into the RAM 1106 when executed. Part of the program may be stored in the ROM 1104.

[0156] The CPU 1100 reads the program from RAM 1106 according to the address indicated by an internal register called the program counter (not shown), and interprets each instruction contained in the program. The CPU 1100 further reads the data necessary for executing the instruction from RAM 1106, SSD 1108, or other device according to the address specified by the instruction, and executes the processing specified by the instruction. The CPU 1100 stores the execution result data at an address specified by the program, such as RAM 1106, SSD 1108, or a register within the CPU 1100. The value of the program counter is also updated by the program. The computer program may be loaded directly into RAM 1106 from USB memory 1058 or via a network. Among the programs executed by the CPU 1100, tasks that can be executed in parallel among the iterative processes can be executed by the GPU 1102 according to the instructions contained in the program or according to the analysis results when the CPU 1100 executes the instructions, making it possible to output the results in a short time according to the speech recognition system described above.

[0157] The program that implements each function of the speech recognition system and the training method for the speech recognition system in cooperation with the computer 1050 includes a plurality of instructions that are written and arranged to cause the computer 1050 to operate in order to implement those functions. The operation method of the computer 1050 that executes these instructions is well known. Therefore, the explanation of that operation method will not be repeated here.

[0158] 6. Effects As described above, in each embodiment of this invention, in the first stage for the audio signal, a token sequence obtained as a result of speech recognition by the text LLM is obtained with contextual information obtained from the embedding vector DB added. By adding this contextual information to the speech recognition result, the accuracy of generating the speech recognition result in the second stage by the text LLM is improved. Furthermore, when using the text LLM, by providing a prompt instructing it to generate a speech recognition string from the output of the speech encoder, referring to the contextual information, it can be expected that the output of the text LLM will better reflect the contextual information. The embedding vector DB used during training and the embedding vector DB 134 used during inference do not need to be the same. Therefore, the speech encoder 112 and text LLM 124 can be trained using a wide range of existing information.

[0159] For example, using an embedding vector database containing confidential information during inference allows for speech recognition results that reflect the confidential information, compared to using only embedding vectors obtained from publicly available data during inference. This increases the likelihood of correctly recognizing participants' voices according to the content of the confidential information, such as in meetings where confidential information is handled. Furthermore, using an embedding vector database containing the latest information during inference allows for the use of new information that was not available when training the speech encoder and text LLM as contextual information for speech recognition. As a result, the likelihood of obtaining speech recognition results that reflect the latest information increases. In short, by using an embedding vector database containing information not used in training the speech encoder and text LLM during inference, it becomes possible to perform speech recognition with high accuracy even for information not included in the training data, i.e., out-of-domain information.

[0160] The embodiments disclosed herein are illustrative and not limited to those embodiments. The scope of the present invention is defined by each claim, with reference to the detailed description of the invention, and includes all modifications within the meaning and scope equivalent to the wording contained herein.

[0161] 50, 100 Speech Recognition System 60 Training Data 62, 110, 350 Speech Signal 64 Transcript 66, 112 Speech Encoder 68, 113 Adapter 70, 114, 130, 232, 282 Memory 72 Related Text Creation Unit 74, 124 Text LLM 76 Inference Result 116 Context Addition Unit 118 Prompt Addition Unit 120 Prompt Storage Unit 128 Control Unit 132, 142, 144, 146, 234, 284 Token Sequence 134, 752 Embedding Vector DB 136, 780 DB Access Unit 138, 140 Speech Token Sequence 150 DB Construction Unit 160 Text Corpus 162 Chunking Unit 164, 200 Embedding Unit 202 Chunk Search Unit 230 Tokenization unit 250 Context token sequence 280 Prompt reading unit 300 Prompt token sequence 330, 750 Training device 340 Learning preparation data 342, 770 Learning data creation unit 344 Learning data storage unit 346 LLM learning unit 348 Speech encoder / adapter learning unit 352 Related text 1030 Computer system 1050 Computer

Claims

1. A speech recognition system comprising: a speech token sequence output unit that receives a speech signal, performs speech recognition on the speech signal and converts it into a speech token sequence; a large-scale language model trained to generate an output having a predetermined relationship with an input; a database access unit that accesses a predetermined database and retrieves related data having a predetermined relationship with the output of the large-scale language model; and an input extension unit that expands the input to the large-scale language model and re-inputs it to the large-scale language model using the output of the speech token sequence output unit, the output of the large-scale language model that receives the output from the speech token sequence output unit as input, and the related data retrieved by the database access unit that receives the output of the large-scale language model, wherein the output of the large-scale language model to the re-input becomes the speech recognition result for the speech signal.

2. The speech recognition system according to claim 1, wherein the input extension unit includes a related data addition unit that adds the related data to the end of the output of the speech token sequence output unit.

3. The speech recognition system according to claim 2, wherein the input extension unit further includes a prompt addition unit that adds a prompt for the large-scale language model to the beginning of the output of the speech token sequence output unit.

4. The speech recognition system according to claim 3, wherein the prompt instructs the large-scale language model to modify the output of the speech token sequence output unit included in the extended input with reference to the associated data.

5. A computer program that causes a computer to function as an input extension unit that extends the input to the large language model and re-inputs it to the large language model, using the output of the voice token sequence output unit, the output of the large language model that has received the output of the voice token sequence output unit, and the related data output by the database access unit that has received the output of the large language model, wherein the output of the large language model to the re-input is the result of voice recognition for the voice signal.

6. A speech recognition method comprising the steps of: a computer receiving an audio signal, performing speech recognition on the audio signal and converting it into a sequence of speech tokens; the computer inputting the sequence of speech tokens into a large-scale language model trained to generate outputs having a predetermined relationship with an input; the computer accessing a predetermined database and retrieving related data having a predetermined relationship with the output of the large-scale language model; the computer expanding the input to the large-scale language model using the sequence of speech tokens, the output of the large-scale language model, and the related data, and re-inputting the input to the large-scale language model; and outputting the output of the large-scale language model for the re-input as a speech recognition result for the audio signal.

Images