Speech recognition device, speech recognition method, and speech recognition program
The speech recognition device addresses computational challenges by dividing acoustic features into chunks and using a latency-controlled selective state-space model to achieve efficient and accurate speech recognition.
Patent Information
- Application Number
- PCT/JP2025/016750
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-12
AI Technical Summary
Conventional speech recognition techniques face challenges in achieving high-performance and high-speed recognition, particularly in streaming processing, due to computational complexity issues with Transformer models, difficulty in operating in chunk units with selective state-space models, and the need for frame-by-frame prediction of output sequences.
A speech recognition device that divides acoustic feature sequences into chunks, using a machine learning model to estimate symbol sequences including blanks, and employs a latency-controlled selective state-space model to operate in chunk units, updating latent states in chronological and reverse order to optimize recognition processing.
Enables high-performance and high-speed speech recognition by reducing computational complexity and improving accuracy through chunk-based processing, without the need for complex path search configurations.
Smart Images

Figure JP2025016750_12112026_PF_FP_ABST
Abstract
Description
Speech recognition device, speech recognition method, and speech recognition program
[0001] This invention relates to a speech recognition device, a speech recognition method, and a speech recognition program.
[0002] Recent speech recognition systems using neural networks can directly output a sequence of speech features (phonemes, characters, subwords, and words). Such speech recognition systems are called end-to-end speech recognition systems.
[0003] For example, Non-Patent Document 1 describes a Recurrent Neural Network Transducer (RNN-T) as a learning method for an End-to-End speech recognition system.
[0004] RNN-T introduces a "blank" symbol (described as "null output" or "φ" in Non-Patent Literature 1) to represent redundancy. Therefore, with RNN-T, if only the output sequence corresponding to the speech content is provided, it becomes possible to dynamically learn the correspondence between speech and the output sequence from the training data. In this case, the output sequence does not need to be frame-by-frame with respect to the speech. That is, the input length T and the output length U may not correspond (generally T >> U). However, RNN-T can operate frame-by-frame during inference.
[0005] Furthermore, a technique is known that modifies the Transformer used in LLMs (Large Language Models) for streaming processing (see Non-Patent Document 2). In addition, for streaming processing, a selective state-space model (for example, Mamba described in Non-Patent Document 3) is sometimes used as a method that has the potential to operate faster and with higher performance than the Transformer.
[0006] Also, there is known a technique called HAT (Hybrid Autoregressive Transducer), which is an advancement of the RNN-T described in Non-Patent Document 1 and models the blank symbol and the symbols of the output sequence with separate distributions (see, for example, Non-Patent Document 4).
[0007] Alex Graves, "Sequence Transduction with Recurrent Neural Networks," in Proc. of ICML, 2012. Xie Chen et al., "DEVELOPING REAL-TIME STREAMING TRANSFORMER TRANSDUCER FOR SPEECH RECOGNITION ON LARGE-SCALE DATASET," in Proc. of ICASSP, 2021. Albert Gu and Tri Dao., "Mamba: Linear-Time Sequence Modeling with Selective State Spaces," arXiv, 2024. Yu Zhang et al., "HIGHWAY LONG SHORT-TERM MEMORY RNNS FOR DISTANT SPEECH RECOGNITION," in Proc. of ICASSP 2016 Ehsan Variani, et al., "Hybrid Autoregressive Transducer (HAT)," in Proc. of ICASSP, 2020.
[0008] However, the conventional techniques have a problem that high-performance and high-speed speech recognition may not be achieved in some cases.
[0009] For example, the Transformer used in the technique described in Non-Patent Document 2 has a computational complexity that increases on an exponential order as the length of the input speech increases.
[0010] A selective state-space model such as Mamba described in Non-Patent Document 3 always has a constant computational complexity relative to the input sequence length during inference. On the other hand, in speech recognition that operates in a streaming manner, Transformer can operate in units of chunks, but it is difficult for selective state-space models to operate in units of chunks. In streaming speech recognition, operation in chunk units is common, and operation in chunk units is desirable because it enables utilization of information within chunks to achieve high accuracy.
[0011] Furthermore, RNN-T described in Non-Patent Document 1 models blank symbols and symbols of output sequences (phonemes, characters, subwords, words) with a single probability distribution, and it is necessary to perform prediction of output sequence symbols (calculation of output probability sequences) every frame, which reduces the speed of recognition processing.
[0012] To solve the aforementioned problems, a speech recognition apparatus of the present invention updates a first latent state in chronological order for each of a plurality of times based on an input feature sequence, which is a sequence of input feature values at each of the plurality of times of speech to be recognized and is divided into a plurality of chunks, and a forward feature calculation unit that calculates a first feature value based on the first latent state each time the first latent state is updated; based on the input feature sequence, updates a second latent state in the reverse order of chronological order for each of the plurality of times for each of the chunks, and a backward feature calculation unit that calculates a second feature value based on the second latent state each time the second latent state is updated; and a delay control unit that initializes the second latent state each time the calculation of the second feature value for each of the chunks is completed.
[0013] According to the present invention, high-performance and high-speed speech recognition can be performed.
[0014] Figure 1 is a diagram showing an example configuration of the speech recognition device according to the first embodiment. Figure 2 is a flowchart showing the flow of the speech recognition process according to the first embodiment. Figure 3 is a diagram illustrating the speech recognition process. Figure 4 is a diagram showing an example of the speech recognition algorithm. Figure 5 is a diagram showing the experimental results of the first embodiment. Figure 6 is a diagram showing an example configuration of the learning device according to the first embodiment. Figure 7 is a diagram showing an example configuration of the intermediate input feature calculation unit according to the second embodiment. Figure 8 is a diagram illustrating the method for calculating intermediate input features. Figure 9 is a flowchart showing the processing flow for calculating intermediate input features according to the second embodiment. Figure 10 is a diagram showing the experimental results of the second embodiment. Figure 11 is a diagram showing an example configuration of a computer that executes the program.
[0015] The embodiments for carrying out the present invention will be described below with reference to the drawings. The present invention is not limited to these embodiments.
[0016] [First Embodiment] [Configuration of the First Embodiment] The configuration of the speech recognition device of the first embodiment will be explained using Figure 1. Figure 1 is a diagram showing an example of the configuration of the speech recognition device of the first embodiment. As shown in Figure 1, the speech recognition device 10 receives an input of an acoustic feature sequence and outputs a recognition hypothesis.
[0017] An acoustic feature sequence is a sequence whose elements are the acoustic features of the speech to be recognized at each time step. For example, acoustic features include MFCC (Mel frequency cepstral coefficient), log Mel filterbank (FBANK: log Mel filterbank coefficients), ΔMFCC (first derivative of MFCC), ΔΔMFCC (second derivative of MFCC), log power, Δlog power (first derivative of log power), etc., calculated from the audio signal obtained by recording.
[0018] The recognition hypothesis is data used to identify the symbol sequence corresponding to the acoustic feature sequence. For example, the recognition hypothesis includes the probability distribution and score (e.g., likelihood) of each symbol at each time step as elements. That is, the speech recognition device 10 obtains the symbol sequence corresponding to the speech. The symbols can be phonemes, letters, subwords, words, etc. Also, the symbols may be blank.
[0019] The speech recognition device 10 also accepts input for chunk size and beam size. Chunk size and beam size are parameters used in the processing described later. The speech recognition device 10 may store the chunk size and beam size in advance.
[0020] As shown in Figure 1, the speech recognition device 10 has a division unit 11, an estimation unit 12, and a hypothesis generation unit 13. The division unit 11 divides the acoustic feature sequence into multiple chunks. The estimation unit 12 estimates the probability distribution of each symbol at each time step for each divided chunk, i.e., the output probability distribution. The hypothesis generation unit 13 generates a recognition hypothesis based on the output probability distribution. The hypothesis generation unit 13 outputs the generated recognition hypothesis. Note that estimation for each chunk may be rephrased as estimation on a chunk-by-chunk basis.
[0021] The estimation unit 12 includes an intermediate input feature calculation unit 121, an intermediate symbol feature calculation unit 122, and a probability distribution calculation unit 123. The estimation unit 12 also stores first model information 151, second model information 152, and third model information 153.
[0022] The intermediate input feature calculation unit 121, the intermediate symbol feature calculation unit 122, and the probability distribution calculation unit 123 perform calculations using a machine learning model. The first model information 151, the second model information 152, and the third model information 153 are parameters of the machine learning model used in the intermediate input feature calculation unit 121, the intermediate symbol feature calculation unit 122, and the probability distribution calculation unit 123, respectively.
[0023] The intermediate input feature calculation unit 121 calculates intermediate input features based on acoustic features. The intermediate input feature calculation unit 121 calculates intermediate input features using a multi-stage neural network constructed based on the first model information 151. For example, the intermediate input feature calculation unit 121 obtains intermediate input features by encoding acoustic features.
[0024] The calculation performed by the intermediate input feature calculation unit 121 is expressed as shown in equation (1).
[0025]
[0026] The left-hand side of equation (1) is the intermediate input feature. enc (•) represents the parameter θ of the first model information 151. enc This is a neural network based on x. t is an acoustic feature. t is the number (time) that identifies an element in the acoustic feature sequence of the acoustic feature. Let T be the number of elements in the acoustic feature sequence.
[0027] The intermediate symbol feature calculation unit 122 calculates intermediate symbol features based on each symbol in the symbol sequence. The intermediate symbol feature calculation unit 122 converts the symbols into one-hot vectors. Then, the intermediate symbol feature calculation unit 122 calculates continuous intermediate symbol features based on the one-hot vectors. The intermediate symbol feature calculation unit 122 performs the calculation using a multi-stage neural network constructed based on the second model information 152. For example, the intermediate symbol feature calculation unit 122 obtains intermediate symbol features by encoding the one-hot vectors.
[0028] The calculation performed by the intermediate symbol feature calculation unit 122 is expressed as shown in equation (2).
[0029]
[0030] The left-hand side of equation (2) is the intermediate symbol feature. pred (•) represents the parameter θ of the second model information 152. pred This is a neural network based on y. u-1is a symbol. A symbol is represented using K tokens, excluding blanks. K is the number of symbols prepared in advance as candidates. u is the number that identifies the element in the symbol sequence corresponding to the final sound obtained. However, the number of elements U in the symbol sequence may be unknown here. The left side of equation (2) can be said to be the information for predicting the u-th symbol.
[0031] The probability distribution calculation unit 123 calculates the probability distribution based on the intermediate input features and intermediate symbol features. The probability distribution may be calculated separately for the blank and the symbols, or a two-headed model called HAT, which is a variation of RNN-T as shown in Non-Patent Document 5, may be used. The probability distribution calculation using HAT is described below.
[0032] The probability distribution calculation unit 123 calculates the probability that the u-th symbol is blank using equation (3), based on the intermediate input features at time t and the u-th intermediate symbol features.
[0033]
[0034] Sigmoid(•) is the sigmoid function. fjoint-blank(•) is a neural network based on the parameter θjoint-blank of the third model information 153.
[0035] Furthermore, the probability distribution calculation unit 123 calculates the probability that the u-th symbol (non-blank) corresponds to each of the K tokens' corresponding labels, based on the intermediate input features at time t and the u-th intermediate symbol features, using equation (4).
[0036]
[0037] Softmax(•) is the softmax function. fjoint-label(•) is a neural network based on the parameter θjoint-label of the third model information 153.
[0038] Note that fjoint-blank(•) and fjoint-label(•) may constitute a single two-headed neural network. As shown in equations (3) and (4), fjoint-blank(•) and fjoint-label(•) share a common input. The number of elements in the vector on the left side of equation (4) is K, which is the same as the number of tokens.
[0039] Furthermore, the probability distribution calculation unit 123 calculates the probability distribution for the blank and each symbol as shown in equation (5), based on the calculation results of equations (3) and (4).
[0040]
[0041] Since there are K+1 labels including blanks, the number of elements in the vector on the left side of equation (5) is K+1. The elements in the vector on the left side of equation (5) represent the probability of a blank at each time step, and the probability of each non-blank symbol.
[0042] Furthermore, the probability distribution calculation unit 123 performs the same calculation as in equation (5) for each time step. Therefore, the final calculation result of the probability distribution calculation unit 123 during learning is a three-dimensional tensor of size T × U × (K+1). (During inference, it becomes a two-dimensional matrix of size (T + U') × (K+1), where U' represents the number of output tokens during inference.)
[0043] The method for calculating the probability distribution considering blanks by the estimation unit 12 described here is the same as the method for calculating the probability distribution in HAT described in Non-Patent Document 5. Furthermore, the hypothesis generation unit 13 can generate recognition hypotheses based on the output probability distribution in the same manner as ALSD described in Reference Document 1. That is, as shown in Fig. 1 of Reference Document 1, the hypothesis generation unit 13 searches for a path in a plane with time on the horizontal axis and the number of tokens on the vertical axis, based on the output probability distribution, such that the Manhattan distance is minimized and the probability is maximized. The hypothesis generation unit 13 outputs a recognition hypothesis based on the obtained path. Reference Document 1: George Saon et al., “Alignment-length Synchronous Decoding For RNN-Transducer,” in Proc. of ICASSP, 2020.
[0044] For example, the estimation unit 12 can obtain a sequence of symbols representing the estimation result by selecting the symbol with the highest probability among all symbols, including blanks, for each time step, based on each element of the vector on the left side of equation (5).
[0045] In ALSD described in Reference 1, the maximum number of tokens was predicted in advance, and based on that maximum number, paths were searched for the entire acoustic feature sequence at once. In contrast, the speech recognition device 10 of the first embodiment calculates the probability distribution and searches for paths for each chunk divided by the division unit 11. This enables high-performance and high-speed speech recognition. The speech recognition device 10 can obtain a final recognition hypothesis by integrating the paths.
[0046] Here, we will explain the flow of speech recognition processing by the speech recognition device 10 using Figure 2. Figure 2 is a flowchart showing the flow of speech recognition processing in the first embodiment.
[0047] As shown in Figure 2, first, the splitting unit 11 divides the acoustic feature sequence into N chunks (step S101). N is a number determined by the chunk size. The splitting unit 11 divides the acoustic feature sequence into multiple consecutive time chunks of a fixed length.
[0048] For example, if speech recognition processing is batch processing and a sequence of acoustic features of length T and a chunk size are given in advance, then N may be the smallest integer greater than the value obtained by dividing T by the chunk size. However, if T is not divisible by the chunk size, the length of the acoustic features in the Nth chunk will be the remainder.
[0049] Also, for example, if speech recognition processing is streaming, the model (e.g., f enc If the chunk size that can be input to (・) is predetermined, inputting the chunk size is unnecessary. The splitting unit 11 passes the input acoustic feature quantities to the subsequent processing in chunk units based on the chunk size.
[0050] In subsequent processing, the estimation unit 12 uses a machine learning model to estimate the sequence of symbols, including blanks, for each chunk.
[0051] Here, the estimation unit 12 initializes the chunk number n to 1 (step S102). The estimation unit 12 calculates the intermediate input feature sequence of the nth chunk (step S103). The estimation unit 12 calculates the intermediate symbol feature sequence of the nth chunk (step S104). The estimation unit 12 calculates the output probability distribution of the nth chunk (step S105).
[0052] If n is not N (step S106; No), the estimation unit 12 increases n by 1 (step S107) and returns to step S103 to repeat the process.
[0053] If n=N (step S106; Yes), the hypothesis generation unit 13 generates a recognition hypothesis based on the output probability distribution (step S108). The hypothesis generation unit 13 outputs the generated recognition hypothesis (step S109). If the intermediate input features in step S103 are extracted before step S101, and the intermediate input features are split instead of the acoustic features in step S101, Figure 2 can also be applied to offline speech recognition processing.
[0054] Figure 3 illustrates the speech recognition process. In Figure 3, the paths explored as recognition hypotheses by the estimation unit 12 (dotted lines in the figure) are represented on a plane with time t on the horizontal axis and the number of tokens u on the vertical axis. In reality, as shown in Fig. 1 of Reference 1, paths other than the recognition hypotheses may exist.
[0055] The symbol sequence of the correct recognition hypothesis output for the input acoustic feature sequence is "ko", "n", "ni", "chi", "ha", "ima", "hi", "ha", "ne", with 9 tokens. However, the symbol sequence of the recognition hypothesis and the number of tokens are unknown at the time the acoustic feature sequence is input.
[0056] For example, if the length T of the input acoustic feature sequence is 18, and the chunk size is L, cis set to 3. The division unit 11 calculates the number of chunks, where N=18÷3=6, and divides the acoustic feature sequence into 6 blocks each corresponding to a chunk. The estimation unit 12 calculates a probability distribution for each of block 501, block 502, block 503, block 504, block 505, and block 506. Further, the hypothesis generation unit 13 searches for a symbol sequence. When a blank corresponds to a time, the path proceeds to the right (the positive direction of t).
[0057] The estimation unit 12 estimates, for each of a plurality of time points included in each chunk, information for identifying one or more symbols on the basis of speech features. For example, the information for identifying one or more symbols including a blank is the probability when the blank corresponds to a speech feature, and the probability when each of a plurality of symbols (tokens) other than the blank corresponds to the speech feature.
[0058] For example, at time points t1, t2, t3, t4, t5, t6, t7, t8, and t9, the probability of the blank is lower than the probability of any token, the token with the highest probability is output, and the path proceeds upward. At other time points, the probability of the blank is higher than the probability of the token, no token is output, the blank is output, and the path proceeds to the right.
[0059] For example, block 501 is L c ×L c space where 0<=t<=L c ). When the estimation unit 12 calculates the probability distribution for block 501, the number of tokens (the amount of upward movement of the path) does not exceed L c . This will be explained using FIG. 4. FIG. 4 is a diagram illustrating an example of an algorithm for speech recognition processing.
[0060] The algorithm in FIG. 4 is an improvement over "Algorithm 2: Alignment-length synchronous decoding" in Reference Document 1, that is, the ALSD algorithm. The estimation unit 12 performs probability distribution calculation and path search in accordance with the algorithm in FIG. 4.
[0061] Line 7 is the process by which the intermediate input feature calculation unit 121 calculates the sequence of intermediate input features for the acoustic feature sequence of the nth chunk. Line 17 is the process by which the intermediate symbol feature calculation unit 122 calculates the intermediate symbol features. Line 18 is the process by which the probability distribution calculation unit 123 calculates the probability distribution. joint (•) represents a function that combines the calculations of equations (3), (4), and (5).
[0062] As shown in lines 8, 12, and 24, the number of tokens in a chunk is cnt. i This is reset for each chunk and becomes equal to the number of time points within the chunk. In this case, the estimation unit 12 does not calculate the probability distribution for a number of tokens that exceeds the chunk size.
[0063] On the other hand, in ALSD described in Reference 1, the range of the number of tokens that are the target of probability distribution calculation and path search is U max It is determined by the following. For example, in the example in Figure 3, the optimal U max The value of is 9, but the optimal U is when the symbol sequence is unknown. max Setting this is difficult. Also, as described later, according to the first embodiment, the optimal U in ALSD max It can perform speech recognition with the same performance as when this setting is enabled, and furthermore, it can shorten the processing time compared to ALSD.
[0064] Here, T=18, L c Let = 3. In the first embodiment, the area of one block is 3 × 3 = 9. Since there are 6 blocks, the total area to be searched is 9 × 6 = 54. On the other hand, if we assume that U in ALSD max If this value is set to 9, the total area to be searched is 18 × 9 = 162, which is greater than 54. The larger this area, the greater the computational complexity in the speech recognition process (for example, the number of times equations (2), (3), (4), and (5) are calculated).
[0065] Furthermore, in ALSD, U max A sufficiently large value (for example, U maxSetting it to =T (=18>>9) could also be considered to prevent performance degradation. However, in this case U max Compared to the case where the value is optimal, performance does not improve, and the computational complexity only increases due to redundant calculations.
[0066] Furthermore, in ALSD, U max If it is too small (for example, U max =4), some symbol sequences may not be obtained, resulting in reduced performance.
[0067] [Effect] As explained above, the division unit 11 divides the acoustic feature sequence, whose elements are the feature quantities at each of multiple time points in the speech to be recognized, into multiple chunks. The estimation unit 12 uses a machine learning model to estimate a sequence of symbols, including blanks, for each chunk. For example, the division unit 11 divides the acoustic feature sequence into multiple consecutive time chunks of a certain length. In this way, the speech recognition device 10 can search for symbols for each chunk. As a result, the speech recognition device 10 can perform high-performance and high-speed speech recognition by reducing the amount of processing without lowering the accuracy compared to ALSD.
[0068] Furthermore, the estimation unit 12 estimates information for identifying one or more symbols for each of the multiple time points included in each chunk, based on the speech features. For example, the estimation unit 12 calculates the probability distribution for each of the multiple time points included in each chunk where a blank corresponds to a speech feature, and the probability distribution for each of the multiple symbols other than blanks that corresponds to a speech feature. As a result, the speech recognition device 10 can take into account that the symbol sequence includes blanks and optimize the length of the symbol sequence regardless of the length of the acoustic feature sequence.
[0069] [Experiment concerning the first embodiment] Figure 5 shows the results of an experiment conducted to confirm the effect of the first embodiment. Figure 5 is a diagram showing the results of the experiment of the first embodiment. ALSD is the method described in Reference 1. Optimal U maxIt is 200. AdaALSD is the method of the first embodiment. The chunk size is 15.
[0070] WER (word error rate) is the percentage of incorrect symbols output; a lower WER indicates better performance. RTF (real time factor) is the processing time relative to the length of the audio; a lower RTF indicates faster processing speed.
[0071] As shown in Figure 5, AdaALSD is U max Regardless of the value, it outperformed ALSD in terms of performance and processing speed. Also, the optimal U max Setting it up can be difficult and may require a lot of effort. For this reason, AdaALSD is U max It is also useful because it eliminates the need for configuration.
[0072] [About the Learning Device] The configuration of the learning device for realizing speech recognition by the speech recognition device 10 will be explained using Figure 6. Figure 6 is a diagram showing an example of the configuration of the learning device according to the first embodiment.
[0073] As shown in Figure 6, the learning device 20 includes an estimation unit 22, a loss calculation unit 23, and an update unit 24.
[0074] The estimation unit 22 has a configuration similar to that of the estimation unit 12 of the speech recognition device 10. The estimation unit 22 includes an intermediate input feature calculation unit 221, an intermediate symbol feature calculation unit 222, and a probability distribution calculation unit 223. The estimation unit 22 also stores a first model information 251, a second model information 252, and a third model information 253.
[0075] The intermediate input feature calculation unit 221, the intermediate symbol feature calculation unit 222, and the probability distribution calculation unit 223 perform calculations using a machine learning model. The first model information 251, the second model information 252, and the third model information 253 are parameters of the machine learning model used in the intermediate input feature calculation unit 221, the intermediate symbol feature calculation unit 222, and the probability distribution calculation unit 223, respectively.
[0076] The intermediate input feature calculation unit 221 calculates intermediate input features based on acoustic features. The intermediate input feature calculation unit 221 calculates intermediate input features using a multi-stage neural network constructed based on the first model information 251. For example, the intermediate input feature calculation unit 221 obtains intermediate input features by encoding acoustic features. The probability distribution calculation unit 223 calculates a probability distribution based on the intermediate input features and intermediate symbol features.
[0077] The loss calculation unit 23 calculates the loss based on the degree of deviation between the probability distribution calculated by the probability distribution calculation unit 223 and the correct symbol sequence. The update unit 24 updates the model parameters, namely the first model information 251, the second model information 252, and the third model information 253, so as to minimize the loss.
[0078] The updated first model information 251, second model information 252, and third model information 253 are used as the first model information 151, second model information 152, and third model information 153 of the speech recognition device 10.
[0079] The learning device 20 can calculate losses and update parameters using known machine learning methods (for example, the RNN-T learning method described in Non-Patent Document 1).
[0080] [Second Embodiment] In the second embodiment, a method for calculating an intermediate input feature sequence quickly and accurately using a latency-controlled selective state-space model (SSM) will be described. Note that explanations of matters common to the first and second embodiments will be omitted as appropriate.
[0081] Non-patent document 3 describes SSM. SSM is a method for calculating the intermediate acoustic feature sequence and latent state at a given time based on the intermediate acoustic feature sequence at the target time and the latent state at the time immediately preceding (past) the target time. Information from past input sequences is stored in the latent state.
[0082] Because SSM operates in a unidirectional direction (left-to-right, as time progresses), it cannot access future information. On the other hand, in offline speech recognition where an acoustic feature sequence for the entire target period is prepared, future information is available at each time point. In this case, by preparing bidirectional SSMs (left-to-right and right-to-left) and utilizing the bidirectional information, the accuracy of speech recognition can be improved.
[0083] A method for extending the unidirectional SSM to bidirectional operation is defined by equations (6) to (11). Note that the letters used in equations (6) to (11) may have different meanings than the same letters used in other drawings and equations. For example, x, y, and h in equations (6) to (11) may have different meanings than x, y, and h in equations (1) to (5) and in Figure 4.
[0084]
[0085]
[0086]
[0087]
[0088]
[0089]
[0090] x t x is the intermediate input feature at time t. t h may be an element of an intermediate input feature sequence obtained by encoding an acoustic feature sequence. t is the latent state at time t. t This represents the intermediate input features at time t. In other words, this method calculates and outputs further intermediate input features based on the input intermediate input features.
[0091] Furthermore, a rightward-pointing arrow directly above a variable indicates left-to-right movement (hereinafter referred to as forward movement). Similarly, a leftward-pointing arrow directly above a variable indicates right-to-left movement (hereinafter referred to as backward movement).
[0092] Even if multiple variables are represented by the same letter, if the arrows pointing in different directions indicate that the variables are distinct. For example, h with a rightward-pointing arrow. t This represents a forward latent state, and is indicated by an h with a leftward-pointing arrow. t In other words, it is different from a backward-looking latent state.
[0093] α, β, γ, and δ are parameters used to compute latent states or intermediate input features (e.g., regression coefficients, neural network weight matrices, etc.). As in the previous example, even if multiple parameters are represented by the same letter, if the arrows pointing in different directions, the parameters are distinct from each other. For example, α with a rightward arrow is different from α with a leftward arrow.
[0094] Equation (6) is the formula for calculating the forward latent state at time t based on the forward latent state at time t-1 and the intermediate input features at time t. Equation (7) is the formula for calculating the forward intermediate input features at time t based on the forward latent state at time t. Equation (8) is the formula for calculating the backward latent state at time t based on the backward latent state at time t+1 and the intermediate input features at time t. Equation (9) is the formula for calculating the backward intermediate input features at time t based on the backward latent state at time t.
[0095] Equation (10) is the formula for calculating the intermediate input features at time t by concatenating the forward intermediate input features and the backward intermediate input features at time t and then performing a linear projection. The parameter δ is a parameter for the linear projection.
[0096] Alternatively, the intermediate input features at time t may be calculated by adding the forward intermediate input features at time t and the backward intermediate input features at time t, as shown in equation (11).
[0097] The SSM described so far operates frame by frame (at time t) and only supports left-to-right or right-to-left, making it difficult to operate on a chunk-by-chunk basis.
[0098] In contrast, the speech recognition device and learning device of the second embodiment can operate the SSM in chunk units by controlling the delay.
[0099] Figure 7 shows an example of the configuration of the intermediate input feature calculation unit in the second embodiment. Here, the processing of the intermediate input feature calculation unit 121 of the speech recognition device 10 will be explained. However, the intermediate input feature calculation unit 221 of the learning device 20 may have the same configuration as the intermediate input feature calculation unit 121 in Figure 7.
[0100] As shown in Figure 7, the intermediate input feature calculation unit 121 includes an encoder 121a, a forward feature calculation unit 121b, a backward feature calculation unit 121c, a delay control unit 121d, and a coupling unit 121e.
[0101] The encoder 121a encodes the acoustic feature sequence, which has been divided into chunks, and outputs an intermediate acoustic feature sequence for each chunk. The intermediate acoustic feature sequence is an example of an intermediate input feature sequence. The encoder 121a may perform the same processing as the intermediate input feature calculation unit 121 in the first embodiment.
[0102] The forward feature calculation unit 121b calculates the forward latent state at time t based on the forward latent state at time t-1 and the intermediate input features at time t. The forward feature calculation unit 121b also calculates the forward intermediate acoustic features at time t based on the forward latent state at time t. For example, the forward feature calculation unit 121b performs the calculations of equations (6) and (7).
[0103] The backward feature calculation unit 121c calculates the backward latent state at time t based on the backward latent state at time t+1 and the intermediate acoustic features at time t. The backward feature calculation unit 121c also calculates the backward intermediate acoustic features at time t based on the backward latent state at time t. For example, the backward feature calculation unit 121c performs the calculations of equations (8) and (9).
[0104] The delay control unit 121d controls the backward feature calculation unit 121c so that backward intermediate acoustic features are calculated for each chunk.
[0105] The coupling unit 121e combines the forward intermediate acoustic features at time t with the backward intermediate acoustic features at time t. For example, the coupling unit 121e performs the calculation of equation (10) or equation (11).
[0106] The intermediate input feature calculation unit 121 performs the processing shown in the 7th line of Figure 4. For example, the acoustic feature sequence input to the encoder 121a is X shown in the 7th line of Figure 4. n This may be the case. Also, the sequence whose elements are the intermediate acoustic features joined by the coupling part 121e is H in the 7th row of Figure 4. n enc That's fine.
[0107] The processing of the delay control unit 121d will be explained in detail using Figure 8. Figure 8 is a diagram illustrating the calculation method of intermediate input features. It is assumed that the latent state variables are shared at each time step. For example, the forward latent state at time t-1 stored in a variable is used to calculate the forward latent state at time t, and then updated (overwritten) with the forward latent state at time t. Similarly, for example, the backward latent state at time t+1 stored in a variable is used to calculate the backward latent state at time t, and then updated (overwritten) with the backward latent state at time t.
[0108] As shown in Figure 8, in the forward processing, that is, the process in which the forward feature calculation unit 121b calculates forward intermediate acoustic features, the forward latent state should be continuously updated from the initial state (init.state) without considering chunks from time t=1 to time t=T.
[0109] On the other hand, the backward processing, that is, the process by which the backward feature calculation unit 121c calculates backward intermediate acoustic features, is performed for each chunk. Therefore, the backward latent state is initialized for each chunk.
[0110] The delay control unit 121d performs this initialization. For example, after the calculation of the backward intermediate acoustic features of the chunk shown by H1 in Figure 8 is completed, the delay control unit 121d updates the backward latent state stored in the variable to its initial value (e.g., 0). As a result, when the calculation of the backward intermediate acoustic features of the chunk shown by H2 in Figure 8 is started, the backward latent state is at its initial value.
[0111] If initialization by the delay control unit 121d is not performed, the time t = 2 × L of the chunk shown by H2 in Figure 8 cThe calculation of the backward latent state will be based on the backward latent state at time t=1, which could lead to a decrease in the accuracy of speech recognition.
[0112] Figure 9 is a flowchart showing the process flow for calculating intermediate input features in the second embodiment. The process in Figure 9 corresponds to step S103 in Figure 2. That is, the process in Figure 9 is the process within the nth chunk. The process in Figure 9 is executed for each of the N chunks.
[0113] As shown in Figure 9, first, the intermediate input feature calculation unit 121 encodes the acoustic feature sequence of the nth chunk and calculates the intermediate input feature sequence (step S201).
[0114] The intermediate input feature calculation unit 121 calculates ((n-1) × L) for t. c Substitute ((n-1) × L) + 1 (step S202). c )+1 is the time at the beginning of the nth chunk. Also, the intermediate input feature calculation unit 121 calculates t f Substitute t into t b (n × L c Substitute t - t + 1 (step S203). f and t b These represent the time points for which forward intermediate input features (hereinafter referred to as forward features) and backward intermediate input features (hereinafter referred to as backward features) are calculated, respectively.
[0115] Next, the intermediate input feature calculation unit 121 calculates time t f Read the forward latent state of -1 (step S204). Time t f The forward latent state of -1 is already computed during the process of calculating the intermediate feature sequence of the previous chunk. Time t f If the forward latent state of -1 has not been calculated, the intermediate input feature calculation unit 121 calculates at time t f You may initialize to a forward latent state of -1.
[0116] Meanwhile, the intermediate input feature calculation unit 121 calculates time t b Initialize the backward latent state of +1 (step S205). For example, the intermediate input feature calculation unit 121 calculates at time t bThe system initializes the backward latent state regardless of whether the +1 backward latent state has already been calculated. The initialization is performed by the delay control unit 121d. In this way, the intermediate input feature calculation unit 121 initializes the backward latent state each time the calculation of backward features for each chunk is completed.
[0117] Next, the intermediate input feature calculation unit 121 calculates time t f -1 forward potential state and time t f Based on the intermediate input features, time t f The forward latent state is calculated (step S206). The intermediate input feature calculation unit 121 calculates the time t f Based on the positive latent state at time t f The forward features are calculated (step S207).
[0118] In this way, the intermediate input feature calculation unit 121 updates the backward latent state for each of the multiple time points in each chunk based on the input feature sequence, in the reverse order of the time series, and calculates backward features based on the backward latent state each time the backward latent state is updated.
[0119] The intermediate input feature calculation unit 121 calculates time t b +1 backward latent state and time t b Based on the intermediate input features, time t b The backward latent state is calculated (step S208). The intermediate input feature calculation unit 121 calculates the time t b Based on the backward latent state, time t b The backward features are calculated (step S209).
[0120] In this way, the intermediate input feature calculation unit 121 updates the forward latent state in chronological order for each of the multiple time points based on the input feature sequence, which is a sequence of input features at each of the multiple time points of the speech to be recognized and is divided into multiple chunks. Each time the forward latent state is updated, it calculates the forward features based on the forward latent state.
[0121] t f =n × L cIf not (step S210; No), i.e., if the calculation of the intermediate features of the nth chunk is not completed, the intermediate input feature calculation unit 121 will f Increase t by 1, b The value is reduced by 1 (step S211). Then, the intermediate input feature calculation unit 121 returns to step S206 and repeats the process.
[0122] t f =n × L c If the condition is met (step S210; Yes), i.e., if the calculation of intermediate features for the nth chunk is complete, the intermediate input feature calculation unit 121 calculates combined features based on the forward and backward features for each time step of the nth chunk (step S212). The intermediate input feature calculation unit 121 also outputs the sequence of combined features as the intermediate input feature sequence (step S213).
[0123] [Experiment concerning the second embodiment] Figure 10 shows the results of an experiment conducted to confirm the effects of the second embodiment. Figure 10 is a diagram showing the results of the experiment of the second embodiment. Attention is a method using a Conformer based on the Transformer described in Non-Patent Literature 2. Unidirectional SSM is an SSM that operates only left-to-right. Latency-controlled SSM is the method of the second embodiment.
[0124] As shown in Figure 10, Unidirectional SSM has a worse word error rate than Attention, which operates in chunks, because it cannot access future information. Also, because Unidirectional SSM operates frame-by-frame, its RTF is worse than Attention.
[0125] Furthermore, Latency-controlled SSM shows improved word error rate and RTF compared to Attention with the same chunk size. Therefore, according to the second embodiment, higher performance and faster speech recognition is possible than before.
[0126] [Regarding the learning method] As mentioned above, the learning device 20 can calculate intermediate input features using the method of the second embodiment when learning the model.
[0127] Here, the dimension of the intermediate input feature sequence encoded from the acoustic feature sequence of the training data is B × T × D, where B is the batch size, T is the input sequence length, and D is the feature dimension. c is the chunk size. N is T = L c This is the value obtained by dividing by and then applying the ceiling function. Note that any fractional parts of the chunk where no data exists (for example, up to a maximum length of T-1) are padded with zeros. Also, the letters used here may have different meanings than the same letters used in other diagrams and formulas.
[0128] To speed up the learning process, the learning device 20 changes the dimension of the intermediate input feature sequence from B × T × D to (B × N) × L c The format is converted to ×D and processed backward. Then, after the backward processing, the learning device 20 converts the dimension of the intermediate input feature sequence back to B×T×D and inputs it to the next layer (for example, the coupling unit 121e). Through this operation, L c Under the condition = 15, the learning speed increased by 1.5 times. Previously, it was necessary to recursively process data spanning multiple chunks chunk by chunk, but (B × N) × L c By converting to ×D, data spanning multiple chunks can be treated as a single batch and processed all at once.
[0129] In this way, the learning device 20 transforms the input feature sequence, which is the first data set having three dimensions: batch size, sequence length, and number of features, into second data sets having three dimensions: the product of the batch size and the number of chunks, the chunk size, and the number of features, and uses the second data sets to calculate backward features.
[0130] [System Configuration, etc.] Furthermore, the components of each part shown in the diagram are functional concepts and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown in the diagram, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. In addition, all or any part of the processing functions performed by each device can be realized by a CPU and the program executed on that CPU, or by hardware using wired logic.
[0131] Furthermore, among the processes described in the embodiments described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.
[0132] [Program] The voice recognition device 10 described above can be implemented by installing a program as packaged software or online software on a desired computer. For example, by having the above program run on an information processing device, the information processing device can be made to function as the voice recognition device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), and terminals such as PDA (Personal Digital Assistant).
[0133] Figure 11 shows an example configuration of a computer running a program. Computer 1000 has, for example, memory 1010 and a CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0134] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.
[0135] The hard disk drive 1090 stores, for example, the OS 1091, application program 1092, program module 1093, and program data 1094. That is, the program that defines each process executed by the speech recognition device 10 is implemented as a program module 1093 in which executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration of the speech recognition device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).
[0136] Furthermore, the data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.
[0137] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.
[0138] The functions of the elements disclosed herein may be implemented using circuits or processing circuitry that include general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), conventional circuits, and / or combinations thereof that are programmed using one or more programs stored in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered processing circuitry or circuitry because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, circuits, units, or means are hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions.
[0139] There is a memory for storing a computer program that includes computer instructions. These computer instructions provide logic and routines that enable hardware (e.g., processing circuitry or circuitry) to perform the methods disclosed herein. This computer program can be implemented in commonly known forms, such as computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs and DVDs, and / or memory of FPGAs and ASICs.
[0140] 10 Speech recognition device 11 Splitting unit 12 Estimation unit 13 Hypothesis generation unit 20 Learning device 23 Loss calculation unit 24 Update unit 121, 221 Intermediate input feature calculation unit 121a Encoder 121b Forward feature calculation unit 121c Backward feature calculation unit 121d Delay control unit 121e Coupling unit 122, 222 Intermediate symbol feature calculation unit 123, 223 Probability distribution calculation unit 151, 251 First model information 152, 252 Second model information 153, 253 Third model information 501, 502, 503, 504, 505, 506 Block
Claims
1. A speech recognition device comprising: a forward feature calculation unit that updates a first latent state in chronological order for each of the multiple time points based on an input feature sequence divided into multiple chunks, and calculates a first feature based on the first latent state each time the first latent state is updated; a backward feature calculation unit that updates a second latent state in reverse chronological order for each of the multiple time points based on the input feature sequence, and calculates a second feature based on the second latent state each time the second latent state is updated; and a delay control unit that initializes the second latent state each time the calculation of the second feature for each chunk is completed.
2. The speech recognition device according to claim 1, further comprising a coupling portion for combining the first feature quantity and the second feature quantity.
3. The speech recognition device according to claim 1, further comprising an estimation unit that estimates a sequence of symbols including blanks for each of the plurality of chunks using a machine learning model based on the first and second features.
4. The speech recognition device according to claim 3, further comprising an update unit that updates the parameters of the machine learning model so as to minimize the loss calculated based on the degree of deviation between the symbol sequence estimated by the estimation unit and the correct symbol sequence.
5. The speech recognition device according to claim 4, wherein the backward feature calculation unit converts the input feature sequence, which is first data having three dimensions, each having a batch size, sequence length, and number of features, into second data having three dimensions, each having a batch size multiplied by the number of chunks, the size of the chunk, and the number of features, and calculates the second features using the second data.
6. A speech recognition method in which a computer performs the following processes: updating a first latent state in chronological order for each of the multiple time points based on a sequence of input features of the speech to be recognized at each of multiple time points, which is divided into multiple chunks; calculating a first feature based on the first latent state each time the first latent state is updated; updating a second latent state in reverse chronological order for each of the multiple time points based on the input feature sequence for each of the multiple time points, in the reverse order of chronological order; calculating a second feature based on the second latent state each time the calculation of the second feature for each chunk is completed; and initializing the second latent state each time the calculation of the second feature for each chunk is completed.
7. A speech recognition program that causes a computer to perform the following processes: updating a first latent state in chronological order for each of the multiple time points based on a sequence of input features of the speech to be recognized at each of multiple time points, which is divided into multiple chunks; calculating a first feature based on the first latent state each time the first latent state is updated; updating a second latent state in reverse chronological order for each of the multiple time points based on the input feature sequence for each of the multiple time points, in the reverse order of chronological order; calculating a second feature based on the second latent state each time the calculation of the second feature for each chunk is completed.
