Speech recognition method and apparatus, speech recognition model training method and apparatus, electronic device, and vehicle

Abstract

A speech recognition method and apparatus, a speech recognition model training method and apparatus, an electronic device, and a vehicle. The speech recognition method comprises: performing feature extraction on a mixed audio stream to obtain a first audio feature of a target object and a second audio feature of another object other than the target object, wherein the mixed audio stream comprises speeches of at least two objects, and the target object is an object distinguished from the at least two objects; on the basis of the first audio feature, obtaining a first feature vector of the speech uttered by the target object; on the basis of the second audio feature, predicting a second feature vector associated with the speech possibly uttered by the target object; and on the basis of the first feature vector and the second feature vector, recognizing a target text corresponding to the speech uttered by the target object in the mixed audio stream. A speech of a target object in a mixed audio stream is recognized using features extracted in two paths, so that the recognition accuracy of mixed audio streams can be improved.

Description

Speech recognition methods, model training methods, devices, electronic devices, and vehicles

[0001] This application claims priority to Chinese Patent Application No. 2025100952031, filed on January 21, 2025, entitled "Speech Recognition Method, Model Training Method, Apparatus, Electronic Device and Vehicle", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of artificial intelligence technology, specifically to a speech recognition method, a model training method, an apparatus, an electronic device, and a vehicle. Background Technology

[0003] Speech recognition refers to the human-computer interaction process of converting human language into text or commands. Currently, speech recognition technology is widely used in scenarios such as speech interaction, speech transcription, speech analysis, and audio and video recording.

[0004] In related technologies, speech recognition utilizes deep learning models trained on massive datasets to learn the relationship between audio signals and corresponding text, thus enabling the conversion of user speech into text. In human-computer interaction scenarios, the collected speech is typically mixed (multiple voices or environmental noise), making the recognition of mixed speech streams a more widespread application. Deep learning models in related technologies process mixed speech streams by continuously extracting the speech features of individual objects from the mixed speech stream and then identifying the target text corresponding to those features. This speech recognition method has low accuracy. Furthermore, for mixed speech stream recognition, the model needs to continuously recognize data. As the length of the speech stream increases, the recognition accuracy gradually decreases. Therefore, after recognizing a certain length of mixed speech stream, the model needs to pause for a period before resuming recognition. Consequently, the speech recognition methods in related technologies suffer from low accuracy and require frequent pauses.

[0005] Therefore, there is an urgent need to provide a speech recognition method with higher accuracy to reduce pauses. Summary of the Invention

[0006] One objective of this invention is to provide a speech recognition method to solve the problem of low accuracy in mixed speech recognition in the prior art; a second objective is to provide a training method for a speech recognition model; a third objective is to provide a speech recognition device; a fourth objective is to provide a training device for a speech recognition model; a fifth objective is to provide an electronic device; and a sixth objective is to provide a vehicle.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A speech recognition method, the method comprising:

[0009] Feature extraction is performed on the mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects besides the target object. The mixed audio stream includes the speech of at least two objects, and the target object is an object distinguished from the at least two objects.

[0010] Based on the first audio feature, a first feature vector of the speech emitted by the target object is obtained;

[0011] Based on the second audio features, a second feature vector associated with the speech that the target object may emit is predicted;

[0012] Based on the first feature vector and the second feature vector, the target text corresponding to the speech emitted by the target object in the mixed audio stream is identified.

[0013] Based on the aforementioned technical means, when extracting features from a mixed audio stream, the features of the mixed audio stream are extracted into a first audio feature and a second audio feature to distinguish the mixed audio stream. Then, two processing paths are used to process the first and second audio features respectively. The first audio feature is processed to obtain a first feature vector, which serves as the first path for feature processing; the second audio feature is processed to obtain a second feature vector, which serves as the second path for feature processing. By fusing the first and second feature vectors, the target text is finally identified. First, the mapping probability between the first audio feature and the target text is calculated using the first path, while the mapping probability between the first audio feature and the target text is predicted using the second path. The prediction results from the two paths are then fused, which can improve the accuracy of speech recognition. Alternatively, it can be understood that in the same voice interaction scenario, the voices of other objects in the mixed voice stream have a certain correlation with the voice of the target object. This embodiment predicts the mapping probability between the voice of the target object and the target text by adding a second feature vector associated with the voice that the target object may emit, which can improve the recognition accuracy of each target object in the mixed audio stream. Therefore, in the recognition scenario of mixed audio stream, the voice recognition accuracy is high and the voice recognition pauses during the recognition process can be reduced.

[0014] Further, the step of identifying the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first feature vector and the second feature vector includes:

[0015] The first feature vector and the second feature vector are fused in time synchronization to obtain a fused feature vector;

[0016] Based on the fused feature vector, the target text corresponding to the speech emitted by the target object in the mixed audio stream is identified.

[0017] Based on the above technical means, by converting the first audio feature into a first feature vector and the second audio feature into a second feature vector, and then fusing them, the accuracy can be improved and the amount of computation can be reduced by the post-fusion method, which provides the possibility for deploying the recognition method on the edge.

[0018] Furthermore, the identification method also includes:

[0019] Predict the backward probability of the speech that the target object may emit in the next time step based on the first audio feature;

[0020] The step of identifying the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first feature vector and the second feature vector includes:

[0021] Based on the first feature vector, the second feature vector, the backward probability, and the forward probability of the speech emitted by the target object predicted in the previous time step, the target text corresponding to the speech emitted by the target object in the mixed audio stream is predicted by a preset conditional probability.

[0022] Based on the aforementioned technical means, forward probability and backward probability are used to associate the features of the preceding and following parts of a speech sequence. For the recognition of streaming audio features, the features of words or sentences composed of preceding and following features are used to predict the target text corresponding to the speech emitted by the target object based on the combination of forward probability and backward probability. This can further improve the accuracy of recognition and reduce the frequency of pauses in the speech recognition model due to the decrease in recognition accuracy.

[0023] Furthermore, the feature extraction from the mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects besides the target object includes:

[0024] Preprocess the mixed audio stream to obtain mixed audio features;

[0025] Predict the mask of the speech emitted by the target object based on the mixed audio features;

[0026] Based on the mask and the mixed audio features, the first audio feature of the target object is obtained;

[0027] Based on the complement of the mask and the mixed audio features, the second audio features of other objects besides the target object are obtained.

[0028] Based on the above technical means, the mixed audio stream is preprocessed to extract mixed audio features that conform to the format, which facilitates masking. By masking the mixed audio stream, the features corresponding to the speech of different objects in the mixed audio stream can be distinguished, and the mixed audio stream can be separated into the audio features required for at least two paths.

[0029] Furthermore, it is applied to computing devices equipped with speech recognition models, wherein the speech recognition models include masking networks, long short-term memory networks, prediction networks, and joint networks, wherein:

[0030] The masking network is used to extract features from the mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects besides the target object;

[0031] The long short-term memory network is used to obtain a first feature vector of the speech emitted by the target object based on the first audio feature;

[0032] The prediction network is used to predict a second feature vector associated with the speech that the target object may emit, based on the second audio features.

[0033] The joint network is used to identify the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first feature vector and the second feature vector.

[0034] Based on the above technical means, since the masking network, long short-term memory network and prediction network in the speech recognition model are all extracting audio features, and the probability of the text is directly mapped after the features are fused by the joint network, the target text is obtained. After the mixed audio stream is input into the speech recognition model, the corresponding target text is directly output without multiple cascading, realizing end-to-end streaming speech recognition and improving recognition accuracy.

[0035] Furthermore, after identifying the target text corresponding to the speech emitted by the target object in the mixed audio stream, the method further includes:

[0036] When it is detected that the target text corresponding to the speech emitted by the target object is abnormal at the current moment, the model parameters of the speech recognition model are optimized based on the abnormal scenario of the target text, the application field of speech recognition, and the corrected text, wherein the corrected text is the corrected text corresponding to the abnormal target text.

[0037] Based on the aforementioned technical methods, during speech recognition, abnormal scenarios are monitored, and the causes and types of recognition anomalies are learned and correlated with the speech recognition application domain. By continuously optimizing the model parameters of the speech recognition model in abnormal scenarios, the model gradually adapts to potentially abnormal scenarios, reducing the occurrence of speech recognition anomalies and thus increasing the model's recognition accuracy. When enough abnormal scenarios in the speech recognition application domain have been learned, uninterrupted streaming speech recognition can be achieved. In other words, based on the above technical methods, the speech recognition model adapts to dynamically changing, variable-length data streams, achieving uninterrupted streaming recognition.

[0038] Furthermore, the optimization of the speech recognition model parameters based on the abnormal scenarios of the target text, the application domain of speech recognition, and the corrected text includes:

[0039] The speech recognition model is trained based on the abnormal scenarios of the target text, the application field of speech recognition, and the corrected text, and the loss of the speech recognition model is adjusted.

[0040] Adjust the parameters of the forward probability algorithm model and the backward probability algorithm model based on the adjusted loss;

[0041] Based on the adjusted forward probability algorithm model parameters, predict the forward probability of the speech that the target object may emit at the time before the current time, and predict the backward probability of the speech that the target object may emit at the time after the current time.

[0042] The gradient is calculated based on the forward probability, the backward probability, and the corrected text.

[0043] The model parameters of the speech recognition model are adjusted based on the gradient.

[0044] Based on the above technical means, the model parameters of the forward probabilistic algorithm and the backward probabilistic algorithm can be adjusted, and then the gradient can be calculated to update the model parameters. On the one hand, the model parameters can be optimized, and on the other hand, the model parameters with the minimum loss can be found quickly.

[0045] Furthermore, the model parameters of the speech recognition model are optimized via the cloud.

[0046] Based on the above technical means, the model parameters of the speech recognition model can be optimized in the cloud. Since the cloud and the terminal can be OTA in real time, the deployment in the cloud can reduce the computing power of the terminal and avoid affecting the normal use of other functions of the terminal when optimizing the model parameters.

[0047] A method for training a speech recognition model, comprising:

[0048] Obtain training samples, which include a sample mixed audio stream and sample labels corresponding to the sample mixed audio stream;

[0049] The sample mixed audio stream is input into a speech recognition model. The speech recognition model performs the following processing to identify the target text corresponding to the speech emitted by the target object in the sample mixed audio stream: Feature extraction is performed on the sample mixed audio stream to obtain a first audio feature of the target object and second audio features of other objects besides the target object; based on the first audio feature, a first feature vector of the speech emitted by the target object is obtained; based on the second audio feature, a second feature vector associated with possible speech emitted by the target object is predicted; based on the first feature vector and the second feature vector, the target text is identified.

[0050] The loss value of the target text relative to the corresponding target sample label is calculated based on the total loss function;

[0051] The model parameters of the speech recognition model are adjusted based on the loss value.

[0052] Based on the aforementioned technical means, by setting the speech recognition model to extract features from the mixed audio stream into first audio features and second audio features during feature extraction, the mixed audio stream can be distinguished. Then, it is divided into two processing paths to process the first and second audio features respectively. The first audio features are processed to obtain a first feature vector, which serves as the first feature processing path; the second audio features are processed to obtain a second feature vector, which serves as the second feature processing path. By fusing the first and second feature vectors, the target text is finally recognized. Firstly, the mapping probability between the first audio features and the target text is calculated using the first path, and simultaneously, the mapping probability between the first audio features and the target text is predicted using the second path. The prediction results from both paths are fused, which can improve the accuracy of speech recognition. Alternatively, it can be understood that in the same voice interaction scenario, the voices of other objects in the mixed voice stream have a certain correlation with the voice of the target object. This embodiment predicts the mapping probability between the voice of the target object and the target text by adding a second feature vector associated with the voice that the target object may emit, which can improve the recognition accuracy of each target object in the mixed audio stream. Therefore, in the recognition scenario of mixed audio stream, the voice recognition accuracy is high and the voice recognition pauses during the recognition process can be reduced.

[0053] Furthermore, the speech recognition model includes a masking network, and the feature extraction from the mixed audio stream of samples to obtain the first audio features of the target object and the second audio features of other objects besides the target object includes:

[0054] The sample mixed audio stream is input into the preprocessing encoder of the masking network, and the sample mixed audio stream is preprocessed based on the preprocessing encoder to obtain sample mixed audio features;

[0055] The sample audio features are input into the mask encoder of the mask network, and the sample audio features are processed based on the mask encoder to predict the mask of the speech emitted by the target object.

[0056] Based on the mask and the sample audio features, the first audio feature of the target object is obtained;

[0057] The second audio features of objects other than the target object are obtained by combining the complement of the mask with the sample audio features.

[0058] Based on the above technical means, by masking the mixed audio stream, the speech features corresponding to different objects in the mixed audio stream can be distinguished, and the mixed audio stream can be separated into the audio features required for at least two paths.

[0059] Furthermore, the speech recognition model includes a long short-term memory network, and obtaining the first feature vector of the speech emitted by the target object based on the first audio feature includes:

[0060] The first audio feature is input into the long short-term memory network, and feature extraction is performed on the first audio feature based on the long short-term memory network to obtain the first feature vector of the speech emitted by the target object.

[0061] And / or, the speech recognition model includes a prediction network, wherein predicting a second feature vector of the speech that the target object may emit based on the second audio features includes:

[0062] The second audio feature is input into the prediction network, and the prediction network predicts the speech that the target object corresponding to the second audio feature may emit, thereby generating the corresponding second feature vector.

[0063] Based on the above technical means, training the speech recognition model can achieve end-to-end speech recognition by directly extracting audio features from the mixed audio stream after inputting it into the speech recognition model, thus improving recognition accuracy. This is done without multiple cascading steps.

[0064] Furthermore, the speech recognition model includes a joint network, and the step of recognizing the target text based on the first feature vector and the second feature vector includes:

[0065] The first feature vector and the second feature vector are fused based on the joint network to obtain a fused feature vector;

[0066] Based on the fused feature vector, the target text corresponding to the speech emitted by the target object in the sample mixed audio stream is identified.

[0067] Based on the above technical means, the relationship between the first feature vector, the fused second feature vector, and the text is learned, thereby improving the accuracy of speech recognition.

[0068] Furthermore, the training method also includes:

[0069] Predict the backward probability of the speech that the target object may emit in the next time step based on the first audio feature;

[0070] The step of identifying the target text based on the first feature vector and the second feature vector includes:

[0071] Based on the first feature vector, the second feature vector, the backward probability, and the forward probability of the speech emitted by the target object predicted in the previous time step, the target text is predicted by a preset conditional probability.

[0072] Based on the aforementioned technical means, the correlation between the predicted probability and the forward and backward probabilities of the training samples and target sample labels at the current time step is learned, based on the forward probability of the training samples corresponding to the same sequence labels at the previous time step and the backward probability of the training samples corresponding to the same sequence labels at the next time step. This allows the loss of the calculated target text and target sample labels to be updated, thereby improving the accuracy of speech recognition.

[0073] Furthermore, the speech recognition model also includes an audio encoder and a label encoder, and the training method further includes:

[0074] The first audio feature is input into the audio encoder, the first audio feature is encoded based on the audio encoder, and the sample label is encoded according to the label encoder;

[0075] Predict the backward probability of the speech that the target object may emit in the next time step based on the first audio feature;

[0076] Based on the backward probability and the forward probability of the speech emitted by the target object predicted in the previous time step, the gradient of the loss function with respect to the model parameters is predicted.

[0077] The step of adjusting the model parameters of the speech recognition model based on the loss value includes:

[0078] When the loss value is greater than or equal to a preset threshold, the model parameters are adjusted according to the gradient.

[0079] Based on the aforementioned techniques, the gradient reflects the sensitivity of the speech recognition model to changes in model parameters (partial derivatives of the model parameters) as the difference (loss value) between the target text and the sample labels changes. By calculating the gradient and then adjusting the model parameters accordingly, the speech recognition model can quickly learn the correlation between the mixed audio stream and the sample labels, reducing the number of training samples and improving learning efficiency. Furthermore, since the gradient is calculated based on forward and backward probabilities, its influence on the model parameters tends to minimize the difference between the target text and the sample labels recognized by the speech recognition model.

[0080] Furthermore, the training method also includes:

[0081] The total loss function is constructed based on the error between the target text and the target sample label predicted by the first audio feature and the error between the target text and the target sample label predicted by the second audio feature.

[0082] A voice recognition device, comprising:

[0083] The extraction module is used to extract features from the mixed audio stream to obtain the first audio features of the target object and the second audio features of other objects besides the target object;

[0084] The first prediction module is used to obtain a first feature vector of the speech emitted by the target object based on the first audio feature;

[0085] The second prediction module is used to predict a second feature vector associated with the speech that the target object may emit, based on the second audio feature.

[0086] The recognition processing module is used to identify the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first feature vector and the second feature vector.

[0087] A training device for a speech recognition model, comprising:

[0088] The acquisition module acquires training samples, which include a sample mixed audio stream and sample labels corresponding to the sample mixed audio stream;

[0089] A speech recognition model is used to perform the following processing on the input sample mixed audio stream: identifying the target text corresponding to the speech emitted by a target object in the sample mixed audio stream; extracting features from the sample mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects besides the target object; obtaining a first feature vector of the speech emitted by the target object based on the first audio feature; predicting a second feature vector associated with possible speech emitted by the target object based on the second audio feature; and identifying the target text based on the first feature vector and the second feature vector.

[0090] The training module is used to calculate the loss value of the target text relative to the corresponding sample label based on the total loss function, and adjust the model parameters of the speech recognition model according to the loss value.

[0091] An electronic device, comprising: a memory and a processor;

[0092] The memory is used to store computer programs / instructions; the processor is used to implement the method described above according to the computer programs / instructions stored in the memory.

[0093] A vehicle includes: a memory and a processor;

[0094] The memory is used to store computer programs / instructions; the processor is used to implement the method described above according to the computer programs / instructions stored in the memory.

[0095] A computer-readable storage medium / computer program product, wherein the computer-readable storage medium stores a computer program / instructions, or the computer program product includes a computer program / instructions that, when executed by a processor, are used to implement the method described above.

[0096] The beneficial effects of this invention are:

[0097] (1) The present invention extracts audio features by setting up a dual path. One path extracts the first audio feature quantity through the first audio feature of the target object, and the other path predicts the second feature vector corresponding to the speech that the target object may emit through the second audio feature of other objects. The speech emitted by the target object is identified by the two feature vectors. The prediction of the two features is combined and fused, which can improve the accuracy of identifying the speech emitted by the target object in the mixed audio stream.

[0098] (2) This invention uses a speech recognition model to map acoustic feature sequences to label sequences and convert them synchronously, without the need for multiple cascaded models, thus simplifying the model.

[0099] (3) This invention calculates the partial derivatives (gradients) of the model loss function with respect to the model parameters using forward and backward probabilities, which guides how to adjust the model parameters to minimize the loss function, so that the speech recognition model has a small speech recognition error; combined with the heuristic error allocation method, it can process audio streams of varying lengths, which can reduce or even avoid stuttering.

[0100] (4) The system resources (CPU (processor), RAM (memory), EMMC (Flash storage)) are low and the accuracy is high, making it easy to deploy the speech recognition model on the edge. Attached Figure Description

[0101] Figure 1 is a hardware system example diagram of the speech recognition method provided in the embodiments of this application;

[0102] Figure 2 is a schematic diagram of an application scenario of the speech recognition method provided in the embodiment of this application;

[0103] Figure 3 is a schematic diagram of the software system of the speech recognition method provided in the embodiment of this application;

[0104] Figure 4 is a schematic flowchart of an embodiment of the speech recognition method provided in this application;

[0105] Figure 5 is a schematic diagram of the processing flow of the speech recognition model provided in the embodiments of this application;

[0106] Figure 6 is a schematic diagram of an application process of the speech recognition method provided in an embodiment of this application;

[0107] Figure 7 is a flowchart illustrating another embodiment of the speech recognition method provided in this application.

[0108] Figure 8 is a flowchart illustrating an embodiment of the training method for the speech recognition model provided in this application.

[0109] Figure 9 is a schematic diagram of the result of the speech recognition method provided in the embodiment of this application recognizing mixed audio streams. Detailed Implementation

[0110] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0111] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0112] Speech recognition refers to the human-computer interaction process of converting human language into text or commands. Currently, speech recognition technology is widely used in scenarios such as speech interaction, speech transcription, speech analysis, and audio and video recording.

[0113] In related technologies, speech recognition uses deep learning models to train on large amounts of data, learning the relationship between audio signals and corresponding text, thereby enabling the conversion of user speech into text.

[0114] However, in human-computer interaction scenarios, the collected speech is usually mixed speech (the voices of multiple people talking or environmental noise). Mixed speech affects the accuracy of recognition, and the recognition accuracy of the above speech recognition method is low.

[0115] Based on this, this application provides a speech recognition method that recognizes speech through dual-path feature prediction results. One path is based on the audio features of the target object, and the other path is based on the prediction of other audio features of objects other than the target object in the mix, which can improve the accuracy of recognition.

[0116] The following description covers the hardware system, software system, and specific implementation embodiments involved in the application of this application.

[0117] Figure 1 is an example diagram of the hardware system of the speech recognition method provided in the embodiment of this application. As shown in Figure 1, the hardware system is applied on the device side, which includes a microphone (MIC), a signal processor, a low-pass filter, a digital-to-analog converter (ADC / A2B) bus, an audio digital signal processor (Audio DSP), a main controller (AVN / HU / IVI SOC), and a communication module, etc.

[0118] The microphone (MIC) is used to acquire audio signals (or, for streaming audio, the audio signal stream). The acquired audio signal is transmitted to a signal processor, which amplifies the signal (e.g., using an active filter amplifier or a programmable amplifier) ​​and reduces noise. After gain amplification and noise reduction, the signal is transmitted to a low-pass filter, which removes high-frequency signals. The low-pass filter then converts the high-frequency-removed audio signal into a digital audio signal via an ADC / A2B bus. This digital audio signal is then input to an audio DSP for processing.

[0119] Optionally, in this embodiment, the low-pass filter filters high frequencies above 20kHz.

[0120] Optionally, the processing of audio signals can be implemented based on the aforementioned hardware system.

[0121] As an example, in the application scenario of intelligent networked vehicles, the hardware system for voice recognition is deployed in the car. As shown in Figure 2, a microphone (MIC) is placed inside the intelligent cockpit to collect audio signals of people speaking within the cockpit. When applied to a car, the driver or passengers can use voice recognition interaction, voice transcription, voice analysis, audio and video recording, and other functions within the vehicle. Through voice interaction control, the driver can focus on driving while simultaneously controlling the vehicle via voice, thereby reducing the risk of accidents caused by distracted operation of the in-vehicle system and improving driving comfort. Therefore, the embodiments of this application, when applied to automobiles, have better performance.

[0122] Figure 3 is an example diagram of the software system for the speech recognition method provided in this application embodiment. As shown in Figure 3, the software system can be configured within the main controller on the device side. The software system includes: a hardware abstraction layer (HAL layer), a service layer, a framework layer, and an application layer, wherein:

[0123] The Audio DSP performs digital noise reduction and data processing on the audio digital signal again, and sends it to the HAL (Hardware Abstraction Layer) of the AVN / HU / IVI SOC (Intelligent Cockpit Master Controller) system through the IIS (Inter-IC Sound) ( / PCM (Pulse Code Modulation)) / TDM (Time Division Multiplexing) / PDM (Pulse Density Modulation)) bus.

[0124] In the HAL layer, the audio digital signal is processed to identify the corresponding text. Specifically, a dual-path modeling algorithm is used to extract the speech feature sequence corresponding to the audio digital signal. Based on the pre-learned association probability between the speech feature sequence and the text, the corresponding text sequence is identified. The dual-path modeling algorithm can improve the accuracy of speech recognition, especially when the audio signal contains mixed audio. It can distinguish between different speech segments and predict and recognize the distinguished speech based on other speech segments, thus improving recognition accuracy.

[0125] Furthermore, in the HAL layer, a time-synchronized sequence conversion model is used to realize the recognition of streaming speech.

[0126] For details on the specific processing flow and implementation of speech signals in the HAL layer, please refer to the following embodiments.

[0127] Figure 4 is a flowchart illustrating a speech recognition method proposed in this embodiment. As shown in Figure 4, the method includes:

[0128] S401, Perform feature extraction on the mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects besides the target object;

[0129] A mixed audio stream is a stream of mixed audio captured by a microphone or microphone array, referring to continuously acquired audio. The speech recognition method in this application embodiment can process the mixed audio stream, enabling real-time processing and feedback of the mixed audio.

[0130] After acquiring the mixed audio stream, feature extraction is performed to obtain the first audio feature of the target object and the second audio features of other objects besides the target object. The target object refers to a speaking object in the mixed audio stream, which includes the speech of multiple objects; the target object is the one distinguished from these multiple objects. It should be noted that all audio features are sorted according to the temporal order of the mixed audio stream input, forming a time series. The first audio feature refers to the audio feature corresponding to the speech emitted by the target object at a certain time step in the time series extracted from the mixed audio stream. The second audio feature refers to the audio feature corresponding to the speech emitted by other objects at that time step. The first and second audio features are the audio features corresponding to speech emitted at the same time step.

[0131] As an example, features can be extracted from a mixed audio stream based on Mel-frequency cepstral coefficients (MFCCs) to distinguish different sound sources. The mixed audio stream can then be sequentially separated according to time sequence, and the separated objects can be used as target objects in turn.

[0132] For example, suppose a mixed audio stream contains the voices of two people, A1 and A2. If the audio features emitted by A1 are detected, A1 is identified as the target object, and the audio features corresponding to the voice emitted by A1 are extracted as the first audio feature. The audio features of other objects besides A1 are extracted as the second audio feature. When the voice emitted by A2 is detected, A2 is identified as the target object, and the audio features corresponding to the voice emitted by A2 are extracted as the first audio feature. The audio features of other objects besides A2 are extracted as the second audio feature. By distinguishing the voices of each object in turn, the speech of each object can be differentiated from the mixed audio stream, and then the speech of each object can be recognized.

[0133] Suppose the mixed audio stream contains the speech of three or more people, designated as object A1, object A2, ..., object An. If the audio features emitted by object A1 are identified, object A1 is determined to be the target object. The audio features corresponding to the speech emitted by object A1 are extracted as the first audio feature. The audio features of the other objects (A2 to An) are extracted as the second audio features. At this point, whether based on the second audio feature or the mixed audio features, the mixed audio stream is updated with the second audio feature, and speech recognition in the mixed audio stream continues.

[0134] As an example, referring to Figure 5, the speech recognition method in this embodiment uses an end-to-end speech recognition model for recognition. In this example, the speech recognition method is applied to a computing device with a deployed speech recognition model, wherein the speech recognition model includes a masking network, wherein:

[0135] Masking networks are used to extract features from mixed audio streams, obtaining the first audio features of the target object and the second audio features of other objects besides the target object.

[0136] This embodiment extracts audio features of different objects using a masking network to distinguish their audio features. Optionally, the masking network includes a masking encoder that predicts a mask based on the mixed audio stream. This mask reflects the contributions of different speakers in the mixed audio stream, thereby highlighting the audio features of one object and suppressing the audio features of another. Therefore, the audio features of one object can be extracted using the mask, thus distinguishing the audio features of different objects.

[0137] Alternatively, S401 can specifically be:

[0138] Predict the mask of the speech emitted by the target object based on the mixed audio stream;

[0139] Based on the mask, the mixed audio stream, and the two's complement of the mask, the mixed audio stream is processed to obtain the first audio feature of the target object and the second audio feature of other objects besides the target object.

[0140] In the speech recognition model training, the mask encoder learns predictive codes to distinguish individual sound sources in the mixed audio stream. Therefore, the mask encoder can predict the mask of the speech emitted by the target object, and use this mask to extract features from the mixed audio stream, obtaining the first audio feature. Audio features other than the first audio feature are then called the second audio feature.

[0141] For example, a mask M is predicted based on the mixed audio stream to correspond to the speech emitted by the target object. The mask value can be selected as [0,1], and the complement of the mask is 1-M. The first audio feature corresponding to the target object is determined by the mask, specifically the first audio feature H1. Correspondingly, the second audio feature corresponding to other objects besides the target object is determined by the mask. Specifically, the second audio feature is H2. in, For mixed audio streams, or mixed audio features after preprocessing of mixed audio streams, such as:

[0142] As an example, to further improve the accuracy of feature extraction, after obtaining the mixed audio stream, it is preprocessed to obtain mixed audio features. Preprocessing includes converting the mixed audio stream into a format or feature set for further processing.

[0143] Alternatively, in examples applied to speech recognition models, the masking network also includes a preprocessing encoder (MixEnc), which processes the mixed audio stream to obtain mixed audio features.

[0144] Optionally, the preprocessing encoder's processing includes: frame segmentation, window function, fast Fourier transform, power spectral density, Mel filter bank, logarithmic compression, pooling, deep learning feature extraction, batch normalization, activation function, etc., to extract features and obtain mixed audio features.

[0145] As an example, a preprocessing encoder preprocesses a mixed audio stream using the following formula:

[0146] Where t is time; x(t) is continuous audio; ω(t) is the window function; h(f′,f) is the response of the Mel filter at frequency f′ corresponding to the Mel frequency f; ε is a constant; σ 2 denoted as variance; μ is the batch mean.

[0147] It should be noted that in the above formulas, ReLU is the activation function; pool is the pooling function; log is the logarithm; and STFT is the Short-Time Fourier Transform.

[0148] After preprocessing the mixed audio stream based on the above formula, MixkEnc(X) is obtained, corresponding to the mixed audio features. for:

[0149] As an example, a mask encoder uses the following formula to analyze mixed audio features. Perform masking: MaxkEnc(X) = W f φ(Conv (l) (H (l-1) )+b f ;

[0150] Among them, H (l) The audio features output by layer l of the convolutional layer; H (0) φ represents the input features (audio features) received by the mask encoder; φ is the activation function; Conv is the convolutional neural network; W f The weight b of the fully connected layer f This is the bias for the fully connected layer.

[0151] It should be noted that in the above formula, MaskEnc convolutional neural network (CNN) is used to learn feature representations of different speakers in a mixed audio stream.

[0152] After preprocessing the mixed audio stream based on the above formula, MaxkEnc(X) is obtained, and the prediction expression for the mask of the corresponding target object is:

[0153] Where X represents the mixed audio feature. The sigmoid activation function is used to map the output of MaskEnc to the (0,1) interval, ensuring that the element values ​​of the mask are reasonable.

[0154] The mask is a two-dimensional weight matrix. The first audio feature is obtained by element-wise multiplication of the mask and the mixed audio stream. The second audio feature is obtained by element-wise multiplication of the mask and the mixed audio stream. The first audio feature represents the audio characteristics of the target object, and the second audio feature represents the audio characteristics of other objects besides the target object.

[0155] In this embodiment, when preprocessing the mixed audio stream, time-series encoding is also performed on the mixed audio stream. The first audio feature obtained by masking and the mixed audio stream, and the second audio feature obtained by complement code and the mixed audio stream, are time-synchronized features. This embodiment is based on the first and second audio features of time synchronization for identification.

[0156] S402, Based on the first audio features, obtain the first feature vector of the speech emitted by the target object;

[0157] In this step, the first audio feature is processed to obtain the relationship between the first audio feature and the context, the dependency relationship with the text, and the time series, forming a first feature vector, so that the corresponding text can be more accurately identified through the first feature vector.

[0158] As an example, referring to Figure 5, the speech recognition model also includes a Long Short-Term Memory (LSTM) network. This LSM network is used to obtain a first feature vector of the speech emitted by the target object based on the first audio features. By recognizing, classifying, and processing the first audio features in real time through the LSM network, capturing time dependencies and change patterns, and modeling and predicting the first audio features, continuous and dynamic processing and recognition are achieved, enabling the recognition of streaming speech.

[0159] S403, based on the second audio features, predict the second feature vector associated with the speech that the target object may emit;

[0160] In this step, a second feature vector is predicted by analyzing the second audio features to identify possible speech patterns emitted by the target object. This second feature vector includes the relationship between the second audio features and the context, their dependence on the text, and time series data.

[0161] The second audio feature is correlated with the speech (first audio feature) that the target object may emit: the first and second audio features are audio features corresponding to the same time step. In the same scene, the speech emitted by different objects has a certain correlation. In this embodiment, this correlation is defined as a positive correlation feature that assists in the recognition of the speech that the target object may emit. For example, in a meeting scene, different objects speak on the same topic, and the speech emitted at the same time step is correlated. This embodiment is applied to a mixed audio stream, including the speech emitted by objects A1 and A2. Based on the fact that the speech content of objects A1 and A2 must have a correlation, the correlation between the speech of different objects in the mixed audio stream in the same scene can be set. After recognizing the audio feature of object A1 as the first audio feature, the speech that object A1 may emit is inferred through the audio feature of object A2, and then the second feature vector corresponding to the speech is obtained.

[0162] For example, if object A1 is the target object, the text corresponding to the first audio feature might be "you". If the text corresponding to the second audio feature might be "hi", then the predicted speech of object A1 is "you". Combining the first and second feature vectors increases the accuracy of the first audio feature recognition. Similarly, if A2 is the target object, feature processing is performed in the same way.

[0163] Alternatively, in a mixed audio stream, including speech emitted by objects A1 and A2, the probability of the corresponding text is predicted by recognizing the audio features of object A1, the probability of the corresponding text is predicted by recognizing the audio features of object A2, and then the probability of the text emitted by object A1 is predicted by recognizing the audio features of object A2. The probability of the text emitted by object A1 is then combined with the probability of the text emitted by object A2 to predict the probability of the final target text. The prediction method in this example is similar to or equivalent to the method described above.

[0164] For example, if A1 is the target object, the text corresponding to the first audio feature might be "you". If the text corresponding to the second audio feature might be "hi", then combining the first feature vector corresponding to the speech of "you" and the second feature vector corresponding to the speech of "hi" increases the recognition accuracy of the first audio feature. Similarly, if A2 is the target object, feature processing is performed in the same way.

[0165] Optionally, the association between the second audio feature and the speech that the target object may emit (the first audio feature) can also be: the second audio feature can also be the background noise corresponding to the speech that the target object may emit. By recognizing the background noise, the difference in the speech that the target object may emit is increased, thereby improving the accuracy of speech recognition.

[0166] As an example, referring to Figure 5, the speech recognition model also includes a prediction network that predicts a second feature vector associated with the speech that the target object may emit, based on the second audio features. During the processing of the second audio features, the prediction network uses time-series synchronous processing to predict the speech that the target object may emit at the same time step, thereby improving the accuracy of speech recognition.

[0167] S404, based on the first feature vector and the second feature vector, identify the target text corresponding to the speech emitted by the target object in the mixed audio stream.

[0168] As an example, the first text prediction probability corresponding to the first feature vector can be obtained through the mapping relationship between the first feature vector and the text; the second text prediction probability corresponding to the first feature vector can be obtained through the mapping relationship between the second feature vector and the text; and then the target text can be determined by combining the first text prediction probability and the second text prediction probability.

[0169] As an example, the first and second feature vectors can be fused to obtain a fused feature vector; then, the target text can be identified based on the fused feature vector.

[0170] As an example, the speech recognition model also includes a joint network for identifying the target text corresponding to the speech emitted by the target object in the mixed audio features based on a first feature vector and a second feature vector.

[0171] As an example, this application embodiment processes a mixed audio stream to identify the speech of different objects. For instance, when the mixed audio stream includes the speech of two people, one object is identified as the target object. After S401-S404, when the target text corresponding to the speech of the target object is identified, the second audio feature is used as the mixed audio feature, and the process of S401-S404 is repeated to identify another object as the target object and identify the target text corresponding to that object. When the mixed audio stream includes the speech of three or more people, one object is identified as the target object. After S401-S404, when the target text corresponding to the speech of the target object is identified, the mixed audio feature is updated based on the second audio feature, and the process of S401-S404 is repeated to identify another object as the target object and identify the target text corresponding to that object. Then, the mixed audio feature is updated based on the second audio feature, and the process of S401-S404 is repeated to identify yet another object as the target object and identify the target text corresponding to that object, until the speech of all objects in the mixed audio stream is identified, generating text corresponding to each object. As shown in Figure 9, X1 is the speech emitted by object 1 in the mixed audio stream, and Y1 is the text recognized corresponding to the speech emitted by object 1; X2 is the speech emitted by object 2 in the mixed audio stream, and Y2 is the text recognized by the speech emitted by object 2. The text corresponding to each object is output respectively.

[0172] In this embodiment, when extracting features from a mixed audio stream, the features of the mixed audio stream are extracted into a first audio feature and a second audio feature to distinguish the mixed audio stream. Then, it is divided into two processing paths to process the first audio feature and the second audio feature respectively. Specifically, the first audio feature is processed to obtain a first feature vector, which serves as the first path of feature processing; the second audio feature is processed to obtain a second feature vector, which serves as the second path of feature processing. By fusing the first feature vector and the second feature vector, the target text is finally identified. First, the mapping probability between the first audio feature and the target text is calculated through the first path, and the mapping probability between the first audio feature and the target text is predicted through the second path. The prediction results of the two paths are fused to improve the accuracy of speech recognition. Alternatively, it can be understood that in the same voice interaction scenario, the voices of other objects in the mixed voice stream have a certain correlation with the voice of the target object. This embodiment predicts the mapping probability between the voice of the target object and the target text by adding a second feature vector associated with the voice that the target object may emit, which can improve the recognition accuracy of each target object in the mixed audio stream. Therefore, in the recognition scenario of mixed audio stream, the voice recognition accuracy is high and the voice recognition pauses during the recognition process can be reduced.

[0173] Figure 6 is a schematic diagram of an application flow of the speech recognition method provided in this application embodiment. As an example, as shown in Figure 6, the speech recognition scenarios in this example include, but are not limited to, speech control recognition, speech transcription, speech analysis, and audio / video recording. Different speech recognition scenarios correspond to different ways of outputting target text. Based on this, in practical applications, based on the selected speech recognition scenario, the first feature vector and the second feature vector are processed to identify the target text corresponding to the speech emitted by the target object in the mixed audio stream, and the target text is responded to according to the recognition scenario. For example, in the case of a speech control recognition scenario, the first feature vector and the second feature vector are identified based on the speech recognition scenario to obtain the target text reflecting the control command, and the control command corresponding to the target text is executed.

[0174] In the example, combining the specific context of speech recognition to identify mixed audio streams can increase the probability of text matching the corresponding attributes or features during the recognition process, based on the attributes or features of the speech recognition context, thereby further improving the accuracy of speech recognition.

[0175] Figure 7 is a flowchart illustrating the speech recognition method provided in this application embodiment, which is based on the above embodiment. Streaming speech technology can process and provide feedback in real time, improving the efficiency of human-computer interaction and user experience, and is a development trend in speech recognition. However, it also faces challenges in technical implementation and application scenarios. For example, in model-based speech recognition scenarios, the longer the input speech, the greater the model's recognition error. When the recognition error is large, the model cannot continue to recognize the speech. Therefore, in related technologies, after recognizing a certain length of audio, a pause is required before re-recognizing; for example, after recognizing a 30-second audio stream, a pause occurs, and these pauses are frequent. Based on this, this application embodiment reduces the number of pauses and extends the recognition length of the speech stream through the following embodiments.

[0176] As shown in Figure 7, the identification methods include:

[0177] S701, perform feature extraction on the mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects besides the target object;

[0178] S702, based on the first audio features, obtain the first feature vector of the speech emitted by the target object;

[0179] S703, based on the second audio features, predict the second feature vector associated with the speech that the target object may emit;

[0180] In this embodiment, the specific implementation process of steps S501 to S503 is similar to that of S401 to S403 in the embodiment of Figure 4 above, and will not be described again here.

[0181] S704, predict the backward probability of the speech that the target object may emit in the next time step based on the first audio features;

[0182] In this step, the forward probability refers to the probability that the speech emitted by the target object at the previous time step (t-1) corresponds to the first text, predicted at the current time step t. The backward probability refers to the probability that the speech emitted by the target object at the next time step (t+1) corresponds to the second text, predicted at the current time step t based on the first audio features. The second text can include multiple texts, and the predicted backward probability of each second text is different. The first text, the target text, and the second text are the texts corresponding to the speech emitted by the target object in sequence, forming a continuous text corresponding to the speech stream of the target object.

[0183] There are no restrictions on the execution order of this step with S702 and S703; they can be executed sequentially or simultaneously.

[0184] S705 predicts the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first eigenvector, the second eigenvector, the backward probability, and the forward probability through a preset conditional probability.

[0185] As an example, predicting the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first eigenvector, the second eigenvector, the backward probability, and the forward probability through a preset conditional probability is specifically as follows:

[0186] Fuse the first eigenvector and the second eigenvector to obtain a fused feature vector;

[0187] Predict the backward probability of the speech that the target object may emit at the next time step based on the fused feature vector;

[0188] Predict the target text corresponding to the speech emitted by the target object in the mixed audio stream through a preset conditional probability based on the forward probability predicted for the target object at the previous time step and the backward probability of the speech that may be emitted at the next time step.

[0189] As an example, the preset conditional probability is:

[0190] where α(t, u) is the forward probability, β(t, u) is the backward probability, P(X) is the normalization constant; P(Y|X) is the probability of predicting the target text based on the first eigenvector and the second eigenvector.

[0191] The target text corresponding to the speech is obtained through probability. In the above conditional probability, X is the speech, Y is the target text, P(Y|X) is the probability of the text corresponding to the speech. If the probability of the speech corresponding to a text is the highest, then determine this text as the target text. For example, if the probability of the speech corresponding to "Hello" is 98%, then the target text corresponding to the speech is "Hello".

[0192] During the model training process, establish the association relationship between the conditional probability and the forward probability and the backward probability (such as the above conditional probability). During the actual application process, after obtaining the fused feature vector by fusing the first eigenvector and the second eigenvector, predict the backward probability of the speech that the target object may emit at the next time step through the fused feature vector, and then through the above conditional probability, after predicting the probability of the target text, that is, the probability of the speech emitted at the current time step and the target text, predict through the forward probability predicted at the previous time step and the backward probability of the speech that may be emitted at the next time step. Based on the strong correlation between the previous and subsequent speeches in the streaming speech, this prediction method can improve the prediction accuracy at the current time step and the recognition accuracy.

[0193] Illustrate the relationship between the forward probability, backward probability, and the prediction result at the current moment. Assume that the mixed audio stream includes audio at time steps t1, t2, t3... tn. Suppose the target text predicted at time step t1 is "big"; when recognizing the audio at time step t2, the forward probability at this time is Q1. According to the fused feature vector based on the first audio feature and the second audio feature, predict the backward probability Q2 of the speech that the target object may emit at the next time step (t3). Then, based on Q1 and Q2, predict the probability of the target text, which is P2. For example, the probability of Q1 being "big" is the highest, and the target text corresponding to the fused feature vector may be "home". Through this feature vector, the probability that the text corresponding to the next speech of the target object is "good" is the highest. Then, based on the forward probability Q1 and the backward probability Q2, increase the prediction probability that the target text at the current time step t2 is "home". When recognizing the audio at time step t3, the P2 predicted based on t2 is the forward probability Q1 at time step t3. According to the fused feature vector of the first audio feature and the second audio feature, predict the backward probability Q2 of the speech that the target object may emit at the next time step corresponding to the text. Based on Q1 and Q2, predict the probability of the target text, which is P3. This P3 is the forward probability corresponding to time step t4. And so on. When recognizing the audio at time step tn, the Pn - 1 predicted based on the t - 1 moment is the forward probability Q1 at time step tn. According to the fused feature vector of the first audio feature and the second audio feature, predict the backward probability Q2 of the speech that the target object may emit at the next time step corresponding to the second text. Based on Q1 and Q2, predict the probability of the target text, which is Pn.

[0194] It should be noted that at time step t1, the forward probability Q1 is zero. At time step tn, the backward probability Q2 is zero, or when it is determined based on the temporal encoding of the mixed audio stream that there is no audio input after time step tn, there is no need to predict the forward probability of the second text corresponding to the speech that the target object may emit at the next time step of time step tn.

[0195] In this embodiment, first, based on the fused feature vector of the first feature vector and the second feature vector, predict the backward probability of the speech that may be emitted at the next time step of the current time step corresponding to the text. Then, use the forward probability and the backward probability to predict the target text, improving the prediction accuracy. Since at the current time step, the backward probability of the text at the next time step is predicted in advance, the speech recognition accuracy is high, and the speech recognition model can continuously recognize, reducing the pause frequency.

[0196] As an example, based on all the above embodiments, after recognizing the target text corresponding to the speech emitted by the target object in the mixed audio stream, it further includes:

[0197] When an anomaly is detected in the target text corresponding to the speech emitted by the target object at the current moment, the model parameters of the speech recognition model are optimized based on the anomaly scenario of the target text, the application field of speech recognition, and the correction text.

[0198] The corrected text refers to the corrected text that corresponds to the error message when the target text is abnormal.

[0199] Target text anomalies include, but are not limited to, the following scenarios:

[0200] 1. In a voice control scenario, after the target text is recognized, the control command indicated by the target text is sent to the corresponding execution terminal. If the corresponding execution terminal does not provide a response, or if the execution terminal that provides a response does not match the execution terminal corresponding to the voice instruction (in which case user feedback is required), then the target text is abnormal.

[0201] Taking a voice control scenario in a car as an example, suppose the mixed audio stream includes commands like "turn on the air conditioner" and "close the windows." The voice recognition model identifies the target text corresponding to the mixed audio stream as "turn on the air conditioner" and "lock the doors," and the car performs the actions of turning on the air conditioner and locking the doors. At this point, the user, based on the difference between the car's executed control and the user's voice control commands, will manually or verbally provide relevant feedback, and an anomaly in the target text will be detected.

[0202] 2. In speech-to-text scenarios, if the target text is in an unconventional font or differs significantly from the preceding and following text, then the target text is considered abnormal.

[0203] 3. In speech-to-text scenarios, if the target text does not match the application domain of speech recognition, such as in a vehicle application scenario, if the text contains medical terms, then the target text is considered abnormal.

[0204] Furthermore, by combining information such as the type and cause of the anomaly, the model parameters of the speech recognition model are optimized, resulting in more accurate recognition results.

[0205] Furthermore, embodiments of this application optimize the model parameters of the speech recognition model in the following ways:

[0206] The speech recognition model is trained based on the abnormal scenarios of the target text, the application domain of speech recognition, and the corrected text, and the loss of the speech recognition model is adjusted. The corrected text can be obtained from user feedback or predicted. The corrected text is used as a label to train the speech recognition model, and the loss of the speech recognition model is adjusted. Optionally, a smaller loss in the adjusted speech recognition model can improve the recognition accuracy.

[0207] Then, the parameters of the forward probability algorithm model and the backward probability algorithm model are adjusted based on the adjusted loss.

[0208] Optionally, the relationship between the loss of the speech recognition model and the forward and backward probabilities is as follows:

[0209] After Lrnnt(X, Y) is adjusted, the algorithm model parameters of the forward probability α(t, u) and the algorithm model parameters of the backward probability β(t, u) are also adjusted.

[0210] Based on the adjusted forward probability algorithm model parameters, predict the forward probability of the speech that the target object may emit at the previous moment before the current moment, and predict the backward probability of the speech that the target object may emit at the next moment after the current moment based on the adjusted backward probability algorithm model parameters.

[0211] Optionally, the forward probability is predicted as follows: α(t, u) = P(y[1: u], x[1: t]).

[0212] Optionally, the backward probability is predicted as follows: β(t,u)=P(x[t:T],y[u:U]|y[(u-1),(t-1)]).

[0213] The gradient is calculated based on the forward probability, backward probability, and corrected text.

[0214] Alternatively, the gradient can be calculated as follows:

[0215] Furthermore, the model parameters of the speech recognition model are adjusted based on gradients.

[0216] This embodiment uses the above method to quickly optimize the model parameters of the speech recognition model.

[0217] Optionally, in this embodiment, as shown in Figure 6, the model parameters of the speech recognition model are optimized in the cloud. Optimizing the model parameters of the speech recognition model in the cloud, based on the real-time OTA capability of the cloud and the terminal, and by deploying it in the cloud, can reduce the computing power of the terminal and avoid affecting the normal use of other functions of the terminal when optimizing the model parameters.

[0218] It is understood that the speech recognition model in this application is based on a heuristic error allocation method, using heuristic rules to quickly select the optimal model parameters. Specifically, during model training, a basic model is pre-trained using limited labeled data. Compared to the conventional approach of relying on complete labeled data for model training, this application can reduce data acquisition costs and training difficulty.

[0219] Then, this basic model is applied to the endpoint. By evaluating the identification results of this model during its use at the endpoint, the optimality of the current solution is determined.

[0220] Through statistical analysis of application domains and error patterns, and based on error types, causes, and scenarios, new model parameters are obtained using heuristic rules. If these model parameters improve the accuracy of the speech recognition model—that is, if the current solution is better—then the speech recognition model is updated. In this continuous process, the model is gradually improved using heuristic error training methods, learning and reducing similar errors during correction, thus gradually increasing the accuracy of the speech recognition model. Within the same application domain, through continuous learning, uninterrupted streaming speech recognition can be achieved to a certain extent.

[0221] In this embodiment, the heuristic rule for obtaining new model parameters can refer to the methods listed above, that is, calculating the gradient through forward probability and backward probability, and then reflecting the direction of modification of model parameters based on the gradient, thereby obtaining the modified model parameters.

[0222] In this embodiment, when anomalies occur in the recognition of the target text, the model parameters of the speech recognition model are optimized by considering the abnormal scenario of the target text, the application field of speech recognition, and the corrected text, thereby improving the recognition accuracy of the speech recognition model. It should be noted that, in the application process of this embodiment, upon encountering anomalies, the model begins learning how to obtain the minimum loss of the speech recognition model under abnormal conditions, and then continuously corrects the speech recognition model, rather than correcting or re-recognizing the speech after the gradient of the speech recognition model explodes. Therefore, compared with related technologies, this embodiment, based on continuous optimization and correction of the speech recognition model, can achieve increasingly higher accuracy of the speech recognition model, thus enabling the recognition of indeterminate, continuous speech.

[0223] Furthermore, if a speech recognition model is migrated to a new application area, it can be gradually adjusted based on heuristic error allocation rules to adapt to the characteristics of the new area, making the speech recognition model easy to adapt to various fields.

[0224] The speech recognition model mentioned in the above embodiments is obtained after training using the speech recognition model training method provided in this application embodiment. Referring specifically to Figure 8, the training method in this embodiment includes:

[0225] S801, Obtain training samples, which include the mixed audio stream of samples and the sample labels corresponding to the mixed audio stream of samples;

[0226] In this step, the sample label is the actual target text (hereinafter referred to as the sample label). It can be the text annotation of the sample mixed audio stream or the text set according to the mixed sample audio stream.

[0227] S802, the sample mixed audio stream is input into the speech recognition model. In the speech recognition model, the following processing is performed to identify the target sample label corresponding to the speech emitted by the target object in the sample mixed audio stream: feature extraction is performed on the sample mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects besides the target object; based on the first audio feature, the first feature vector of the speech emitted by the target object is obtained; based on the second audio feature, the second feature vector associated with the speech that the target object may emit is predicted; based on the first feature vector and the second feature vector, the target sample label is identified.

[0228] In this embodiment, the speech recognition model includes a masking network, a long short-term memory (LSTM) network, a prediction network, and a joint network. The masking network extracts features from the mixed audio stream of samples to extract first and second audio features from different speakers. Two prediction paths are formed by the LTM network and the prediction network. The LTM network is the first path (main path), used to predict the corresponding first feature vector based on the first audio features of the target object. The prediction network is the second path (auxiliary path), used to predict the second feature vector of the speech that the target object may emit based on the second audio features. In other words, the LTM network and the prediction network form a dual prediction path, making the speech recognition model a dual-path model. After obtaining the first and second feature vectors based on the dual prediction paths, the joint network fuses and predicts the first and second feature vectors to obtain the probability of the target text.

[0229] Based on the structure of this speech recognition model, after the sample mixed audio stream is input into the speech recognition model, the sample mixed audio stream is first input into a masking network, and the masking network performs masking processing on the sample mixed audio stream. Specifically, in some embodiments, the masking network includes a masking encoder. The sample mixed audio stream is input into the masking encoder, and the sample mixed audio stream is processed based on the masking encoder to predict the mask of the speech emitted by the target object. According to the mask, the sample mixed audio stream, and the complement of the mask, the sample mixed audio stream is processed to obtain the first audio feature of the target object and the second audio feature of other objects besides the target object, so as to distinguish the audio features of different objects.

[0230] Alternatively, in some embodiments, the masking network includes a preprocessing encoder and a mask encoder. The sample mixed audio stream is input to the preprocessing encoder, which preprocesses the stream to obtain sample mixed audio features. These features are then input to the mask encoder, which processes them to predict a mask of the speech emitted by the target object. Based on the mask, the sample mixed audio features, and the mask's complement, the features are further processed to obtain a first audio feature of the target object and second audio features of other objects, thus distinguishing the audio features of different objects.

[0231] As an example, the first audio feature of the target object is obtained based on the mask and sample mixed audio features; the second audio feature is obtained based on the complement code and sample mixed audio features.

[0232] After the mixed audio stream is processed by a masking network, feature extraction is performed, resulting in first and second audio features. The first audio features are then input into a Long Short-Term Memory (LSTM) network for feature extraction, yielding a first feature vector representing the speech emitted by the target object. The second audio features are input into a prediction network, which predicts the possible speech emitted by the target object corresponding to the second audio features, generating a corresponding second feature vector. By processing the first and second audio features through dual paths, the audio features of a single target object in the mixed audio stream can be separated to obtain the first feature vector. Simultaneously, the second feature vector can be predicted based on the second audio features of other objects in the mixed audio stream, increasing the probability between the first feature vector and the target text, thereby improving recognition accuracy.

[0233] The first audio feature is processed by a long short-term memory network to obtain a first feature vector, and the second audio feature is processed by a prediction network to obtain a second feature vector. Then, the first and second feature vectors are input into a joint network, and the first and second feature vectors are fused based on the joint network to obtain a fused feature vector. Based on the fused feature vector, the target text corresponding to the speech emitted by the target object in the sample mixed audio stream is identified.

[0234] As an example, before the first audio feature is input into the Long Short-Term Memory (LSTM) network, it needs to be encoded along with the sample labels to facilitate the time-synchronized mapping of audio features and sample labels. Therefore, the speech recognition model also includes an audio encoder and a label encoder. The audio encoder encodes the first audio feature, and the label encoder encodes the sample labels (i.e., the label samples). Specifically, the audio encoder encodes the first audio feature based on time sequence, and the label encoder also encodes the sample labels based on time sequence, thus obtaining time-synchronized first audio features and sample labels. For example, the audio features corresponding to the mixed audio stream are X = {x1, x2, ..., x...}. t The corresponding sample labels are Y = {y1, y2, ..., y}. u}, where t is the temporal length of the mixed audio stream and u is the length of the sample label.

[0235] In this embodiment, during the processing of the mixed audio stream of samples by the speech recognition model, the sample labels are synchronously converted based on time series. During time synchronization, the original time series data is converted into a feature vector representation. The Long Short-Term Memory (LSTM) network then identifies, classifies, and processes the data in real time, capturing time dependencies and patterns of change. This continuous and dynamic processing and recognition of time series data improves the accuracy and reliability of predictions. It also provides the possibility of deploying the speech recognition model on the edge.

[0236] S803, calculates the loss value of the target text relative to the corresponding sample label based on the total loss function;

[0237] S804 adjusts the model parameters of the speech recognition model based on the loss value.

[0238] It should be noted that, in this embodiment, the mapping relationship between the target text and the mixed audio stream is represented by the association probability. For example, if the association probability between the speech emitted by the target object at the current time step and the text "you" in the mixed audio stream is A%, which is the maximum probability, then the text "you" is determined to be the target text corresponding to the speech emitted by the target object at the current time step. This embodiment calculates the loss value of the target text relative to the corresponding sample label using a total loss function, and then continuously adjusts the magnitude of the association probability based on the loss function, resulting in high recognition accuracy of the speech recognition model.

[0239] As an example, a total loss function is constructed based on the error between the target text and the target sample label predicted by the first audio feature and the error between the target text and the target sample label predicted by the second audio feature.

[0240] Optionally, the total loss function is: L heat (X,Y1,Y2)=L rnnt(Y1,H1)+L rnnt (Y2,H2);

[0241] Where X is the mixed audio stream, H1 is the first audio feature, H2 is the second audio feature, Y1 is the first sample label corresponding to the first audio feature, and Y2 is the second sample label corresponding to the second audio feature; Lrnnt is the loss function between the input audio feature H and the corresponding sample label Y; L heat This is the loss value.

[0242] It should be noted that the total loss function reflects a heuristic error allocation method, specifically in heuristic error allocation (where a label is selected based on heuristic information). The model is trained to minimize L. heat The model learns to match feature representations H1 and H2 with their corresponding label sequences (target text sequences) Y1 and Y2, respectively. The mixed audio stream X obtains H1 and H2 based on masking. The error between the target text predicted by H1 and the first sample label Y1, and the error between the target text predicted by H2 and the second sample label Y2 are calculated using probability conditions. The total loss function reflects the sum of these errors.

[0243] When the loss value is greater than or equal to the preset threshold, the model parameters of the speech recognition model are adjusted so that the loss value can be reduced when learning the next training sample, thereby improving the recognition accuracy of the speech recognition model.

[0244] In this embodiment, the speech recognition model extracts features from the mixed audio stream into first audio features and second audio features during feature extraction, thereby distinguishing the mixed audio stream. Then, it divides the process into two processing paths to process the first and second audio features respectively. The first audio features are processed to obtain a first feature vector, which serves as the first feature processing path; the second audio features are processed to obtain a second feature vector, which serves as the second feature processing path. By fusing the first and second feature vectors, the target text is finally identified. By learning the association between the first and second audio features and the text, and simultaneously learning the association between the second and second audio features, the probability of learning the relationship between the mixed audio stream and the text is increased through these two paths, thereby improving the recognition accuracy of each target object in the mixed audio stream.

[0245] In a further embodiment, the training method for the speech recognition model further includes: predicting the backward probability of the speech that the target object may emit at the next time step corresponding to the second sample label based on the first audio features;

[0246] Based on the first feature vector, the second feature vector, the backward probability, and the forward probability of the speech emitted by the target object corresponding to the first sample label predicted in the previous time step, the target text is predicted by a preset conditional probability.

[0247] In other words, in this embodiment, the target text corresponding to the speech emitted by the target object in the mixed audio stream of the sample is comprehensively identified by combining the first feature vector, the second feature vector, the backward probability, and the forward probability, so as to improve the accuracy of speech recognition.

[0248] Optionally, the preset conditional probability is:

[0249] Where α(t, u) is the forward probability, β(t, u) is the backward probability, P(X) is the normalization constant, and P(Y|X) is the probability of predicting the target text based on the first feature vector and the second feature vector.

[0250] In this example, the RNNT model, consisting of an audio encoder, a label encoder, a Long Short-Term Memory (LSTM) network, and a joint network, models the conditional probability relationship between audio features (acoustic features) and sample labels (target text). It directly maps the acoustic feature sequence to the label sequence (target text sequence) and performs the conversion synchronously. The training and decoding processes of the model both rely on the calculation of conditional probabilities.

[0251] As an example, a speech recognition model also includes an audio encoder and a label encoder, and the training methods also include:

[0252] Obtain training samples, which include the mixed audio stream of samples and the sample labels corresponding to the mixed audio stream of samples;

[0253] In this step, the sample label is the actual target text (hereinafter referred to as the sample label). It can be the text annotation of the sample mixed audio stream or the text set according to the mixed sample audio stream.

[0254] The sample mixed audio stream is input into the speech recognition model. In the speech recognition model, the following processing is performed to identify the target text corresponding to the speech emitted by the target object in the sample mixed audio stream: feature extraction is performed on the sample mixed audio stream to obtain the first audio feature of the target object and the second audio feature of other objects besides the target object; based on the first audio feature, the first feature vector of the speech emitted by the target object is obtained; based on the second audio feature, the second feature vector associated with the speech that may be emitted by the target object is predicted; based on the first feature vector and the second feature vector, the target text is identified.

[0255] Based on the above, in this embodiment, the training process of the speech recognition model can be as follows: the system is powered on, the MIC receives the mixed acoustic sequence in the cockpit, and the preprocessing encoder (MixEnc) extracts features. The mask encoder (MaskEnc) estimates a mask M. Using the mask M and the preprocessed features, the feature representations H1 and H2 of the two speakers in the overlapping audio can be calculated. The overlapping audio speech acoustic feature sequence X = {x1, x2, ..., x...} is set. t} and its corresponding label sequence (target text sequence) Y = {y1, y2, ..., y u} where t is the length of the acoustic sequence and u is the length of the label sequence (target text sequence). The label sequence (target text sequence) Y is converted into an embedded representation label encoder. Dual-path modeling is used in single-person and multi-person audio environments, and under different signal-to-noise ratio conditions. By capturing the sound features and temporal dependencies between human voices, local features and global contextual information of the speech signal are captured simultaneously. The audio encoder and label encoder networks calculate the forward probability α(t, u) and the backward probability β(t, u) at each time step. The probabilities of all possible label sequence (target text sequence) paths (including blank markers) are calculated. The normalized probability distribution is calculated using the forward and backward probabilities. The RNNT model of the audio encoder, label encoder, Long Short-Term Memory (LSTM) network and joint network models the conditional probability relationship between acoustic features and label sequences (target text sequences) with time-synchronized sequence transformation. It maps the acoustic feature sequence to the label sequence (target text sequence) and synchronously converts the audio features into text to predict the corresponding label sequence (target text sequence).

[0256] As an example, the speech recognition model also performs the following processing: inputting a first audio feature into an audio encoder, encoding the first audio feature based on the audio encoder, encoding sample labels according to a label encoder, predicting the backward probability of the second sample label corresponding to the speech that the target object may emit in the next time step based on the first audio feature; and predicting the gradient of the model parameters corresponding to the loss function based on the backward probability and the forward probability of the speech emitted by the target object corresponding to the first sample label predicted in the previous time step.

[0257] As an example, the gradient is calculated using the following formula:

[0258] Where, θ and These are the model parameters; Lrnnt is the loss function between the input audio feature H and the corresponding sample label Y; α(t, u) is the forward probability, and β(t, u) is the backward probability.

[0259] The loss value of the target text relative to the corresponding sample label is calculated based on the total loss function; the order of this step and the gradient prediction step above is not restricted, and they can be executed simultaneously or sequentially.

[0260] When the loss value is greater than or equal to a preset threshold, the model parameters of the speech recognition model are adjusted according to the gradient.

[0261] In this example, the gradient reflects the sensitivity of the difference (loss value) between the target text and the sample labels recognized by the speech recognition model to changes in model parameters (partial derivatives of the model parameters). It guides how to adjust the model parameters to minimize the total loss function. By calculating the gradient and then adjusting the model parameters based on it, on the one hand, the difference between the target text and the sample labels recognized by the speech recognition model can be kept within a small range, allowing the speech recognition model to continuously process sequences of variable length (streaming speech) and avoid pausing for 30 seconds when the probability of recognition is low. On the other hand, it enables the speech recognition model to quickly learn the correlation between the mixed audio stream and the sample labels, reducing the number of training samples and improving learning efficiency.

[0262] In this embodiment, the forward probability α(t, u) represents the probability of observing sample audio features x[1:t] and sample labels y[1:u] before time step t and sample label u, and is defined as the forward probability. Optionally, the forward probability is predicted by the following formula: α(t, u) = P(y[1:u], x[1:t]);

[0263] The backward probability β(t,u) represents the probability of observing the sample audio feature x[t:T] and sample label y[u:U] after time step t and sample label u, and is defined as the backward probability. Optionally, the backward probability is predicted by the following formula: β(t,u)=P(x[t:T],y[u:U]|y[(u-1),(t-1)]);

[0264] In this embodiment, the total loss function is: L heat (X,Y1,Y2)=L rnnt (Y1,H1)+L rnnt (Y2,H2);

[0265] Where X is the mixed audio stream, H1 is the first audio feature, H2 is the second audio feature, Y1 is the first sample label corresponding to the first audio feature, and Y2 is the second sample label corresponding to the second audio feature; Lrnnt is the RNNT loss function, which is the loss function between the input audio feature H and the corresponding sample label Y; L heat This is the loss value.

[0266] In this example, the total loss function is composed of Lrnnt(Y1, H1) + Lrnnt(Y2, H2), where H1 represents the first audio feature extracted from the mixed audio stream, H2 represents the second audio feature extracted from the sample audio stream, Y1 is the sample label identified based on the first audio feature, Y2 is the sample label identified based on the second audio feature, Lrnnt(Y1, H1) represents the loss value based on the probability of identifying the first sample label based on the first audio feature, and Lrnnt(Y2, H2) represents the loss value based on the probability of identifying the second sample label based on the second audio feature. For the mixed audio stream, the total loss value is calculated based on the loss values ​​of different objects. On the one hand, this ensures that the speech recognition model balances the learning efficiency of different objects during the learning process, resulting in balanced recognition accuracy for each object. On the other hand, it allows for more detailed model tuning during training to reduce specific types of errors, resulting in good scalability, high efficiency, and avoids considering all possible label permutations, thereby reducing computational and memory requirements.

[0267] Alternatively, Lrnnt can be calculated using the following formula:

[0268] Where α(t, u) is the forward probability, β(t, u) is the backward probability, U is the length of the sample label, and u is the sample label corresponding to the current time step.

[0269] In this example, the loss between the input audio features and the sample labels is calculated by combining forward and backward probabilities. This allows the loss to be updated during the speech recognition model's learning process. As a result, the learned speech recognition model has a smaller recognition error in application and can learn longer audio streams.

[0270] Based on the embodiments of this application, the trained speech recognition model can realize timely recognition and response to mixed audio streams, avoiding the need to pause after recognizing an audio stream of a certain length.

[0271] It should be noted that in this embodiment, a base model is pre-trained using a limited number of sample labels (allowing for errors), reducing the number of training samples and lowering data acquisition costs. Furthermore, the base model has low computational requirements, making it easy to deploy on the edge, such as in automobiles, where the speech recognition model can be applied to the car's electronic devices.

[0272] During the use of the speech recognition model, the model parameters are updated using heuristic error allocation rules. For details, refer to the update process described in the above embodiments. The model parameters are updated based on specific abnormal situations during use, making the model more suitable for speech recognition in application scenarios and improving recognition accuracy.

[0273] This application also provides a voice recognition device, including:

[0274] The extraction module is used to extract features from the mixed audio stream to obtain the first audio features of the target object and the second audio features of other objects besides the target object;

[0275] The first prediction module is used to obtain the first feature vector of the speech emitted by the target object based on the first audio features;

[0276] The second prediction module is used to predict the second feature vector of the speech that the target object may emit based on the second audio features.

[0277] The recognition processing module is used to identify the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first feature vector and the second feature vector.

[0278] The speech recognition device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0279] The embodiment also provides a training device for a speech recognition model, including:

[0280] The acquisition module acquires training samples, which include the mixed audio stream of samples and the corresponding sample labels of the mixed audio stream.

[0281] A speech recognition model is used to perform the following processing on an input sample mixed audio stream: Recognize the target text corresponding to the speech emitted by a target object in the sample mixed audio stream; extract features from the sample mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects besides the target object; obtain a first feature vector of the speech emitted by the target object based on the first audio feature; predict a second feature vector of the speech that the target object may emit based on the second audio feature; and recognize the target text based on the first and second feature vectors.

[0282] The training module is used to calculate the loss value of the target text relative to the corresponding sample label based on the total loss function, and adjust the model parameters of the speech recognition model according to the loss value.

[0283] The speech recognition model training device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0284] This application also provides an electronic device, which includes: a memory and a processor;

[0285] The memory is used to store computer programs / instructions; the processor is used to execute the computer programs / instructions stored in the memory to implement the methods involved in the above embodiments.

[0286] This electronic device also includes a communication interface and a CAN bus. The processor provides computing and control capabilities and can be a GPU, CPU, NPU, MCU, FPGA, etc. Storage devices include internal memory and non-volatile memory. The non-volatile memory stores the computer programs implementing the above methods. Internal memory provides the environment for program startup and execution. The communication interface is used for wired or wireless communication with external terminals.

[0287] In this embodiment, the electronic device can be an electronic component in a car, or it can be the car itself.

[0288] The present invention also provides a computer-readable storage medium / computer program product, wherein the computer-readable storage medium stores computer-executable instructions / computer program product including computer-executable instructions, which, when executed by a processor, are used to implement the methods involved in the above embodiments.

[0289] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A speech recognition method, characterized in that, The method includes: Feature extraction is performed on the mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects besides the target object. The mixed audio stream includes the speech of at least two objects, and the target object is an object distinguished from the at least two objects. Based on the first audio feature, a first feature vector of the speech emitted by the target object is obtained; Based on the second audio features, a second feature vector associated with the speech that the target object may emit is predicted; Based on the first feature vector and the second feature vector, the target text corresponding to the speech emitted by the target object in the mixed audio stream is identified.

2. The identification method according to claim 1, characterized in that, The step of identifying the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first feature vector and the second feature vector includes: The first feature vector and the second feature vector are fused in time synchronization to obtain a fused feature vector; Based on the fused feature vector, the target text corresponding to the speech emitted by the target object in the mixed audio stream is identified.

3. The identification method according to claim 1, characterized in that, The identification method further includes: Predict the backward probability of the speech that the target object may emit in the next time step based on the first audio feature; The step of identifying the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first feature vector and the second feature vector includes: Based on the first feature vector, the second feature vector, the backward probability, and the forward probability of the speech emitted by the target object predicted in the previous time step, the target text corresponding to the speech emitted by the target object in the mixed audio stream is predicted by a preset conditional probability.

4. The identification method according to claim 1, characterized in that, The step of extracting features from the mixed audio stream to obtain the first audio features of the target object and the second audio features of other objects besides the target object includes: Preprocess the mixed audio stream to obtain mixed audio features; Predict the mask of the speech emitted by the target object based on the mixed audio features; Based on the mask and the mixed audio features, the first audio feature of the target object is obtained; Based on the complement of the mask and the mixed audio features, the second audio features of other objects besides the target object are obtained.

5. The identification method according to any one of claims 1 to 4, characterized in that, This is applied to computing devices equipped with a speech recognition model, which includes a masking network, a long short-term memory network, a prediction network, and a joint network, wherein: The masking network is used to extract features from the mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects besides the target object; The long short-term memory network is used to obtain a first feature vector of the speech emitted by the target object based on the first audio feature; The prediction network is used to predict a second feature vector associated with the speech that the target object may emit, based on the second audio features. The joint network is used to identify the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first feature vector and the second feature vector.

6. The identification method according to claim 5, characterized in that, After identifying the target text corresponding to the speech emitted by the target object in the mixed audio stream, the method further includes: When it is detected that the target text corresponding to the speech emitted by the target object is abnormal at the current moment, the model parameters of the speech recognition model are optimized based on the abnormal scenario of the target text, the application field of speech recognition, and the corrected text, wherein the corrected text is the corrected text corresponding to the abnormal target text.

7. The identification method according to claim 6, characterized in that, The optimization of the speech recognition model parameters based on the abnormal scenarios of the target text, the application field of speech recognition, and the corrected text includes: The speech recognition model is trained based on the abnormal scenarios of the target text, the application field of speech recognition, and the corrected text, and the loss of the speech recognition model is adjusted. Adjust the parameters of the forward probability algorithm model and the backward probability algorithm model based on the adjusted loss; Based on the adjusted forward probability algorithm model parameters, predict the forward probability of the speech that the target object may emit at the time before the current time, and predict the backward probability of the speech that the target object may emit at the time after the current time. The gradient is calculated based on the forward probability, the backward probability, and the corrected text. The model parameters of the speech recognition model are adjusted based on the gradient.

8. The identification method according to claim 6 or 7, characterized in that, The model parameters of the speech recognition model are optimized in the cloud.

9. A method for training a speech recognition model, characterized in that, include: Obtain training samples, which include a sample mixed audio stream and sample labels corresponding to the sample mixed audio stream; The sample mixed audio stream is input into a speech recognition model. The speech recognition model performs the following processing to identify the target text corresponding to the speech emitted by the target object in the sample mixed audio stream: Feature extraction is performed on the sample mixed audio stream to obtain a first audio feature of the target object and second audio features of other objects besides the target object; based on the first audio feature, a first feature vector of the speech emitted by the target object is obtained; based on the second audio feature, a second feature vector associated with possible speech emitted by the target object is predicted; based on the first feature vector and the second feature vector, the target text is identified. The loss value of the target text relative to the corresponding target sample label is calculated based on the total loss function; The model parameters of the speech recognition model are adjusted based on the loss value.

10. The training method according to claim 9, characterized in that, The speech recognition model includes a masking network, which extracts features from the sample mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects besides the target object, including: The sample mixed audio stream is input into the preprocessing encoder of the masking network, and the sample mixed audio stream is preprocessed based on the preprocessing encoder to obtain sample mixed audio features; The sample mixed audio features are input into the mask encoder of the mask network. Based on the mask encoder, the sample mixed audio features are processed to predict the mask of the speech emitted by the target object. Based on the mask and the sample mixed audio features, the first audio feature of the target object is obtained; Based on the complement of the mask and the sample mixed audio features, the second audio features of other objects besides the target object are obtained.

11. The training method according to claim 9, characterized in that, The speech recognition model includes a long short-term memory network. The step of obtaining a first feature vector of the speech emitted by the target object based on the first audio feature includes: The first audio feature is input into the long short-term memory network, and feature extraction is performed on the first audio feature based on the long short-term memory network to obtain the first feature vector of the speech emitted by the target object. And / or, the speech recognition model includes a prediction network, wherein predicting a second feature vector of the speech that the target object may emit based on the second audio features includes: The second audio feature is input into the prediction network, and the prediction network predicts the speech that the target object corresponding to the second audio feature may emit, thereby generating the corresponding second feature vector.

12. The training method according to claim 9, characterized in that, The training method further includes: Predict the backward probability of the speech that the target object may emit in the next time step based on the first audio feature; The step of identifying the target text based on the first feature vector and the second feature vector includes: Based on the first feature vector, the second feature vector, the backward probability, and the forward probability of the speech emitted by the target object predicted in the previous time step, the target text is predicted by a preset conditional probability.

13. The training method according to any one of claims 9 to 12, characterized in that, The speech recognition model further includes an audio encoder and a label encoder, and the training method further includes: The first audio feature is input into the audio encoder, the first audio feature is encoded based on the audio encoder, and the sample label is encoded according to the label encoder; Predict the backward probability of the speech that the target object may emit in the next time step based on the first audio feature; Based on the backward probability and the forward probability of the speech emitted by the target object predicted in the previous time step, the gradient of the loss function with respect to the model parameters is predicted. The step of adjusting the model parameters of the speech recognition model based on the loss value includes: When the loss value is greater than or equal to a preset threshold, the model parameters are adjusted according to the gradient.

14. The training method according to claim 13, characterized in that, The training method also includes: The total loss function is constructed based on the error between the target text and the target sample label predicted by the first audio feature and the error between the target text and the target sample label predicted by the second audio feature.

15. A voice recognition device, characterized in that, include: The extraction module is used to extract features from the mixed audio stream to obtain the first audio features of the target object and the second audio features of other objects besides the target object; The first prediction module is used to obtain a first feature vector of the speech emitted by the target object based on the first audio feature; The second prediction module is used to predict a second feature vector associated with the speech that the target object may emit, based on the second audio feature. The recognition processing module is used to identify the target text corresponding to the speech emitted by the target object in the mixed audio stream based on the first feature vector and the second feature vector.

16. A training device for a speech recognition model, characterized in that, include: The acquisition module acquires training samples, which include a sample mixed audio stream and sample labels corresponding to the sample mixed audio stream; A speech recognition model is used to perform the following processing on the input sample mixed audio stream to identify the target text corresponding to the speech emitted by the target object in the sample mixed audio stream; Feature extraction is performed on the sample mixed audio stream to obtain a first audio feature of the target object and a second audio feature of other objects besides the target object; based on the first audio feature, a first feature vector of the speech emitted by the target object is obtained; based on the second audio feature, a second feature vector associated with the speech that may be emitted by the target object is predicted; based on the first feature vector and the second feature vector, the target text is identified. The training module is used to calculate the loss value of the target text relative to the corresponding sample label based on the total loss function, and adjust the model parameters of the speech recognition model according to the loss value.

17. An electronic device, characterized in that, The electronic device includes: a memory and a processor; The memory is used to store computer programs / instructions; the processor is used to implement the method as described in any one of claims 1 to 14 according to the computer programs / instructions stored in the memory.

18. A vehicle, characterized in that, include: Memory, processor; The memory is used to store computer programs / instructions; The processor is configured to implement the method as described in any one of claims 1 to 8 according to the computer program / instructions stored in the memory.

19. A computer-readable storage medium / computer program product, characterized in that, The computer-readable storage medium stores a computer program / instructions, or the computer program product includes a computer program / instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1 to 14.

Images