Speech interaction model systems and methods

Abstract

The present disclosure relates to speech interaction models, and in particular to methods and systems for training a speech interaction model. The methods and systems comprise receiving a training sample set including a plurality of question texts, corresponding sample response texts for each question text, and question audios associated with the question texts, and inputting the training sample set into the speech interaction model to train the model. The speech interaction model is configured to generate question text features corresponding to each question text, generate predicted response texts, and generate question audio features corresponding to the question audios. The method further comprises determining a first loss value of the trained speech interaction model based on the question text features and the question audio features corresponding to each question text, determining a second loss value of the trained speech interaction model based on the predicted response texts and the sample response texts corresponding to each question text, determining if the trained speech interaction model has converged based on the first and second loss values, and establishing the trained speech interaction model if the trained speech interaction model is determined to be convergent based on the first and second loss values.

Description

[0001] SPEECH INTERACTION MODEL SYSTEMSAND METHODS

[0002] Technical Field

[0003] The present disclosure relates generally to speech interaction models, and in particular to systems and methods fortraining a speech interaction model.

[0004] Background

[0005] With the development of artificial intelligence technology, speech interaction models are becoming widely used in speech interaction products. For example, speech interaction models may be used to recognize and process user inputs via voice / speech, and may output corresponding responses, achieving voice interaction between users and products.

[0006] Currently, when training speech interaction models, reliance is primarily placed on samples from the single modality of question texts, combined with corresponding response texts, to generate the speech interaction model. Speech interaction models trained in this manner cannot directly recognize and process user inputs in voice form; instead, they require the user's voice input to be first converted into text. Only then, with the converted text as input, can the speech interaction model recognize and process the input text to output a corresponding response.

[0007] Currently, speech interaction models primarily rely on samples from the single modality of question texts, combined with corresponding response texts, fortraining. The training process lacks samples from other modalities such as audio. As a result, when applying the speech interaction model for voice interaction, user-inputted voice cannot be directly used as input for the model. Instead, the user-inputted voice must first be converted into text, and then the converted text is used as input for the speech interaction model, which then recognizes and processes the input text to output a corresponding response. However, during the process of converting voice to text, inaccuracies in the conversion may lead to the text obtained from the voice not truly reflecting the actual semantics expressed by the voice. This, in turn, causes the speech interaction model to be unable to accurately output responses tailored to the voice, thereby affecting the voice interaction effect between voice interaction products and users and providing a poor experience for users.

[0008] Therefore, the text obtained from voice conversion using current speech interaction models may not accurately reflect the actual semantics expressed in the voice, leading to inaccurate responses from the speech interaction model that are not well-suited to the voice input.Therefore, how to train a speech interaction model that can accurately output responses tailored to user voice inputs has become a pressing problem to solve.

[0009] Therefore, in order to address or alleviate at least one of the aforementioned problems and / or disadvantages, there is a need to provide improved speech interaction models, systems, and methods, particularly with an improved speech recognition function.

[0010] Summary

[0011] In accordance with a first aspect of the present disclosure, a method for training a speech interaction model is provided. The method comprises receiving a training sample set including a plurality of question texts, corresponding sample response texts for each question text, and question audios associated with the question texts, and inputting the training sample set into the speech interaction model to train the model. The speech interaction model is configured to generate question text features corresponding to each question text, generate predicted response texts, and generate question audio features corresponding to the question audios. The method further comprises determining a first loss value of the trained speech interaction model based on the question text features and the question audio features corresponding to each question text, determining a second loss value of the trained speech interaction model based on the predicted response texts and the sample response texts corresponding to each question text, determining if the trained speech interaction model has converged based on the first and second loss values, and establishing the trained speech interaction model if the trained speech interaction model is determined to be convergent based on the first and second loss values.

[0012] In an embodiment, the method further comprises receiving both a user-inputted audio and corresponding text, or user inputted converted text and corresponding audio, as inputs during a voice interaction to interact with the trained speech interaction model.

[0013] In an embodiment, the speech interaction model is configured to output a response based on both the user-inputted audio and the corresponding text, or user inputted converted text and corresponding audio thereby improving the accuracy of the recognition of user semantics. In an embodiment, the question audios are derived through audio conversion of the question texts.

[0014] In an embodiment, the audio conversion process for user question audio comprises an audio synthesis model.In an embodiment, the method further comprises inputting a combined question text and corresponding preset emotional and / or vocal timbre features of the question audio into an audio synthesis model, wherein the audio synthesis model is configured to convert text into speech with emotions and vocal timbres corresponding to the question audio.

[0015] In an embodiment, the step of generating question audio features comprises generation of initial audio features comprising numerical representations of speech, optionally wherein the numerical representations of speech comprise acoustic information of pitch, timbre, and / or speaking rate, of the question audio.

[0016] In an embodiment, the question audio features comprise one or more of spectral features (e.g., Mel-Frequency Cepstral Coefficients (MFCCs)), fundamental frequency features (representing the pitch of the voice), energy features (representing the loudness of the voice), and / or formant features (representing the resonant characteristics of the voice).

[0017] In an embodiment, the first loss value provides an indication of an alignment between the question text features and the question audio features.

[0018] In an embodiment, the first threshold value is a boundary value indicating the speech interaction model converges, optionally wherein the first threshold value corresponds to a loss value of a previous speech interaction model training.

[0019] In an embodiment, receiving the sample training set comprises extracting target question texts and corresponding target response texts from a question and answer data set, optionally wherein the extracted target question texts and corresponding target response texts relate to a scenario to which the model will be applied.

[0020] In an embodiment, the step of determining if the trained speech interaction model has converged comprises:

[0021] - processing a first loss value and a second loss value through addition to obtain a total loss value;

[0022] - determining whether the total loss value is not greater than a first threshold value; and - if the total loss value is not greater than the first threshold value, determining that the trained speech interaction model has converged, or if the total loss value is greater than the first threshold value, determining that the trained speech interaction model has not converged. In an embodiment, processing the first loss value and the second loss value through addition further comprises performing a weighted calculation to obtain the total loss value, wherein theweights represent the degree of influence of the first and second loss values on the convergence of the trained speech interaction model.

[0023] In an embodiment, the weights for the first and second loss values are determined based on historical data from prior training of the trained speech interaction model.

[0024] In an embodiment, the first threshold value is based on historical training data of the trained speech interaction model and is determined by observing the model’s performance under various values of the first loss value.

[0025] In an embodiment, the first loss value is calculated based on question text features and question audio features, and the second loss value is calculated based on predicted reply text and sample reply text.

[0026] In an embodiment, the first threshold value is determined by analyzing model performance under different first loss values, such that the first threshold corresponds to the model's performance expectations for convergence.

[0027] In an embodiment, the trained speech interaction model is determined to have converged when the total loss value is not greater than the first threshold value, indicating the model has met a performance criteria for accurately responding to user speech.

[0028] In an embodiment, the step of determining if the trained speech interaction model has converged comprises:

[0029] - processing a first loss value and a second loss value through addition to obtain a total loss value;

[0030] - determining whether a difference between the total loss value and a prior or historic total loss value from a previous training of the model is within a first threshold range; and

[0031] - if the difference is not within the first threshold range, determining that the trained speech interaction model has not converged, or

[0032] - if the difference is within the first threshold range, the model is determined to converge if a plurality of differences between the total loss value and prior total loss values shows a downward trend, and a decrease in the difference is within the first threshold range.

[0033] In an embodiment, the first threshold range is preset based on historic or prior total loss values of the speech interaction model when convergence was achieved during prior training.In an embodiment, if the total loss values or differences from the previous first number of training iterations do not show a downward trend or show a downward trend but the decrease in total loss value between adjacent later training iterations is not within the first threshold range, the trained speech interaction model is determined not to have converged.

[0034] In an embodiment, if the total loss values from the previous first number of training iterations show a downward trend, and the decrease in the total loss value of each adjacent later training iteration compared to the previous iteration is within the first threshold range, the trained speech interaction model is determined to have converged.

[0035] In an embodiment, the first threshold range is defined such that the difference between the total loss value from the current training and the total loss value from the previous training is considered within the first threshold range if it falls within a predefined acceptable range of convergence for the trained speech interaction model.

[0036] In an embodiment, the method further comprises comparing the difference between the total loss values from the current training and a previous training and determining if the difference lies within a certain threshold range.

[0037] In an embodiment, the comparison of the total loss values involves checking whether the total loss values from the adjacent N previous training iterations show a downward trend and whether the decrease in the total loss value between adjacent training iterations is within the first threshold range.

[0038] In an embodiment, the method further comprises obtaining second threshold values corresponding to the first loss value and the second loss value, wherein obtaining second threshold values comprises determining a second threshold value for each loss value, with a second question threshold value for the first loss value reflecting a convergence boundary under the factor of question text features and question audio features, and a second response threshold value for the second loss value reflecting a convergence boundary under the factor of predicted response text and sample response text.

[0039] In an embodiment, the method further comprises:

[0040] (a) determining whether the first loss value is not greater than the second question threshold value corresponding to the first loss value; and

[0041] (b) determining whether the second loss value is not greater than the second response threshold value corresponding to the second loss value;wherein if both conditions (a) and (b) are satisfied, the trained speech interaction model is determined to have converged.

[0042] In an embodiment, the second threshold values are preset based on historical training data from the speech interaction model or based on empirical data from prior model training. In an embodiment, if either the first loss value is greater than the second question threshold value corresponding to the first loss value, or the second loss value is greater than the second response threshold value corresponding to the second loss value, the trained speech interaction model is determined not to have converged.

[0043] In an embodiment, the second threshold values are adjusted based on a scenario or specific requirements of the speech interaction model and the desired performance criteria for accurately recognizing question text features, question audio features, predicted response text, and sample response text.

[0044] In an embodiment, the method further comprises clarifying how the second threshold values for the first and second loss values are set, such that these thresholds reflect different performance requirements for model convergence under various factors such as question text features, question audio features, and predicted response text.

[0045] In an embodiment, the first loss value and the second loss value are calculated using weighted factors corresponding to the importance of the question text features, question audio features, predicted response text, and sample response text in evaluating the performance of the speech interaction model.

[0046] In an embodiment, the method further comprises determining whether the differences between the first loss value and the second loss value and a respective first loss value and second loss value obtained from a previous training of the trained speech interaction model fall within corresponding second threshold ranges.

[0047] In an embodiment, if the differences between the first loss value and the second loss value and the respective first loss value and second loss value obtained from the previous training of the trained speech interaction model do not fall within the corresponding second threshold ranges, the trained speech interaction model is determined not to have converged.

[0048] In an embodiment, the method further comprises judging whether the first loss values and second loss values obtained from the preceding second number of trainings of the trained speech interaction model both show a decreasing trend, and whether the decrease in the firstloss value and the decrease in the second loss value in any two adjacent trainings fall within their corresponding second threshold ranges.

[0049] In an embodiment, if at least one of the first loss values and second loss values obtained from the preceding second number of trainings does not show a decreasing trend, or the decrease in the first loss value or second loss value in any two adjacent trainings does not fall within the corresponding second threshold ranges, the trained speech interaction model is determined not to have converged.

[0050] In an embodiment, if the first loss values and second loss values obtained from the preceding second number of trainings both show a decreasing trend, and the decrease in the first loss value and the decrease in the second loss value in any two adjacent trainings fall within the corresponding second threshold ranges, the trained speech interaction model is determined to have converged.

[0051] In an embodiment, the second threshold ranges corresponding to the first loss value and the second loss value are preset based on historical loss values from prior model training when convergence was achieved.

[0052] In an embodiment, the second number of trainings used to judge whether the first loss values and second loss values show a decreasing trend is set in advance based on empirical data from historical model training.

[0053] In an embodiment, the method further comprises comparing whether the difference between the current first and second loss values and the respective loss values obtained from the previous training fall within a predefined threshold range, and determining whether the first and second loss values from the adjacent M previous trainings all show a downward trend. In accordance with a second aspect of the present disclosure, a speech interaction model is provided. The speech interaction model is trained according to the method of any of the embodiments or aspects as described herein.

[0054] In accordance with a third aspect of the present disclosure, a system for training a speech interaction model is provided. The system comprises a training module configured to obtain a training sample set including a plurality of question texts, corresponding sample response texts for each question text, and question audios derived from the question texts. The system further comprises a processing module configured to input the training sample set into the speech interaction model to train the model, and generate question text features corresponding to each question text, predicted response texts, and question audio features corresponding to the question audios. The system further comprises a loss calculation moduleconfigured to determine a first loss value based on the question text features and question audio features, and a second loss value based on the predicted response texts and the sample response texts. The system further comprises a convergence determination module configured to determine if the trained speech interaction model has converged based on the first and second loss values. The system further comprises an output module configured to establish the trained speech interaction model if the model is determined to be convergent. In an embodiment, the system further comprises an interaction module configured to receive both a user-inputted voice and corresponding text as inputs during a voice interaction with the trained speech interaction model.

[0055] In accordance with a fourth aspect of the present disclosure an apparatus fortraining a speech interaction model is provided. The apparatus comprises a processor configured to execute instructions to obtain a training sample set, input the training sample set into a speech interaction model for training, calculate loss values based on the input data and predicted results, and establish the trained speech interaction model if the model converges based on the calculated loss values. The apparatus further comprises a memory configured to store the training sample set, including question texts, corresponding sample response texts, and question audios derived from the question texts, and to store the trained speech interaction model.

[0056] In an embodiment the apparatus further comprises a voice input component and / or a text input component configured to receive user voice and / or text inputs, respectively, and provide them as inputs to the trained speech interaction model during a voice interaction.

[0057] In accordance with a fifth aspect of the present disclosure a system for training a speech interaction model is provided. The system comprises an audio module configured to receive question audio, an input text module, configured to receive question text and sample response text, and a large language model. The large language model comprises an audio features module configured to process the question audio and generate one or more corresponding question audio features; a text processing module configured to process the question text and sample response text and generate one or more question text features, and one or more sample response text features, a multimodal module configured to receive the question audio features, question text features, and sample response text features, and generate a concatenated multimodal feature set, and a prediction module configured to receive the multimodal feature set and generate a predicted response text. The system and / or the large language model further comprises a first loss value module configured to receive the question audio features and question text features and generate a first loss value, a second loss module configured to receive the sample response text and predicted response text features andgenerate a second loss value, and an evaluation module configured to evaluate the first loss value and second loss value to determine if the model is trained.

[0058] In an embodiment, the audio features module comprises an audio encoder configured to receive question audio and generate initial audio features.

[0059] In an embodiment, the text processing module comprises a speech to text module configured to convert a user’s audio speech to text, and / or generate one or more sample response texts to the user’s audio speech..

[0060] In a sixth aspect, a computer-readable medium is provided. The computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform the method of training the speech interaction model as described in any of the embodiments or aspects outlined herein.

[0061] In a seventh aspect, there is provided a data structure. The data structure comprises a set of question texts, corresponding response texts, and corresponding audio files, configured for use in training a speech interaction model as described in any of the embodiments or aspects outlined herein.

[0062] In an eighth aspect, the use of a trained speech interaction model of any of the aspects described herein for recognizing and responding to user inputs in a voice interaction system is provided. The use may include utilising one or more inputs, comprising both voice audio and corresponding text, or text and corresponding voice audio, as described in any of the embodiments or aspects outlined herein.

[0063] Various systems and methods according to the present disclosure are thus disclosed herein. Various features, aspects, and advantages of the present disclosure will become more apparent from the following detailed description of the embodiments of the present disclosure, by way of non-limiting examples only, along with the accompanying drawings.

[0064] Brief Description of the Drawings

[0065] Fig. 1 is a flowchart illustrating a method of training a speech interaction model according to embodiments of the present disclosure.

[0066] Fig. 2 is a block diagram illustrating a system architecture of a system for training a speech interaction model according to embodiments of the present disclosure.Fig. 3 is a block diagram illustrating a system architecture of a large language model, such as a speech interaction model, according to embodiments of the present disclosure.

[0067] Fig. 4 is a flowchart illustrating a first method of determining model convergence according to embodiments of the present disclosure.

[0068] Fig. 5 is a flowchart illustrating an alternative method of determining model convergence according to embodiments of the present disclosure.

[0069] Fig. 6 is a flowchart illustrating an alternative method of determining model convergence according to embodiments of the present disclosure.

[0070] Fig. 7 is a flowchart illustrating an alternative method of determining model convergence according to embodiments of the present disclosure.

[0071] Fig. 8 is a block diagram illustrating a system architecture of a speech interaction model according to embodiments of the present disclosure.

[0072] Fig. 9 is a block diagram illustrating a technical architecture of a computing server according to embodiments of the present disclosure.

[0073] Detailed

[0074] For purposes of brevity and clarity, descriptions of embodiments of the present disclosure are directed to a in accordance with the drawings. While aspects of the present disclosure will be described in conjunction with the embodiments provided herein, it will be understood that they are not intended to limit the present disclosure to these embodiments. On the contrary, the present disclosure is intended to cover alternatives, modifications and equivalents to the embodiments described herein, which are included within the scope of the present disclosure as defined by the appended claims. Furthermore, in the following detailed description, specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be recognized by an individual having ordinary skill in the art, i.e. a skilled person, that the present disclosure may be practiced without specific details, and / or with multiple details arising from combinations of aspects of particular embodiments. In a number of instances, well-known systems, methods, procedures, and components have not been described in detail so as to not unnecessarily obscure aspects of the embodiments of the present disclosure.

[0075] In embodiments of the present disclosure, depiction of a given element or consideration or use of a particular element number in a particular figure or a reference thereto incorresponding descriptive material can encompass the same, an equivalent, or an analogous element or element number identified in another figure or descriptive material associated therewith.

[0076] References to “an embodiment / example”, “another embodiment / example”, “some embodiments / examples”, “some other embodiments / examples”, and so on, indicate that the embodiment(s) / example(s) so described may include a particular feature, structure, characteristic, property, element, or limitation, but that not every embodiment / example necessarily includes that particular feature, structure, characteristic, property, element or limitation. Furthermore, repeated use of the phrase “in an embodiment / example” or “in another embodiment / example” does not necessarily refer to the same embodiment / example. The terms “comprising”, “including”, “having”, and the like do not exclude the presence of other features / elements / steps than those listed in an embodiment. Recitation of certain features I elements / steps in mutually different embodiments does not indicate that a combination of these features / elements / steps cannot be used in an embodiment.

[0077] As used herein, the terms “a” and “an” are defined as one or more than one. The use of “ / ” in a figure or associated text is understood to mean “and / or” unless otherwise indicated. The term “set” is defined as a non-empty finite organisation of elements that mathematically exhibits a cardinality of at least one (e.g. a set as defined herein can correspond to a unit, singlet, or single-element set, or a multiple-element set), in accordance with known mathematical definitions. The terms “first”, “second”, etc. are used merely as labels or identifiers and are not intended to impose numerical requirements on their associated terms. As used in the present disclosure, the terms “component”, “module”, “system”, “interface” and the like are generally intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a module may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more modules may reside within a process and / or thread of execution and a module may be localized on one computer and / or distributed between two or more computers. As another example, an interface can include I / O components as well as associated processor, application, and / or API components. In the context of the present disclosure, the information processing and response generation system and its constituent parts may be implemented as hardware, software, or a combination thereof.

[0078] Furthermore, various embodiments of the present disclosure may be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineeringtechniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. For instance, the claimed subject matter may be implemented as a computer-readable medium embedded with a computer executable program, which encompasses a computer program accessible from any computer-readable storage device or storage media. For example, computer readable media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips, etc ), optical disks (e.g., compact disk (CD), digital versatile disk (DVD), etc.), smart cards, and flash memory devices (e.g., card, stick, key drive, etc ).

[0079] In an example embodiment, during a voice interaction, a user-inputted voice, or voice audio, and corresponding text (for example, provided via speech-to-text recognition) may be simultaneously used as inputs to a speech interaction model. For example, if, during the training of a speech interaction model (such as those disclosed herein), samples from both the modalities of question text and the audio corresponding to the question text are relied upon, then a speech interaction model may be generated through training with response texts corresponding to both the question text and the audio. If this speech interaction model is then used during voice interaction, both a user-inputted voice and a corresponding text can be simultaneously used as inputs to the model. In this example embodiment, since both voice and text are used as inputs to the model, the speech interaction model may accurately recognize one or more actual semantics expressed by the user, by combining the voice (or speech audio) and text, thereby accurately outputting responses tailored to the voice or speech audio of the user.

[0080] Accordingly, example embodiments disclosed herein provide a method, system and / or apparatus for training a speech interaction model. In an example embodiment, such as that illustrated in Fig. 1 , when training a speech interaction model is required, a training sample set 108 may be obtained. The training sample set 108 may include multiple question texts, corresponding sample response texts for each question text, and question audios obtained through audio conversion of the question texts. The training sample set 108 may input into the speech interaction model to be trained for model training, resulting in question text features 122 corresponding to each question text, predicted response texts 126, and question audio features 124 corresponding to the question audios.

[0081] Subsequently, based on the question text features 122 and question audio features 124 corresponding to each question text, a first loss value of the trained speech interaction model may be determined at step 130. Additionally, based on the predicted response texts 126 and sample response texts corresponding to each question text from training data 108, a second loss value of the trained speech interaction model may be determined at step 140.Finally, if at step 150 the trained speech interaction model is determined to be convergent at step 152 based on the first and second loss values, it is established as the trained speech interaction model. Alternatively, if at step 150 the speech interaction model is determined to not be convergent at step 154, systems and methods as disclosed herein may continue training of the speech interaction model.

[0082] Example embodiments may incorporate several key implementations:

[0083] Firstly, during the training of a speech interaction model, samples from two different modalities, namely text and audio, may be utilized. This may allow a trained speech interaction model to respond to user voice inputs by simultaneously leveraging both the corresponding audio and text of the voice, which may enable a more comprehensive and accurate recognition of a user's actual semantics.

[0084] Secondly, during a training process of a speech interaction model, loss calculations may be performed between the trained question text features and question audio features. This may enable the speech interaction model to enhance its semantic understanding of question audios based on question text features while retaining its understanding of unique features of the question audios.

[0085] Thirdly, during the training of a speech interaction model, loss calculations may be conducted between the predicted response texts and sample response texts obtained through training. This may continuously improve the model's fitting degree on training data, which may allow the speech interaction model to predict response texts more accurately.

[0086] Based on these three key implementations, a speech interaction model trained according to embodiments disclosed herein may simultaneously take user voice inputs and corresponding texts as model inputs. By leveraging both the corresponding audio and text of the voice, the model may comprehensively and / or accurately recognize a user's actual semantics, and thereby output responses tailored to the voice based on recognized semantics. This, in turn, may provide for enhanced voice interaction between voice interaction products and users, providing an improved voice interaction experience for users.

[0087] Systems and methods for training a speech interaction model according to example embodiments disclosed herein may include at least the following steps:

[0088] At a first step, such as step 110 in Fig. 1 , systems and methods disclosed herein may acquire a training sample set 110, which may include data 108 comprising multiple question texts, corresponding question audios for each question text, and sample response texts. The question audios may be obtained through audio conversion of the question texts as described herein.When training a speech interaction model, such as those disclosed herein, it may be necessary to acquire a training sample set 110 to train the speech interaction model. To enable the speech interaction model to accurately respond to user-inputted voice, samples from both text and audio modalities may be used during the training of the speech interaction model. This may allow the trained speech interaction model to simultaneously utilize both the audio and text corresponding to the user-inputted voice when responding, achieving a more comprehensive and accurate recognition of the user's actual semantics and outputting responses tailored to the user's voice or audio. Therefore, the training sample set 110 may include data 108 comprising at least one or more question texts, corresponding question audios for each question text, and sample response texts. The question audios may be obtained through audio conversion of the question texts, expressing the questions in an audio modality as described herein.

[0089] The process of acquiring, receiving, obtaining, generating and / or determining 110 the training sample set for training a speech interaction model as disclosed herein may include the following steps:

[0090] At a first step, methods and systems as described herein may extract target question texts and corresponding target response texts for constructing the training sample set from question-and-answer data 108 corresponding to a scenario where the speech interaction model will be applied.

[0091] In example embodiments, question text features may include word embedding features (e.g., word vectors generated by Word2Vec or BERT, etc.), syntactic features (e.g., part-of-speech tagging, dependency relations, etc ), and semantic features (e.g., topic vectors generated by topic models, etc.). These features may capture the linguistic properties of text, aiding the model in understanding the meaning and context of questions. Question text features 122 may be extracted or generated from training sample set 110 as described herein.

[0092] It will be understood that step 110 may comprise generating one or more data of the training sample set as described herein. For example, the speech interaction model as described herein may generate the question text features 122 from data 108, and / or generate question audio features 124 from data 108. Similarly, speech interaction models as described herein may generate one or more predicted response texts 126 from the data 108, and / or from the generated question text features 122 and question audio features 124. The predicted response texts 126 may comprise response texts predicted by the speech interaction model, based on the input data 108 or generated data at 110, that may comprise a suitable response to one or more questions posed as part of question and answer data 108.In example embodiments, the speech interaction model may be applied to specific use case scenarios. Therefore, it may be necessary to acquire a training sample set 110 based on question-and-answer data 108 corresponding to a scenario where the speech interaction model will be applied, in order to train a speech interaction model that can accurately respond to user-inputted voice in that scenario. The scenarios may include, but are not limited to, customer service, companionship, healing, and other specified scenarios. Users may include, but are not limited to, devices, apparatuses, virtual entities, and real individuals that output voice data.

[0093] For example, real individuals may include human users of systems and methods as described herein who interact with smart devices or applications through voice, such as controlling smart homes or querying voice assistants. Devices may include users that are automated devices, such as smart robots, that interact with other devices or humans through voice to complete specific tasks. Apparatus may include users that are devices with specific functions, such as in-car voice systems, that receive user commands through voice and provide services such as navigation and entertainment. Virtual entities may include users that are virtual characters, such as non-player characters (NPCs) in games or virtual reality, that interact with players through voice to enhance a gaming experience. It will be understood the abovementioned list of uses is non-exhaustive, and the term user may refer to any reasonable user of a speech interaction model, both in training of the model and interaction with the model, via an audio or speech or voice interaction.

[0094] After acquiring question-and-answer data 108 corresponding to a scenario as set out herein, the method 100 may comprise the step of extracting one or more target question texts, or question text features 122 and corresponding target response texts, or predicted response texts 126, for constructing a training sample set. This extraction or generation process may include: inputting the question-and-answer data 108 into a preset extraction model, which may extract target question texts 122 and corresponding target response texts from the data 108. The extraction model may be trained using multiple sets of data 108, each set comprising question-and-answer data, question texts included in the question data, and response texts corresponding to the question texts.

[0095] In a second step, methods and systems as described herein may set or determine corresponding emotional features and voice features for each target question text.

[0096] The speech interaction model as described herein may be trained using samples from both text and audio modalities. Therefore, after extracting the target question texts 122 in the first step, systems and methods as described herein may convert the target question texts into corresponding target question audios 124 through audio conversion. This may allow methodsand systems as described herein to effectively utilize both the text and audio modalities of the same question to train the speech interaction model. It will be understood that in some embodiments the target question audios or question audio features 124 may be generated directly from training data 108, and / or may be generated from the previously generated target question text or question text features 122.

[0097] Considering that users may input voice with their own emotional features and voice features, which are unique to audio and affect the speech interaction model's understanding of audio semantics, it may be necessary to set corresponding emotional features and voice features for each target question text before converting the target question text into corresponding target question audio. This may enable the speech interaction model to accurately recognize and understand the semantics of audio features in the audio modality.

[0098] In an example embodiment, to set the emotional features and voice features, the scenario where the speech interaction model will be applied may first be determined, and the potential emotional features and voice features of each user in that scenario may be assessed and / or considered. These features may then be clustered based on user categories (which can be represented by factors including but not limited to gender, age, education, place of origin, etc.), resulting in emotional features and voice features corresponding to each user category. Subsequently, for each target question text 122, methods and systems as described herein may determine a user category with a likelihood of inputting the corresponding question that exceeds a likelihood threshold. If only one user category is determined, its corresponding emotional features and voice features may be set as those of the current target question text. If multiple user categories are determined, copies of the current target question text 122 may be created, with the number of copies equal to the number of determined user categories. Each copy may then be assigned the emotional features and voice features corresponding to its respective user category. As a result, after acquiring the training sample set, the current question texts 122 in the set may be duplicated, with each duplicated text corresponding to question audios 124 converted from the same text but based on different emotional features and / or voice features.

[0099] During a third step, methods and systems as described herein may convert the target question texts 122 into target question audios 124 based on their corresponding emotional features and voice features as determined in the second step outlined above. This third step may comprise, for each target question, performing the following steps:

[0100] If, in the second step, emotional features and voice features were set for the current target question based on only one user category, then input the current target question 122, along with its corresponding emotional features and voice features, into an audio synthesis model.The model may then synthesize the target question audio 124 for the current target question text based on the input.

[0101] If, in the second step, emotional features and voice features were set for the current target question text based on multiple user categories, then for each user category, use the audio synthesis model to convert the current target question text 122 into a corresponding target question audio 124 based on the emotional features and voice features set for that category. It should be noted that the audio synthesis model may be used to convert question texts 122 into question audios 124 with corresponding emotional features and voice features, based on the question texts 122 and their associated features. The model as described herein may be trained using multiple sets of data, each set comprising a question audio 124, its corresponding question text 122, emotional features, and voice features.

[0102] At a fourth step, methods and systems as described herein may obtain or determine a training sample set 110 by combining the target question texts 122 with their corresponding target response texts and target question audios 124.

[0103] After converting the target question texts 122 into corresponding target question audios 124 through audio conversion as set out in the third step above, the training sample set 110 may be obtained by combining the target question texts 122 with their respective target response texts and target question audios 124.

[0104] It should be noted that if there is at least one target question text 122 for which emotional features and voice features were set based on multiple user categories as identified in the second step above, the obtained training sample set 110 will have the following characteristic: the set will include at least some duplicated question texts 122, with each duplicated text corresponding to question audios 124 converted from the same text but based on different target feature audios. Here, target features include at least one of the following: emotional features and voice features.

[0105] Furthermore, to enhance the versatility of the speech interaction model as described herein, the training sample set 110 may include question texts 122 in at least two languages, with the languages of the question audios 124 and sample response texts matching the languages of their corresponding question texts 122. The types of languages may include, but are not limited to, national languages and regional dialects.

[0106] After acquiring the training sample 110 set through the above first to fourth steps, the prerequisite fortraining the speech interaction model using samples from both text and audio modalities may be met. Based on this, systems and methods as disclosed herein may proceed to train the speech interaction model.The process of training a speech interaction model as disclosed herein may include the following steps:

[0107] In an example embodiment, the training sample set 110 determined above may be input into a speech interaction model to be trained for model training. Turning to Fig. 1, this may include obtaining, generating and / or determining question text features corresponding to each question text at 122, predicted response texts at 126, and question audio features corresponding to the question audios at 124.

[0108] Speech interaction models typically undergo multiple iterations of training to accurately output responses tailored to user speech. Based on this, the speech interaction model to be trained according to example embodiments may include the following two scenarios:

[0109] One is when the training sample set 110 is used for a first model training, the speech interaction model to be trained may be one that has not been trained on the training sample set.

[0110] The other is when the training sample set is used for non-initial model training, the speech interaction model to be trained may be one that has not converged after a previous training. The model training process may be related to the specific structure of the speech interaction model. For example, a model training process is explained below in conjunction with the structure of the speech interaction model.

[0111] As set out in Fig. 2, in an example embodiment a system 200, for example a system for training a speech interaction model, may comprise a large language module, and may include an audio encoder 208 and an audio feature module 224 to generate one or more question audio features 224a as described herein, a text processing module to generate one or more question text features 222a and / or sample response text features 222b as described herein, a multimodal module 226 to concatenate one or more of the features, and / or a prediction module 228 to generate one or more predicted response text features 228a. The system 200 may further comprise first and second loss modules to generate first and second loss parameters or values as described herein, and an evaluation module to analyse the first and second loss values and determine if a model is trained.

[0112] In example embodiments, the process of inputting the training sample set into the speech interaction model 200 to be trained for model training, to obtain question text features corresponding to each question text, predicted response texts, and question audio features corresponding to the question audios, may include the following steps:T urning now to Figs. 2 and 3, at a first step, systems 200, 300 as disclosed herein may process an input question audio through an audio encoder 208, 308 to obtain initial audio features, and process an input question text and sample response text through a text processing module 222 or embedding layer 322 to obtain question text features and sample response text features.

[0113] During the training of a speech interaction model 200, 300 according to an example embodiment, each question audio corresponding to the question text may be first input into an audio encoder 208, 308, which processes the input question audio to obtain initial audio features. However, it will be understood that this is an optional step, and question audio may instead proceed directly to audio features module 224 to generate question audio features. The audio encoder 208, 308 may be used to encode and identify initial audio features of the question audio.

[0114] The audio conversion process for user question audio may primarily rely on an audio synthesis model encoder 208, 308. In example embodiments, this process may involve combining a question text with preset emotional and / or vocal timbre features, and then inputting this combined data into an audio synthesis model or encoder 208, 308. Based on deep learning techniques, the audio synthesis model may be capable of converting text into speech with specific emotions and vocal timbres.

[0115] The output of an audio encoder may consist of initial audio features. These features are not necessarily directly audible speech, but rather may comprise numerical representations of speech, i.e., audio features. The audio features may encompass acoustic information such as pitch, timbre, and / or speaking rate, etc., but have not yet been converted into speech waveforms that are directly understandable by the human ear.

[0116] In example embodiments, audio features may typically include one or more of spectral features (e.g., Mel-Frequency Cepstral Coefficients (MFCCs)), fundamental frequency features (representing the pitch of the voice), energy features (representing the loudness of the voice), and / or formant features (representing the resonant characteristics of the voice). These features may collectively constitute the acoustic characteristics of speech and serve as a foundation for speech recognition and understanding.

[0117] It will be understood that the audio encoder 208, 308 can be flexibly selected based on scenario needs, and the systems and methods disclosed herein may utilise a variety of audio encoders 208, 308 depending on the scenario to which the speech interaction model is associated. It should be noted that the selected audio encoder may minimize the loss of the original features of the question audio to avoid the initial audio features failing to reflect thetrue semantics of the question audio. Exemplarily, the audio encoder can be, but is not limited to, the whisper-large-v3 encoder.

[0118] In an example embodiment, audio feature module 224 may generate one or more question audio features 224a from the question audio as outlined herein. The question audio may result from initial training data, such as data 108 as described in relation to Fig. 1, or may be provided by audio encoder 208 as described above. It will be understood that audio encoder 208 and audio feature module 224 may comprise the same module, or may be separate modules. After inputting each question audio corresponding to the question text into the audio encoder 208, 308, the question text and the corresponding sample response text for each question text may be input into an embedding layer 322, or text processing module 222. The input question text and sample response text may be processed through the embedding layer 322 or text processing layer 222 to obtain the question text features 222a, 322a and sample response text features 222b, 322b corresponding to each question text. It will be understood that the order of processing of question audio features 224a, question text features 222a, and sample response text features 222b may be sequential or parallel in any order.

[0119] During a second step, the systems and methods as described herein may semantically align the initial audio features 224a with the question text features 222a of the corresponding question text through the audio feature module 224, and / or a self-attention layer 324 as described in Fig. 3. In addition, the audio feature module 224, or self-attention layer 324 may reduce a dimensionality of the aligned initial audio features to obtain question audio features 224a corresponding to each question text 222a.

[0120] As set out above, after obtaining the initial audio features 224a and the question text features 222a of the corresponding question text, as shown in Fig. 3, the initial audio features and the question text features of the corresponding question text may be input into a self-attention layer 324. The self-attention layer 324 may include the capabilities of semantic alignment and dimensionality reduction: Firstly, the initial audio features may be semantically aligned with the question text features of the corresponding question text through the self-attention layer 324, which may enhance the semantic information of the question audio through the semantics of the question text features, while preserving the unique audio features of the question audio. Then, considering that the amount of audio feature data is large, and a large amount of data is not necessarily conducive for the training efficiency of model training, based on this, considering that the large amount of audio feature data is caused by the high dimensionality of the audio features themselves, it may be necessary to reduce the dimensionality of the aligned initial audio features to decrease the amount of audio feature data.

[0121] The dimensionality reduction methods can include the following two options:One is to determine the dimension of the question text features and reduce the dimension of the aligned initial audio features to be the same or similar to the dimension of the question text.

[0122] The other is to determine a preset or threshold dimension, which is lower than the original dimension of the initial audio features, and reduce the dimension of the aligned initial audio to the threshold dimension.

[0123] The above two dimensionality reduction methods can be flexibly selected based on needs of the scenario to which the model is employed, and it will be understood that embodiments disclosed herein are not limited to one or either option for reduction in audio dimensionality. It will be understood that reducing the dimensions of audio features may be an important part of training a speech interaction model as described herein because raw audio features often contain a large amount of data, requiring significant computational resources for processing. By reducing the dimensionality as described herein, the amount of data can be reduced while retaining key information, thereby improving model training efficiency and runtime speed. Although lower data rate audio features or more powerful processing capabilities may emerge in the future, under current technical conditions, dimensionality reduction may still provide the abovementioned advantages.

[0124] After optionally reducing the dimensionality of the aligned initial audio features, the dimensionality-reduced initial audio features corresponding to each question text may be obtained as the question audio features 224a corresponding to each question text 222a. In a third step, systems and methods as disclosed herein may concatenate the question audio features 224a and question text features 222a corresponding to each question text to obtain multimodal features corresponding to each question text. This may be accomplished via a multimodal module 226, 326 as described herein.

[0125] After obtaining the question audio features 224a, 324a and question text features 222a, 322a, as shown in Figs. 2 and 3, it may be necessary to concatenate the question audio features and question text features corresponding to each question text to obtain multimodal features corresponding to each question text, so as to train the speech interaction model based on samples from the two modalities of text and audio.

[0126] When concatenating the question audio features and question text features, whether to place the question audio features before or the question text features before can be flexibly determined based on needs of the scenario to which the model is employed, and whilst one example embodiment is provided, it will be understood that alternative order is envisaged as described herein.Connecting question audio and text features may combine information from both modalities, i.e. question audio features 224a, 234a and question text features 222a, 322a, which may enhance a speech interaction model's comprehensive understanding of speech and text. For example, audio features may provide acoustic information, while text features may provide semantic information. By connecting these two types of features, speech interaction models as described herein can more accurately understand user intentions and needs.

[0127] In example embodiments, the connection of both modalities as described herein may occur after the generation of multimodal features, i.e., after the question audio features 224a, 324a, and text features 222a, 322a are concatenated into multimodal features. These multimodal features may then be input into a large language model for further processing until a final prediction layer 228, 328 outputs a predicted response text. This is set out below.

[0128] At a fourth step, systems and methods as described herein may train the large language model using the multimodal features and sample response text features 222b corresponding to each question text to obtain predicted response texts corresponding to each question text.

[0129] After obtaining the multimodal features, the large language model may be trained using the multimodal features and sample response text features 222b, 322b corresponding to each question text. In an example embodiment, the process can be, as shown in Fig. 3, the large language model further includes a prediction layer 328. The multimodal features and sample response text features 322b corresponding to each question text are concatenated (the symbol "+" in Fig. 3 represents concatenation), and the concatenated results for each question text may be input into the prediction layer 328 to train the prediction layer using these samples. A similar process may occur in system 200 of Fig. 2, where multimodal module may receive each of question audio features 224a, question text features 222a, and sample response text features 222b, and provide multimodal concatenated features and the sample response text features 222b to prediction module 228. It will be understood that the question audio features 224a may be generated in the audio feature module 224, or the multimodal module 226 as required, and may be based on question text features 22a in either case, or may directly generate audio features 224a from question audio and question text data.

[0130] In example embodiments, the prediction module 228, and / or prediction layer 328, may output or provide or generate one or more predicted response text features 228a. The predicted response text features 228a may comprise one or more predicted responses to questions posed by the user and forming the question text as described herein, and may be output by the speech interaction model as a generated response to a user query. The prediction module 228 may generate the predicted response text features 228a based on the concatenatedquestion text features 22a and question audio features 224a that are connected or linked by the multimodal module 226 as described herein.

[0131] After the first to fourth steps of the process of training a speech interaction model as described herein, data may be obtained, generated and / or used to evaluate the training effect of the speech interaction model after the current training. In an example embodiments, the data to evaluate the model after training may include: question text features 222a, 322a corresponding to each question text, predicted response texts 228a, and question audio features 224a, 324a, corresponding to the question audios. These data may be processed to generate loss values, which may be used to assess whether the speech interaction model, after the current training, has the ability to accurately output responses tailored to user speech. The process of determining first and second loss values according to the embodiments disclosed herein may include the following steps:

[0132] Turning to Fig. 2, based on the question text features 222a and question audio features 224a corresponding to each question text, the first loss value of the trained voice interaction model may be determined via first loss module 230. This may be analogous to method step 130 of Fig. 1, where method 100 may determine a first loss value based on the generated question text features at 122, and the generated question audio features at 124.

[0133] Additionally, or alternatively, based on the predicted response text features 228a and sample response text features 222b corresponding to each question text, the second loss value of the trained voice interaction model may be determined via second loss module 240. This may be analogous to method step 140 of Fig. 1, where method 100 may determine a second loss value based on the generated predicted response texts at 126, and the sample response texts from initial training data 108.

[0134] The ability of the voice interaction model to accurately output responses tailored to user speech may be evaluated based on the following two pieces of data:

[0135] One is the first loss value of the trained voice interaction model obtained based on the question text features 222a and question audio features 224a corresponding to each question text. The other is the second loss value of the trained voice interaction model obtained based on the predicted response text features 228a and sample response text features 222b corresponding to each question text.

[0136] Example embodiment methods for obtaining the first loss value and the second loss value are described in detail below.

[0137] Firstly, regarding the first loss value:In an example embodiment, the first loss value may measure an alignment between the speech interaction model's understanding of the two modalities of question text and question audio by calculating a semantic difference between the question text features 222a and the question audio features 224a. When the speech interaction model can accurately align the text features with the audio features, this indicates that the model can simultaneously understand the acoustic information of speech and the semantic information of language, thereby avoiding loss of audio features.

[0138] The degree of semantic difference between the question text features 222a and question audio features 224a recognized by the voice interaction model affects the model's ability to accurately output responses tailored to user speech. Therefore, it may be necessary to determine the first loss value of the trained voice interaction model based on the question text features 222a and question audio features 224a corresponding to each question text as described herein. The first loss value may be used to reflect the degree of semantic difference between the question text features 222a and question audio features 224a recognized by the voice interaction model. By referencing the first loss value in the judgment of whether the voice interaction model has converged, it is possible to control the trained voice interaction model to align the question text features 222a with the question audio features 224a, thereby avoiding the loss of audio features that affect response accuracy during the recognition process of the semantic interaction model.

[0139] To enable the calculation of the first loss value, as shown in Figs. 2 and 3, the voice interaction model may be equipped with a first loss module 230 or function 330 for calculating a semantic difference between question text features 222a, 322a and question audio features 224a. 324a Based on this, the process for determining the first loss value of the trained voice interaction model based on the question text features 222a, 322a and question audio features 224a, 324a corresponding to each question text can include the following steps:

[0140] At a first step, each type of first loss function may be used to perform arithmetic processing on the question text features and question audio features corresponding to each question text to obtain the loss value calculated by each type of first loss function.

[0141] The first loss function 330 (and / or output calculated by first loss module 230) can be selected based on needs of the scenario to which the model is employed, and may be either one or multiple types. Considering that a single type of first loss function may experience occasional errors, leading to inaccurate calculation of the first loss value, multiple types of first loss functions may be used to reduce the impact of occasional errors of individual first loss functions on the accuracy of the first loss value.The first loss function can be flexibly selected based on needs of the scenario, and this embodiment does not impose limitations on this. Exemplary first loss functions may include, but are not limited to, at least one of the following: Mean Squared Error (MSE) loss function, Mean Absolute Error (MAE) loss function, Cross-Entropy loss function, and Huber loss function.

[0142] At a second step, based on the first function weight corresponding to each type of first loss function, the loss values obtained by each type of first loss function may be weighted to obtain the first loss value of the trained voice interaction model.

[0143] Each type of first loss function may have a corresponding first function weight, which indicates the credibility of the first loss function in calculating the semantic difference between question text features and question audio features. The first function weight corresponding to the first loss function can be pre-set based on the historical credibility of the first loss function in calculating the semantic difference between question text features and question audio features during historical model training.

[0144] After obtaining the corresponding loss value for each type of first loss function, based on the first function weight corresponding to each type of first loss function, the loss values obtained by each type of first loss function may be weighted. The process of weighted calculation can be: determining the product between the loss value and the first function weight for each type of first loss function, and summing the products corresponding to each type of first loss function to determine the first loss value of the trained voice interaction model.

[0145] Secondly, regarding the second loss value:

[0146] The degree of difference between the predicted response text 228a and the sample response text 222b recognized by the voice interaction model may affect the model's ability to accurately output responses tailored to user speech. Therefore, it may be necessary to determine the second loss value of the trained voice interaction model based on the predicted response text 228a and sample response text 222b corresponding to each question text. The second loss value may be generated or calculated by second loss module 240, and / or second loss function 340, and may be used to reflect the degree of difference between the predicted response text 228a recognized by the voice interaction model and the real response text, e.g. a "sample response file".

[0147] To enable the calculation of the second loss value, as shown in Figs. 2 and 3, the voice interaction model may be equipped with a second loss function 340 and / or second loss module 240 for calculating the difference between the predicted response text 228a and the sample response text 222b, and the second loss function may include at least one type. Based on this,the process for determining the second loss value of the trained voice interaction model based on the predicted response text and sample response text corresponding to each question text can include the following steps:

[0148] At a first step, each type of second loss function may be used to perform arithmetic processing on the predicted response text 228a and sample response text 222b corresponding to each question text to obtain the loss value calculated by each type of second loss function.

[0149] The second loss function can be selected based on scenario needs, either one or multiple types. Considering that a single type of second loss function may experience occasional errors, leading to inaccurate calculation of the second loss value, multiple types of second loss functions may be used to reduce the impact of occasional errors of individual second loss functions on the accuracy of the second loss value. The second loss function can be flexibly selected based on scenario needs, and this embodiment does not impose limitations on this. Exemplary second loss functions may include, but are not limited to, at least one of the following: Mean Squared Error (MSE) loss function, Mean Absolute Error (MAE) loss function, Cross-Entropy loss function, and Huber loss function.

[0150] At a second step, based on the second function weight corresponding to each type of second loss function, the loss values obtained by each type of second loss function may be weighted to obtain the second loss value of the trained voice interaction model. The second function weight may indicate the credibility of the second loss function in the difference between the predicted response text and the sample response text.

[0151] Each type of second loss function may have a corresponding second function weight, which indicates the credibility of the second loss function in calculating the difference between the predicted response text 228a and the sample response text 222b. The second function weight corresponding to the second loss function can be pre-set based on a historical credibility of the second loss function in calculating the difference between the predicted response text and the sample response text during historical model training.

[0152] After obtaining the corresponding loss value for each type of second loss function, based on the second function weight corresponding to each type of second loss function, the loss values obtained by each type of second loss function can be weighted. The specific process of weighted calculation can be:

[0153] - determining the product between the loss value and the second function weight for each type of second loss function, and

[0154] - summing the products corresponding to each type of second loss function to determine the second loss value of the trained voice interaction model.In example embodiments, such as that described in relation to Fig. 1, at step 150, if it is determined that the trained speech interaction model converges based on the first loss value, such as from first loss module 230, or first loss function 330, and the second loss value, such as from second loss module 240 or second loss function 340, the speech interaction model is determined to be trained or a well-trained speech interaction mode. Accordingly, the speech interaction model may be deemed ready for operation as a speech interaction model to respond to one or more user questions in a user interaction scenario.

[0155] The first loss value and the second loss value may serve as the basis for determining whether the trained speech interaction model converges. If the trained speech interaction model is determined to converge, it indicates that the model has the ability to accurately output responses tailored to user speech, and the trained speech interaction model can be put into use. If the trained speech interaction model is determined not to converge, it indicates that the model does not have the ability to accurately output responses tailored to user speech, and the trained speech interaction model still requires iterative training or additional training. Based on this, after determining the first loss value and the second loss value, it may be necessary to judge whether the trained speech interaction model converges based on these two values. The methods for judging whether the trained speech interaction model converges based on the first loss value and the second loss value can include at least the following four taken alone or in any combination:

[0156] Method 1: The process forjudging whether the trained speech interaction model converges based on the first loss value and the second loss value can include the following steps, as set out in method 400 as shown in Fig. 4:

[0157] • Processing or summing 420 the first loss value 412 and the second loss value 414 through addition to obtain a total loss value;

[0158] • judging or comparing 430 whether the total loss value is not greater than a first threshold value 432; and

[0159] • if it is not greater 442, then determining that the trained speech interaction model converges;

[0160] • if it is greater 444, then determining that the trained speech interaction model does not converge.

[0161] Both the first loss value 412 and the second loss value 414 are the basis for determining whether the trained speech interaction model converges, so they can be summarized together 420 to determine the convergence of the trained speech interaction model. In this example embodiment, the process of summarizing the first loss value 412 and the second loss value414 is the process of processing them through addition 420 to obtain a total loss value, which may include at least the following two implementation methods:

[0162] One is to obtain loss weights corresponding to the first loss value and the second loss value respectively, where the loss weights indicate the degree of influence of the corresponding loss values on the convergence of the speech interaction model. Then, based on the obtained loss weights, performing weighted calculation on the first loss value and the second loss value to obtain the total loss value. The loss weights can be preset based on the historical influence of the first loss value and the second loss value on the convergence of the speech interaction model during historical model training.

[0163] The other is to determine the sum 420 of the first loss value 412 and the second loss value 414 as the total loss value.

[0164] The first threshold value 432 may be a boundary value used to indicate whether the speech interaction model converges. If the total loss value is judged to be greater than the first threshold value 432, it indicates that the trained speech interaction model does not have good ability to accurately output responses tailored to user speech and still requires iterative training, so it is determined that the trained speech interaction model does not converge 444. If the total loss value is judged to be not greater than the first threshold value 432, it indicates that the trained speech interaction model has good ability to accurately output responses tailored to user speech, and the trained speech interaction model can be put into use, so it is determined that the trained speech interaction model converges 442.

[0165] In an example embodiment, the first threshold 432 may be obtained based on historical training events of the or a speech interaction model. During a training process, an appropriate first threshold 432 can be determined by observing the model's performance under different first loss values 412. When the first loss value 412 is below the first threshold 432, the model's performance may be considered to meet expectations of speech interaction performance. It will be understood that the first threshold 432 may be determined or arrived upon based on a scenario to which the speech interaction model is implemented, and / or based on historical training of the or a speech interaction model.

[0166] As described above, method 1 may include that the first loss value 412 is calculated based on the question text features and question audio features, while the second loss value 414 is calculated based on the predicted reply text and sample reply text. Additional features may include weighted operation of loss values 412 and 414 as described above.Method 2: The process forjudging whether the trained speech interaction model converges based on the first loss value and the second loss value can include the following steps, as set out in method 500 as shown in Fig. 5:

[0167] • Processing or summing 520 the first loss value 512 and the second loss value 514 through addition to obtain a total loss value;

[0168] • judging whether the difference between the total loss value and a total loss value 531 obtained from a previous training of the speech interaction model is within a first threshold range 532; and

[0169] • if it is not 544, then determining that the trained speech interaction model does not converge;

[0170] • if it is 542, then judging whether the total loss values obtained from the previous first number of times of training the speech interaction model show a downward trend, and whether the decrease in the total loss value of each adjacent later time compared to the previous time is within the first threshold range 532,

[0171] • if yes 552, then determining that the trained speech interaction model converges, and • if no 554, then determining that the trained speech interaction model does not converge.

[0172] The process 520 for processing the first loss value 512 and the second loss value 514 through addition to obtain a total loss value can be that as described above in regard to Method 1 and Fig. 4, and is not repeated here.

[0173] In this example embodiment, the first threshold range 532 may be a threshold range to which the total loss value belongs when the speech interaction model converges. The first threshold range 532 can be preset based on the historical total loss value when the model converges during historical model training. If the difference between the total loss value and the total loss value obtained from the previous training 531 of the speech interaction model (if the current training is the first training, the total loss value of the previous training can be defaulted to 0) is judged to be outside the first threshold range 532, it indicates that the trained speech interaction model does not have good ability to accurately output responses tailored to user speech and still requires iterative training, so it is determined that the trained speech interaction model does not converge 544. If the difference is judged to be within the first threshold range 532, it indicates that the trained speech interaction model has the possibility of convergence 542, so it may be necessary to continue judging whether the total loss values obtained from the previous first number of times of training the speech interaction model show a downward trend 542, and whether the decrease in the total loss value of each adjacent latertime compared to the previous time is within the first threshold range. The first number here can be set in advance based on empirical data from historical model training.

[0174] If it is judged that the total loss values obtained from the previous first number of times of training the speech interaction model do not show a downward trend, or show a downward trend but the decrease in the total loss value of any adjacent later time compared to the previous time is not within the first threshold range, it indicates that the difference between the total loss value of the current training and the total loss value obtained from the previous training of the speech interaction model being within the first threshold range is only a coincidence, and the trained speech interaction model does not have good ability to accurately output responses tailored to user speech and still requires iterative training, so it is determined that the trained speech interaction model does not converge 554.

[0175] If it is judged that the total loss values obtained from the previous first number of times of training the speech interaction model show a downward trend, and the decrease in the total loss value of each adjacent later time compared to the previous time is within the first threshold range, it indicates that the difference between the total loss value of the current training and the total loss value obtained from the previous training of the speech interaction model being within the first threshold range is not a coincidence, and the trained speech interaction model has good ability to accurately output responses tailored to user speech, and the trained speech interaction model can be put into use, so it is determined that the trained speech interaction model converges 552.

[0176] Method 2 may include an additional feature of comparing whether the difference between the two total loss values lies within a certain threshold range, and determining whether the total loss values from an adjacent N previous trainings show a downward trend.

[0177] Method 3: The process forjudging whether the trained speech interaction model converges based on the first loss value and the second loss value can include the following steps, as set out in method 600 as shown in Fig. 6:

[0178] • Obtaining second threshold values 616 and 616’ corresponding to the first loss value 612 and the second loss value 614 respectively;

[0179] • judging 622, 624, whether the first loss value 612 and the second loss value 614 are not greater than their corresponding second threshold values 616, 616’ respectively; and

[0180] • ifthey are630, (i.e. loss values are not greater than threshold values) then determining that the trained speech interaction model converges;• if they are not 640, 640’, (i.e., loss values are greater than threshold values) then determining that the trained speech interaction model does not converge.

[0181] Both the first loss value 612 and the second loss value 614 may be the basis for determining whether the trained speech interaction model converges. Accordingly, both the first loss value 612 and the second loss value 614 can be used as separate factors to judge the convergence of the trained speech interaction model.

[0182] The first loss value 612 and the second loss value 614 may have corresponding second threshold values 616, 616’ respectively. The second threshold value 616 corresponding to the first loss value 612 may be the boundary value for the convergence of the speech recognition model under the factor of question text features and question audio features. The second threshold value 616 may comprise a second question threshold value 616.

[0183] The second threshold value 616’ corresponding to the second loss value 614 may be the boundary value for the convergence of the speech recognition model under the factor of predicted response text and sample response text. The second threshold value 616’ may comprise a second response threshold value 616’.

[0184] If it is judged that the first loss value 612 is greater than the corresponding second threshold value 616, and / or the second loss value 614 is greater than the corresponding second threshold value 616’, this indicates that the trained speech interaction model may still not be ideal in recognizing question text features and question audio features, and / or predicting response text. Accordingly, the trained speech interaction model may not have good ability to accurately output responses tailored to user speech and may still require iterative training, i.e. it is determined that the trained speech interaction model does not converge 640, 640’. If it is judged that the first loss value 612 is not greater than the corresponding second threshold value 616, and the second loss value 614 is also not greater than the corresponding second threshold value 616’, this indicates that the trained speech interaction model has reached an ideal state in recognizing question text features and question audio features, as well as predicting response text. Accordingly, the trained speech interaction model may have good ability to accurately output responses tailored to user speech, and the trained speech interaction model can be put into use, i.e. it is determined that the trained speech interaction model converges 630.

[0185] Method 3 may include the additional feature of clarifying how the threshold 616, 616’ for each loss value 612, 614 is set, and how these thresholds reflect different requirements for model convergence.Method 4: The process of judging whether the trained voice interaction model converges based on the first loss value and the second loss value can include the following steps, as set out in method 700 and shown in Fig. 7:

[0186] • Determining whether the differences between the first loss value 712 and the second loss value 714 and the respective first loss value 712’ and second loss value 714’ obtained from a previous training of the voice interaction model fall within corresponding second threshold ranges 716, 716’;

[0187] • if not, determining that the trained voice interaction model has not converged 722, 724;

[0188] and

[0189] • if so, judging 730 whether the first loss values 712’ and second loss values 714’ obtained from a preceding second number of trainings of the voice interaction model both show a decreasing trend, and whether the decrease in the first loss value 712 and the decrease in the second loss value 714 in any two adjacent trainings fall within the corresponding second threshold ranges 716, 716’,

[0190] • if yes 742, determining that the trained voice interaction model has converged; and • if no 744, determining that the trained voice interaction model has not converged. The first loss value 712 and the second loss value 714 may each have corresponding second threshold ranges 716, 716’. The second threshold range 716 corresponding to the first loss value 712 may be the threshold range to which the first loss value 712 belongs when the voice interaction model converges. The second threshold range 716 may comprise a second question threshold value 716.

[0191] The second threshold range 716’ corresponding to the second loss value 714 may be the threshold range to which the second loss value 714 belongs when the voice interaction model converges. The second threshold range 716’ may comprise a second response threshold range 716’.

[0192] The second threshold ranges 716, 716’ can be pre-set based on the historical loss values corresponding to model convergence in historical model trainings.

[0193] If it is determined that the differences between the first loss value 712 and the second loss value 714 and the respective first loss value 712’ and second loss value 714’ obtained from a previous training of the voice interaction model do not fall within the corresponding second threshold ranges 716, 716’, this indicates that the trained voice interaction model's ability to accurately output responses tailored to user speech may be poor, and iterative training may still be required, thus determining that the trained voice interaction model has not converged 722, 724.If it is determined that the differences between the first loss value 712 and the second loss value 714 and the respective first loss value 712’ and second loss value 714’ obtained from the previous training of the voice interaction model fall within the corresponding second threshold ranges 716, 716’, this suggests the possibility of convergence of the trained voice interaction model. Therefore, it may be necessary to continue judging 730 whether the first loss values 712, 712’ and second loss values 714, 714’ obtained from a preceding second number of trainings of the voice interaction model both show a decreasing trend, and whether the decrease in the first loss value and the decrease in the second loss value in any two adjacent trainings fall within the corresponding second threshold ranges 716, 716’. Here, the second number can be set in advance based on empirical data from historical model trainings. If it is determined that at least one of the first loss values and second loss values obtained from the preceding second number of trainings of the voice interaction model does not show a decreasing trend, or both show a decreasing trend but the decrease in the first loss value or the decrease in the second loss value in any two adjacent trainings does not fall within the corresponding second threshold ranges 716, 716’, this indicates that the current training's result, where the differences between the first loss value 712 and the second loss value 714 and the respective first loss value 712’ and second loss value 714’ obtained from the previous training fall within the corresponding second threshold ranges 716, 716’, is merely coincidental. Accordingly, the trained voice interaction model's ability to accurately output responses tailored to user speech may be deemed poor, and iterative training may still be required, thus determining that the trained voice interaction model has not converged 744.

[0194] If it is determined that the first loss values and second loss values obtained from the preceding second number of trainings of the voice interaction model both show a decreasing trend, and the decrease in the first loss value and the decrease in the second loss value in any two adjacent trainings fall within the corresponding second threshold ranges 716, 716’, this indicates that the current training's result is not coincidental and that the trained voice interaction model has a good ability to accurately output responses tailored to user speech. Accordingly, the trained voice interaction model can be put into use, thus determining that the trained voice interaction model has converged 742.

[0195] Method 4 may include the additional feature of comparing whether the difference between the current loss values 712, 714 and previous loss values 712’, 714’ lies within a certain threshold range, and determining whether the loss values from an adjacent M previous trainings all show a downward trend.

[0196] The above four methods for judging whether the trained voice interaction model has converged based on the first loss value and the second loss value can be selected and usedbased on requirements of the scenario to which the model is employed, and example embodiments disclosed herein do not impose limitations on this. For example, when two or more methods are selected for use, the judgment results of each method may be crosschecked, meaning that the voice interaction model is determined to have converged only when all methods indicate convergence, and is determined to have not converged if any method indicates non-convergence.

[0197] In example embodiments, the above described 4 methods can be selected or combined based on specific circumstances or scenarios to which the model is implemented. For example, Method 1 400 can be used for initial judgment, followed by a more detailed analysis using Method 2 500 or Method 4 700. If finer control is required, Method 3600 can be considered to set different thresholds.

[0198] After the above method forjudging whether the trained voice interaction model has converged based on the first loss value and the second loss value, if the trained voice interaction model is determined to have converged based on the first loss value and the second loss value, the trained voice interaction model may be determined as a trained and / or ready-to-use voice interaction model for deployment.

[0199] After the above method forjudging whether the trained voice interaction model has converged based on the first loss value and the second loss value, if the trained voice interaction model is determined to have not converged based on the first loss value and the second loss value, the model parameters of the trained voice interaction model may be adjusted, and the adjusted voice interaction model may be used as the model to be trained. In this case the process may return to the step 108 of Fig. 1 of inputting the training sample set into the model to be trained for model training, obtaining 110 the question text features corresponding to each question text, the predicted response text, and the question audio features corresponding to the question audio, to continue iteratively training the voice interaction model.

[0200] If the trained voice interaction model is determined to have not converged based on the first loss value and the second loss value, this indicates that the trained voice interaction model has still not converged. At this point, the model parameters of the trained voice interaction model can be adjusted. The model parameters described herein may be those related to the voice interaction model's ability to accurately output responses tailored to user speech and can include, but are not limited to, at least one of the following: model parameters and hyperparameters.

[0201] To adjust the model parameters of the trained voice interaction model, at least one of the following trends may be determined:• The trend between the first loss value and the first loss value of the previously trained voice interaction model.

[0202] • The trend between the second loss value and the second loss value of the previously trained voice interaction model.

[0203] • The trend between the total loss value calculated based on the first loss value and the second loss value and the total loss value of the previously trained voice interaction model.

[0204] For each model parameter, the following process may be performed:

[0205] • A preset adjustment table may be searched to determine the adjustment value corresponding to the current model parameter and the determined trend, and

[0206] • the current model parameter may be adjusted based on the adjustment value.

[0207] The preset adjustment table may record various trends, and the corresponding adjustment values for the current model parameter under each trend.

[0208] After the adjustment is complete, the adjusted voice interaction model may be used as the model to be trained, and the process may return to the step 110 of Fig. 1 of inputting the training sample set into the model to be trained for model training, obtaining the question text features corresponding to each question text, the predicted response text, and the question audio features corresponding to the question audio, to reiteratively train the voice interaction model until it converges.

[0209] FIG. 8 is a block diagram illustrating a system architecture of a speech interaction model 800 according to various embodiments of the present disclosure.

[0210] As disclosed herein, the speech interaction model 800 may comprise a self-attention layer 810 configured to process question audio and generate one or more question audio features as described herein.

[0211] The speech interaction model 800 may further include an embedding layer 820 configured to process question text and sample response text, and respectively generate question text features and sample response text features as described herein.

[0212] The speech interaction model 800 may further include a multimodal features layer 830 configured to concatenate and / or generate one or more combined or connected data comprising the question audio features, question text features, and / or sample response text features as described herein.The speech interaction model 800 may further include a prediction layer 840 configured to generate one or more predicted sample response texts from at least the question audio features and the question text features.

[0213] The speech interaction model 800 may further include first and second loss value functions, 850, 860, configured to respectively generate first and second loss values. As described herein, the first value may be generated from the question audio features and question text features, and the second loss value may be generated from the sample response text, and predicted response text.

[0214] Embodiments of the present disclosure may be provided as a network of communicating devices (i.e. a “computerized network”). Embodiments of the invention may be also provided as a software application downloadable into a computer device to facilitate the method. The software application may be a computer program product, which may be stored on a non-transitory computer-readable medium on a tangible data-storage device (such as a storage device of a server, or one within a user device).

[0215] FIG. 9 is a block diagram illustrating a technical architecture of a computing server for implementing the processes or systems of any one of Figs. 1 to 8 according to various embodiments of the present disclosure. The technical architecture 900 represents a computer server suitable for use as the system 200 of Fig. 2, the system 300 of Fig. 3, or the speech interaction model 800 of Fig. 8, and / or for carrying out the training of a speech interaction model 100 of Fig. 1, or 400-700 of Figs. 4-7, according to various embodiments of the present disclosure. While a single computing server is shown, the methods may be implemented across multiple computers in a distributed computing environment.

[0216] The technical architecture 900 includes a processor 902 (which may be referred to as a central processor unit or CPU) that is in communication with memory devices including secondary storage 904 (such as disk drives), read only memory (ROM) 906, random access memory (RAM) 908. The processor 902 may be implemented as one or more CPU chips. The RAM 908 may be partitioned to efficiently process different tasks, such as data pre-processing, feature engineering, model training, and evaluation as described herein. The partitioning of RAM 908 allows for efficient processing of large-scale data processing tasks, which is particularly beneficial for complex predictive modelling operations. The technical architecture may further comprise input / output (I / O) devices 910, and network connectivity devices 912. The secondary storage 904 is typically comprised of one or more disk drives or tape drives and is used for non-volatile storage of data and as an over-flow data storage device if RAM908 is not large enough to hold all working data. Secondary storage 904 may be used to store programs which are loaded into RAM 908 when such programs are selected for execution. In an example embodiment, the secondary storage 904 includes a component 904a comprising non-transitory instructions operative by the processor 902 to perform various operations of the training of a speech interaction model 100 of Fig. 1 and as described herein. For example, the component 904a may implement the functionality of the various components and modules illustrated in Fig. 2, 3 or Fig. 8, and or implement the method steps as set out in Fig. 1, or Figs. 4-7.

[0217] The ROM 906 is used to store instructions and perhaps data which are read during program execution. The secondary storage 904, the RAM 908, and / or the ROM 906 may be referred to in some contexts as computer readable storage media and / or non-transitory computer readable media.

[0218] I / O devices 910 may include printers, video monitors, liquid crystal displays (LCDs), plasma displays, touch screen displays, keyboards, keypads, switches, dials, mice, track balls, voice recognizers, card readers, paper tape readers, or other well-known input devices.

[0219] The processor 902 executes instructions, codes, computer programs, scripts which it accesses from hard disk, floppy disk, optical disk (these various disk-based systems may all be considered secondary storage 904), flash drive, ROM 906, RAM 908, or the network connectivity devices 912. While only one processor 902 is shown, multiple processors may be present. Thus, while instructions may be discussed as executed by a processor, the instructions may be executed simultaneously, serially, or otherwise executed by one or multiple processors.

[0220] The network connectivity devices 912 enable the technical architecture 900 to communicate with external systems, which may include distributed computing resources or cloud-based services.

[0221] Although the technical architecture 900 is described with reference to a computer, it should be appreciated that the technical architecture 900 may be formed by two or more computers in communication with each other that collaborate to perform a task. For example, but not by way of limitation, an application may be partitioned in such a way as to permit concurrent and / or parallel processing of the instructions of the application. Alternatively, the data processed by the application may be partitioned in such a way as to permit concurrent and / or parallel processing of different portions of a data set by the two or more computers. In an embodiment, virtualization software may be employed by the technical architecture 900 toprovide the functionality of a number of servers that is not directly bound to the number of computers in the technical architecture 900. In an embodiment, the functionality disclosed above may be provided by executing the application and / or applications in a cloud computing environment. Cloud computing may comprise providing computing services via a network connection using dynamically scalable computing resources. A cloud computing environment may be established by an enterprise and / or may be hired on an as-needed basis from a third-party provider.

[0222] By programming and / or loading executable instructions onto the technical architecture 900, at least one of the CPU 902, the RAM 908, and the ROM 906 are changed, transforming the technical architecture 900 in part into a specific purpose machine or apparatus having the novel functionality taught by the present disclosure. It is fundamental to the electrical engineering and software engineering arts that functionality that can be implemented by loading executable software into a computer can be converted to a hardware implementation by well-known design rules. Such hardware implementations may include field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), which can further enhance the system's performance for specific information processing tasks.

[0223] In the foregoing detailed description, embodiments of the present disclosure in relation to a speech interaction model, system and / or method of operation or training of a speech interaction model are described with reference to the provided figures. The description of the various embodiments herein is not intended to call out or be limited only to specific or particular representations of the present disclosure, but merely to illustrate non-limiting examples of the present disclosure. The present disclosure serves to address at least one of the mentioned problems and issues associated with the prior art. Although only some embodiments of the present disclosure are disclosed herein, it will be apparent to a person having ordinary skill in the art in view of the present disclosure that a variety of changes and / or modifications can be made to the disclosed embodiments without departing from the scope of the present disclosure. It will be understood by those skilled in the art that many variations of the embodiments can be made within the scope of the following claims. Moreover, features of one or more embodiments may be mixed and matched with features of one or more other embodiments. Therefore, the scope of the disclosure as well as the scope of the following claims is not limited to embodiments described herein.

Claims

1. Claims1. A method for training a speech interaction model, the method comprising:receiving a training sample set including a plurality of question texts, corresponding sample response texts for each question text, and question audios associated with the question texts;inputting the training sample set into the speech interaction model to train the model, wherein the speech interaction model is configured to:generate question text features corresponding to each question text, generate predicted response texts, andgenerate question audio features corresponding to the question audios; determining a first loss value of the trained speech interaction model based on the question text features and the question audio features corresponding to each question text;determining a second loss value of the trained speech interaction model based on the predicted response texts and the sample response texts corresponding to each question text;determining if the trained speech interaction model has converged based on the first and second loss values; andestablishing the trained speech interaction model if the trained speech interaction model is determined to be convergent based on the first and second loss values.

2. The method of claim 1, wherein the question audios are derived through audio conversion of the question texts.

3. The method of claim 2, wherein the audio conversion process for user question audio comprises an audio synthesis model.

4. The method of any one of claims 1 to 3, further comprising inputting a combined question text and corresponding preset emotional and / or vocal timbre features of the question audio into an audio synthesis model, wherein the audio synthesis model is configured to convert text into speech with emotions and vocal timbres corresponding to the question audio.

5. The method of any one of claims 1 to 4, wherein the step of generating question audio features comprises generation of initial audio features comprising numerical representations of speech, optionally wherein the numerical representations of speech comprise acoustic information of pitch, timbre, and / or speaking rate, of the question audio.

6. The method of any one of claims 1 to 5, wherein the question audio features comprise one or more of spectral features, fundamental frequency features, energy features, and / or formant features.

7. The method of any one of claims 1 to 6, wherein the first loss value provides an indication of an alignment between the question text features and the question audio features.

8. The method of any one of claims 1 to 7, wherein the first threshold value is a boundary value indicating the speech interaction model converges, optionally wherein the first threshold value corresponds to a loss value of a previous speech interaction model training.

9. The method of any one of claims 1 to 8, wherein receiving the sample training set comprises extracting target question texts and corresponding target response texts from a question and answer data set, optionally wherein the extracted target question texts and corresponding target response texts relate to a scenario to which the model will be applied.

10. The method of any one of claims 1 to 9, wherein the step of determining if the trained speech interaction model has converged comprises:- processing a first loss value and a second loss value through addition to obtain a total loss value;- determining whether the total loss value is not greater than a first threshold value; and- if the total loss value is not greater than the first threshold value, determining that the trained speech interaction model has converged, or if the total loss value is greater than the first threshold value, determining that the trained speech interaction model has not converged.

11. The method of any one of claims 1 to 10, wherein the step of determining if the trained speech interaction model has converged comprises:- processing a first loss value and a second loss value through addition to obtain a total loss value;- determining whether a difference between the total loss value and a prior or historic total loss value from a previous training of the model is within a first threshold range; and - if the difference is not within the first threshold range, determining that the trained speech interaction model has not converged, or- if the difference is within the first threshold range, the model is determined to converge if a plurality of differences between the total loss value and prior total loss values shows a downward trend, and a decrease in the difference is within the first threshold range.

12. The method of any one of claims 1 to 11, further comprising obtaining second threshold values corresponding to the first loss value and the second loss value, wherein obtaining second threshold values comprises determining a second threshold value for each loss value,with a second question threshold value for the first loss value reflecting a convergence boundary under the factor of question text features and question audio features, and a second response threshold value for the second loss value reflecting a convergence boundary under the factor of predicted response text and sample response text.

13. The method of claim 12, further comprising:(a) determining whether the first loss value is not greater than the second question threshold value corresponding to the first loss value; and(b) determining whether the second loss value is not greater than the second response threshold value corresponding to the second loss value;wherein if both conditions (a) and (b) are satisfied, the trained speech interaction model is determined to have converged.

14. The method of any one of claims 1 to 13, further comprising determining whether the differences between the first loss value and the second loss value and a respective first loss value and second loss value obtained from a previous training of the trained speech interaction model fall within corresponding second threshold ranges.

15. The method of claim 14, wherein if the first loss values and second loss values obtained from the preceding second number of trainings both show a decreasing trend, and the decrease in the first loss value and the decrease in the second loss value in any two adjacent trainings fall within the corresponding second threshold ranges, the trained speech interaction model is determined to have converged.

16. A speech interaction model trained according to the method of any one of claims 1 to 15.

17. A system for training a speech interaction model, comprising:a training module configured to obtain a training sample set including a plurality of question texts, corresponding sample response texts for each question text, and question audios derived from the question texts;a processing module configured to input the training sample set into the speech interaction model to train the model, and generate question text features corresponding to each question text, predicted response texts, and question audio features corresponding to the question audios;a loss calculation module configured to determine a first loss value based on the question text features and question audio features, and a second loss value based on the predicted response texts and the sample response texts;a convergence determination module configured to determine if the trained speech interaction model has converged based on the first and second loss values; andan output module configured to establish the trained speech interaction model if the model is determined to be convergent.

18. The system of claim 17, further comprising an interaction module configured to receive both a user-inputted voice and corresponding text as inputs during a voice interaction with the trained speech interaction model.

19. An apparatus for training a speech interaction model, comprising:a processor configured to execute instructions to obtain a training sample set, input the training sample set into a speech interaction model fortraining, calculate loss values based on the input data and predicted results, and establish the trained speech interaction model if the model converges based on the calculated loss values;a memory configured to store the training sample set, including question texts, corresponding sample response texts, and question audios derived from the question texts, and to store the trained speech interaction model.

20. The apparatus of claim 19, further comprising a voice input component and / or a text input component configured to receive user voice and / or text inputs, respectively, and provide them as inputs to the trained speech interaction model during a voice interaction.

21. A system for training a speech interaction model, comprising:an audio module configured to receive question audio,an input text module, configured to receive question text and sample response text, anda large language model, comprising:an audio features module configured to process the question audio and generate one or more corresponding question audio features;a text processing module configured to process the question text and sample response text and generate one or more question text features, and one or more sample response text features;a multimodal module configured to receive the question audio features, question text features, and sample response text features, and generate a concatenated multimodal feature set;a prediction module configured to receive the multimodal feature set and generate a predicted response text;a first loss value module configured to receive the question audio features and question text features and generate a first loss value;a second loss module configured to receive the sample response text and predicted response text features and generate a second loss value; andan evaluation module configured to evaluate the first loss value and second loss value to determine if the model is trained.

Images