Information processing device and information processing method

Abstract

This information processing device (10) has an emotion recognition unit (20) that calculates an output of emotion recognition and / or gender recognition by using a pre-trained encoder-decoder transformer model that processes speech.

Description

Information processing device and information processing method

[0001] The present invention relates to an information processing device and an information processing method.

[0002] Speech emotion recognition is a technology that automatically recognizes human emotions from speech. Because speech emotion recognition plays an important role in mental health care and customer service, it has been the subject of extensive research in recent years. Speech emotion recognition is a technology that classifies speech utterances into emotion classes such as "Happy" or "Angry."

[0003] One model for speech emotion recognition is one that combines a large-scale speech recognition (Automatic Speech Recognition: ASR) model encoder with a smaller decoder to obtain emotion recognition results.

[0004] Ashish Vaswani, et al., “Attention Is All You Need,” Proc. NeurIPS, vol. 30, 2017.Erik Goron, et al., “IMPROVING DOMAIN GENERALIZATION IN SPEECH EMOTION RECOGNITION WITH WHISPER.”, In Proc. ICASSP 2024, pp.11631-11635, 2024.

[0005] Conventional technologies do not consider the text of the utterances and use small, simple decoders, making them unable to learn complex linguistic knowledge. As a result, conventional technologies cannot make predictions that take linguistic information into account, leading to problems with the accuracy of emotion recognition.

[0006] The present invention has been made in view of the above, and aims to provide an information processing device and an information processing method that can improve the accuracy of speech emotion recognition.

[0007] To solve the above-mentioned problems and achieve the objective, the information processing device according to the present invention is characterized by having a recognition unit that calculates the output of emotion recognition and / or gender recognition using a pre-trained encoder and decoder type Transformer model for processing speech.

[0008] Furthermore, the information processing method according to the present invention is characterized by including the step of calculating the output of emotion recognition and / or gender recognition using a pre-trained encoder-decoder type Transformer model that processes speech.

[0009] According to the present invention, the accuracy of speech emotion recognition can be improved.

[0010] Figure 1 is a diagram showing an example of the configuration of an information processing device according to the embodiment. Figure 2 is a diagram showing an example of the configuration of an emotion recognition model having the emotion recognition unit shown in Figure 1. Figure 3 is a diagram showing an example of a token sequence generated by the emotion recognition model shown in Figure 1. Figure 4 is a flowchart showing the processing procedure of the emotion recognition process in the embodiment. Figure 5 is a flowchart showing the processing procedure of the learning process of the emotion recognition model in the embodiment. Figure 6 is a diagram showing the results of an evaluation experiment. Figure 7 is a diagram showing an example of the recognition result of speech emotion recognition (SER). Figure 8 is a diagram showing the ASR result of the Transformer model before additional learning and the emotion recognition model in the embodiment. Figure 9 is a diagram showing a confusion matrix representing the misclassification of "Happy" and "Angry" in the conventional method and the embodiment. Figure 10 is a diagram showing an example of a computer in which the information processing device is realized when a program is executed.

[0011] [Embodiment] An embodiment of the present invention will be described in detail below with reference to the drawings. However, the present invention is not limited by this embodiment. Also, in the drawings, the same parts are denoted by the same reference numerals. In the following, when "^A" is written for A which is a vector, matrix or scalar, it is equivalent to "a symbol with "^" written directly above it".

[0012] [Information Processing Device] An information processing device according to the embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the information processing device according to the embodiment.

[0013] The information processing device 10 according to this embodiment performs emotion recognition using the entire pre-trained Transformer model. The information processing device 10 realizes emotion recognition using speech by adding tokens that represent emotion classes (emotion recognition) to the output of the Transformer model and making it recursively learnable.

[0014] The information processing device 10 according to this embodiment is an information processing device that is realized by loading a predetermined program into a computer or the like, which includes ROM (Read Only Memory), RAM (Random Access Memory), CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The information processing device 10 also has a communication interface for sending and receiving various types of information with other devices connected via a network or the like.

[0015] The information processing device 10 includes an emotion recognition unit 20 (recognition unit) and a learning control unit 30.

[0016] The emotion recognition unit 20 takes speech as input and outputs speech recognition information, gender information, and emotion class of the input speech. Emotion classes include, for example, "Happy" and "Angry". The emotion recognition unit 20 uses an emotion recognition model 21 (machine learning model) to calculate the output of emotion recognition and / or gender recognition. The emotion recognition model 21 is a pre-trained encoder and decoder type Transformer model that processes speech.

[0017] The emotion recognition model 21 extracts speech features from the audio and predicts speech recognition information, gender information, and emotion class of the input audio based on the extracted speech features. The learning control unit 30 controls the learning process of the emotion recognition model 21.

[0018] Figure 2 shows an example of the configuration of the emotion recognition model 21 that the emotion recognition unit 20 shown in Figure 1 has. As shown in Figure 2, the emotion recognition model 21 has an encoder 211 and a decoder 212.

[0019] The emotion recognition model 21 utilizes an entire pre-trained Transformer model for processing speech. For example, Whisper (Reference 1), an encoder-decoder type Transformer model, can be applied as the Transformer model. Reference 1: Alec Radford, et al., “Robust Speech Recognition via Large-Scale Weak Supervision.”, arXiv preprint arXiv:2212.04356, 2022.

[0020] The Whisper decoder, which is a Transformer model, functions as a language model. Therefore, the information processing device 10 can perform emotion recognition that takes into account more complex linguistic information compared to conventional technology (for example, Non-Patent Document 1). Furthermore, the information processing device 10 performs emotion recognition that explicitly takes into account the text of the utterance by performing generation conditioned on the speech recognition result.

[0021] The encoder 211 has N-layer encoders with Self-Attention. The encoder 211 extracts speech features H from the input speech X (e.g., Longmagnitude Mel spectrogram representation).

[0022]

[0023] In equation (1), ENC(•) represents the encoder 211. θ e This represents the initial parameters of the encoder 211.

[0024] The decoder 212 has an N-layer decoder layer with cross-attention.

[0025] Here, the information processing device 10 expands the vocabulary of the Transformer model (e.g., Whisper) by adding tokens that represent emotion recognition and subtask classes (speech recognition and gender recognition). For example, the token "<|happy|>" means the emotion class "Happy".

[0026] The input to decoder 212 is the speech feature H. The emotion recognition model 21 predicts an emotion class c ∈ {1, ..., C} from the input speech X.

[0027] The emotion recognition model 21 autoregressively generates a sequence of output tokens. The probability p for the i-th output token. i ∈(0,1) V+W This is calculated by equation (2).

[0028]

[0029] In equation (2), V is the original vocabulary size of the Transformer model (e.g., Whisper). W is the number of additional tokens. 1:i-1 This is a token sequence generated in the past. DEC(・) represents decoder 212. θ d This represents the initial parameters of the decoder 212.

[0030] Decoder 212 generates a new token by taking a previously output token sequence as input to the next decoder. Specifically, decoder 212 generates a new token by taking a single token sequence, which is a concatenation of tokens from previously output subtasks and main tasks, as input.

[0031] In this way, the emotion recognition unit 20 autoregressively generates a token sequence by concatenating the prediction results of multiple tasks related to speech recognition, gender recognition, and emotion recognition using a Transformer model.

[0032] [Model Learning Method] In this embodiment, the learning method of the emotion recognition model 21 executed by the learning control unit 30 is referred to as Serial MTL. In Serial MTL, the learning control unit 30 causes the emotion recognition model 21 to learn to predict a single token sequence that concatenates the results of the subtask and the main task.

[0033] The correct token sequence during learning is defined as in Equation (3).

[0034]

[0035] However, although the token sequence includes feature tokens specific to the Transformer model such as language tags and timestamp tags, they are omitted in the mathematical expressions.

[0036] Equation (4) is a token sequence of length I corresponding to the t-th task. (t) corresponding to the t-th task. Equation (5) represents an operation of concatenating the left and right sequences.

[0037]

[0038]

[0039] FIG. 3 is a diagram showing an example of a token sequence generated by the emotion recognition model 21 shown in FIG. 1. FIG. 3 shows an example of a token sequence when the subtasks are speech recognition and gender recognition, and the main task is speech emotion recognition. The subtasks do not necessarily have to include gender recognition.

[0040] In the example of FIG. 3, y (1) = (“Really”, “good”, “.”), y (2) = (“<|male|>”), y (3) = (“<|happy|>”). The other tokens are special tokens specific to the Transformer model (e.g., Whisper).

[0041] Thus, in the emotion recognition model 21, the token indicating the prediction result of the speech recognition task is output first, and then the token indicating the prediction result of the gender task is output. And in the emotion recognition model 21, the token indicating the prediction result of the emotion task is output third. The decoder 212 receives the speech feature amount H from the encoder 211 as an input, and generates new tokens using the token sequences of speech recognition, gender recognition, and emotion recognition output in the past as inputs.

[0042] The SerialMTL loss function L used by the learning control unit 30 is defined by equation (6). The emotion recognition model 21 is trained using the loss function L.

[0043]

[0044] Equation (7) shows that y is included in y (t) The index at the end of the sequence, with a sequence length of I (:0) Let = 0.

[0045]

[0046] P(y i |X) is p i The correct token among them is y i This shows the output probability assigned to it. The learning control unit 30 calculates the loss of each task using the loss function L, based on the sequence length I of the corresponding task. (:t) The data is normalized and then summed. This prevents tasks with long sequences, such as ASR, from dominating the learning process performed by the learning control unit 30, ensuring balanced learning even when the sequence lengths differ for each task.

[0047] [Processing Procedure for Emotion Recognition] Figure 4 is a flowchart showing the processing procedure for emotion recognition in the embodiment.

[0048] As shown in Figure 4, the emotion recognition unit 20 encodes the audio using the encoder 211 (step S11) to generate the audio feature quantity H. Subsequently, the emotion recognition unit 20 adds 1 to i using the decoder 212 (step S12) to generate the i-th token (step S13).

[0049] If the decoder 212 has not generated the special token <|endoftext|> (step S14: No), it returns to step S12 and sequentially generates the i-th token (i=1, 2…) (steps S12, S13). If the special token <|endoftext|> has been generated (step S14: Yes), the decoder 212 terminates token generation. When generating tokens, the decoder 212 uses Cross-Attention to refer to the audio feature H and Self-Attention to refer to the embedding representation of previously generated tokens.

[0050] [Learning Control Process] Figure 5 is a flowchart showing the processing procedure for the learning process of the emotion recognition model 21 in the embodiment.

[0051] As shown in Figure 5, the learning control unit 30 causes the encoder 211 of the emotion recognition model 21 to encode the speech for training (step S21), thereby generating speech feature quantities H. Subsequently, the emotion recognition unit 20 causes the decoder 212 to generate each of the 1st to Tth tokens (steps S22-1 to S22-T). The tokens generated by the decoder 212 indicate either speech recognition, gender recognition, or emotion recognition, and the token indicating speech recognition is variable.

[0052] The learning control unit 30 calculates the loss using the loss function L (equation (6)) (step S23), and optimizes the parameters of the emotion recognition model 21 based on the calculated loss (step S24).

[0053] [Evaluation Experiment] An evaluation experiment of the information processing device 10 was conducted. In the evaluation experiment, a comparative experiment was performed with a conventional method for the speech emotion recognition task.

[0054] The evaluation experiment used IEMOCAP, an English speech emotion recognition dataset. IEMOCAP contains approximately 12 hours of one-on-one conversational performance audio from 10 American English speakers, with transcripts and emotion labels added by three annotators. The evaluation experiment performed categorical emotion classification into four classes (Neutral, Happy+Excited, Sad, Angry).

[0055] A total of 5531 utterances were used in the evaluation experiment (Neutral: 1708, Happy: 1636, Sad: 1084, Angry: 1103). The evaluation experiment was conducted with a speaker-open setting where speakers did not overlap between the training and evaluation data. Leave-one-out cross-validation was performed using conversations from one speaker pair as evaluation data, one pair as validation data, and the remaining three pairs as training data.

[0056] In the evaluation experiment, two Whisper models were prepared as the basis for the speech emotion recognition model. These were a base model consisting of 6 Whisper encoder layers and 6 Whisper decoder layers, and a large-v3 model consisting of 32 Whisper encoder layers and 32 Whisper decoder layers. As evaluation metrics, Weighted Accuracy (WA), which is the average accuracy per utterance, and Unweighted Accuracy (UA), which is the average accuracy per class, were used.

[0057] Figure 6 shows the results of the evaluation experiment. Numbers 1 and 2 in Figure 6 represent emotion recognition models based on Whisper's base model, and numbers 3 and 4 represent emotion recognition models based on Whisper's large-v3 model. Numbers 1 and 3 correspond to the prior art (Non-Patent Literature 1), and numbers 2 and 4 represent the emotion recognition model 21 of the embodiment.

[0058] As shown in Figure 6, according to this embodiment, regardless of whether Whisper's base model or large-v3 model is used, the speech emotion recognition task can be solved with higher accuracy compared to the conventional technology.

[0059] Figure 7 shows an example of the recognition result of speech emotion recognition (SER).

[0060] The utterance in example (i) in Figure 7 contains the positive word "splendid," but the sentence as a whole conveys sarcasm and contempt, and the correct emotion is "Angry." In the conventional technology, the emotion class was incorrectly recognized as "Happy." In contrast, the embodiment correctly recognized the emotion as "Angry."

[0061] Example (ii) in Figure 7 shows an instance where the conventional technology incorrectly recognized the correct answer, "Happy," as "Angry." In this case as well, the embodiment correctly recognized it as "Happy."

[0062] The utterances "Happy" and "Angry" have similar prosodic features, and linguistic information is often needed to distinguish between them. The emotion recognition model 21 of this embodiment can correctly recognize emotion classes because it takes into account more complex linguistic information. As a result, it is believed that the emotion recognition model 21 of this embodiment has achieved an overall improvement in accuracy.

[0063] Figure 8 shows the ASR results of the Transformer model before additional training and the emotion recognition model 21 in the embodiment. Whisper-ER shows the Transformer model without additional training (Figure 5) in the embodiment. GR shows the result of gender recognition.

[0064] As shown in Figure 8, the emotion recognition model 21 shows improved ASR compared to before additional training. Therefore, it was found that even when additional training is performed in the embodiment, the speech recognition accuracy of the Transformer model can be sufficiently maintained.

[0065] Figure 9 is a diagram showing a confusion matrix representing the misclassification of "Happy" and "Angry" in the conventional method and the embodiment.

[0066] As shown in Figure 9(b), in this embodiment, the misclassification of "Happy" and "Angry" is reduced by 37.1% compared to the conventional technology (Figure 9(a)), accounting for a large portion of the overall misclassification reduction of 15.4%. This improvement in accuracy is thought to be due to the embodiment considering more linguistic information than the conventional technology.

[0067] [Effects of the Embodiment] In the emotion recognition model 21 employed by the information processing device 10 according to the embodiment, the decoder 212, which is a Transformer model, has the function of a language model. Therefore, as shown in the evaluation experiment, the information processing device 10 can perform highly accurate emotion recognition that takes into account more complex language information compared to conventional technology by utilizing the linguistic knowledge of the pre-trained Transformer model decoder. Furthermore, the information processing device 10 can perform emotion recognition that explicitly takes into account the text of the utterance by performing generation conditioned on the speech recognition result.

[0068] Furthermore, according to the information processing device 10, predictions can be made while referring to prediction tokens of subtasks generated in the past using Self-Attention. Therefore, by using speech recognition as a subtask, emotion recognition that explicitly considers the text of the utterance can be performed.

[0069] Then, in the information processing device 10, the loss function L calculates the loss of each task based on the sequence length I of that task. (:t) By normalizing the data, tasks with long sequences, such as ASR, can be prevented from becoming dominant, allowing for balanced learning even when sequence lengths differ for each task.

[0070] Therefore, according to this embodiment, when audio is used as input, emotion recognition results and gender recognition results can be output with high accuracy along with the speech recognition results.

[0071] Therefore, when the information processing device 10 is applied, for example, to conversational customer service or mental health care such as counseling or online medical consultation, the ability to recognize the emotions of the other party (customer or patient) allows the responder (customer service representative, medical professional, etc.) to take appropriate action in accordance with the other party's emotions.

[0072] Furthermore, since the information processing device 10 outputs both speech recognition and emotion recognition simultaneously, it is possible to create meeting minutes using automatic speech recognition, while simultaneously recognizing emotions and recording the emotion recognition results in correspondence with the corresponding locations in the meeting minutes. This allows users to review meetings and analyze conversations afterward. For example, if the user is a service provider, they can check whether the client had a positive or negative reaction to the user's service proposal, and then consider what kind of service to propose to the client next.

[0073] [Regarding the System Configuration of the Embodiment] Each component of the information processing device 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific forms of distribution and integration of the functions of the information processing device 10 are not limited to those shown in the figure, and all or part of it can be configured by functionally or physically distributing or integrating in any unit according to various loads and usage conditions.

[0074] Furthermore, each process performed in the information processing device 10 may be implemented, in whole or in part, by a CPU, a GPU (Graphics Processing Unit), and a program that is analyzed and executed by the CPU and GPU. Alternatively, each process performed in the information processing device 10 may be implemented as hardware using wired logic.

[0075] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated may be changed as appropriate unless otherwise specified.

[0076] [Program] Figure 10 shows an example of a computer in which the information processing device 10 is realized when a program is executed. The computer 1000 has, for example, memory 1010 and CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These parts are connected by a bus 1080.

[0077] Memory 1010 includes ROM 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0078] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define each process of the information processing device 10 are implemented as program modules 1093 in which code executable by the computer 1000 is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing processes similar to the functional configuration of the information processing device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0079] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0080] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0081] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in these embodiments. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on these embodiments are included in the scope of the present invention.

[0082] 10 Information processing device 20 Emotion recognition unit 21 Emotion recognition model 30 Learning control unit 211 Encoder 212 Decoder

Claims

1. An information processing device characterized by having a recognition unit that calculates an emotion recognition and / or gender recognition output using a pre-trained encoder and decoder type Transformer model for processing speech.

2. The information processing device according to claim 1, characterized in that the recognition unit autoregressively generates a token sequence by concatenating prediction results of multiple tasks related to speech recognition, gender recognition, and emotion recognition using the Transformer model.

3. The information processing device according to claim 2, characterized in that the Transformer model is learned using a loss calculation that normalizes the loss of each task by the sequence length of the corresponding task and sums the losses after normalization.

4. An information processing method performed by an information processing device, characterized by comprising the step of calculating an emotion recognition and / or gender recognition output using a pre-trained encoder-decoder type Transformer model for processing speech.

Images