System and method for measuring user empathy level in online environment

Abstract

The present invention relates to a system for measuring a user empathy level in an online environment and, particularly, to a system for predicting an empathy level on the basis of multi-modality (image, voice, text, and bio-signal) collected from a user in an online environment. According to the present invention, the system comprises: a data collection unit (100) for collecting, in real time, conversation data of a user in image, voice, text, and bio-signal modalities; a feature extraction unit (200) for extracting features of each modality from the collected data; a learning unit (300) for learning importance by using cross-attention between the respective modalities, and adjusting a weight for each modality through a bi-directional MLP-mixer and a weight-based fusion module; and a classification unit (400) for finally classifying an empathy level of the user by combining the learned features. Therefore, the system can measure a user empathy level on the basis of multi-modality (image, voice, text, and bio-signal) collected from a user in an online environment.

Description

System and Method for Measuring User Empathy Levels in Online Environments

[0001] The present invention relates to a system for measuring user empathy levels in an online environment, and more specifically, to a system for predicting empathy levels based on multimodalities (video, voice, text, biosignals) collected from a user in an online environment.

[0002] One of the social interactions necessary for people to maintain harmonious relationships is empathy. People want to be empathized with while sharing their stories with others, and they feel a sense of intimacy with those who show empathy.

[0003] Various psychological studies have found that when a person meets someone new, sharing personal stories leads to experiencing empathy from the other person. With the advancement of technology, the constraints on the space and time for human interaction are disappearing.

[0004] Previously, friendships were primarily formed offline by meeting in person, talking face-to-face, and sharing empathy, but recently, opportunities to communicate by utilizing online environments are increasing.

[0005] In an online environment, one can easily form new relationships through video chatting or exchanging text messages without meeting in person. Even online, one can see the other person's facial expressions in real time and engage in voice chatting and text conversations through various features. Through this process, one can empathize with or be empathized with others. The reason empathy is important online is that it is a major factor in determining a user's psychological well-being.

[0006] Although non-face-to-face conversations have become commonplace in the metaverse and online environments, there is a lack of system development to measure whether users are empathizing with one another in these settings. Furthermore, since empathy must be measured in real-time, real-time characteristics must be reflected, but research in this area is insufficient. There is a problem in that it is difficult to accurately measure empathy between users using a single modality (text) alone.

[0007] [Prior Art Literature]

[0008] [Patent Literature]

[0009] (Patent Document 1) Korean Published Patent No. 10-2011-0087636 (August 3, 2011)

[0010] According to the present invention, a system and method for measuring user empathy levels based on multimodalities (video, voice, text, biosignals) collected from a user in an online environment are provided.

[0011] According to the present invention, the system includes a data collection unit (100) that collects user conversation data in real time as video, voice, text, and biosignal modalities; a feature extraction unit (200) that extracts features of each modality from the collected data; a learning unit (300) that learns importance using cross-attention between each modality and adjusts weights for each modality through a bidirectional multilayer perceptron mixer and a weight-based fusion module; and a classification unit (400) that combines the learned features to finally classify the user's empathy level.

[0012] According to the present invention, there is an effect of being able to measure the user's empathy level based on multimodalities (video, voice, text, biosignals) collected from a user in an online environment.

[0013] FIG. 1 is a configuration diagram of a user empathy level measurement system in an online environment according to one embodiment of the present invention.

[0014] FIG. 2 shows a multimodal-based empathy level measurement model of a user empathy level measurement system in an online environment according to an embodiment of the present invention.

[0015] Figure 3 shows the Bi-directional MLP-Mixer structure of a user empathy level measurement system in an online environment according to one embodiment of the present invention.

[0016] FIG. 4 is an overall flowchart of a method using a user empathy level measurement system in an online environment according to one embodiment of the present invention.

[0017] In this invention, data of two people conversing with each other is collected as video, voice, text, and biosignals, reflecting real-time characteristics, and a multimodal empathy prediction model is developed using this data. In addition, to improve the accuracy of the prediction in the developed model, a new approach is proposed using a bidirectional multilayer perceptron mixer (Bi-directional MLP-Mixer) model and a weight-based fusion module.

[0018] In other words, the present invention proposes a novel Bi-directional MLP-Mixer layer. Furthermore, it predicts the user's empathy level through a fusion module that learns the weights of cross-attention between features of each modality and combines them based on weight ratios for each modality.

[0019] In this invention, facial features, emotion analysis, gestures, and gaze from a user's conversation video are used as visual features, voice signals are extracted from the user's voice and used as audio signals, and text and biosignals (Electrodermal activity, EDA; Blood Volume Pulse, BVP; Temperature, Metabolic Equivalent of Task, MET) are used for empathy measurement. The invention is configured to train a prediction model after extracting and combining multimodal features.

[0020] To efficiently learn the features of each modality, cross-attention is applied to each modality's features, and the weights of the resulting outputs are trained according to importance within the fusion module. This design allows for dynamic fusion during the model's training process to determine which modalities are considered more important when predicting empathy levels.

[0021] In one embodiment of the present invention, a model is proposed for measuring empathy levels through conversations between users in an online environment. Image features were extracted using Facenet, FER, Pose, and Gaze; voice features were extracted using the pre-trained models HuBERT and Wav2Vec2.0; text features were extracted using DistilKoBERT; and biosignals are embedded using a sequence-based embedding technique.

[0022] A Bi-directional MLP-Mixer model was used for video and audio features, while a Transformer model was used for text and biosignal features. The model was designed to learn during training which modalities play a more significant role in predicting empathy levels during the fusion process. Weights were assigned based on the importance of each modality at the fusion point and used for the final classification.

[0023] A system for measuring user empathy levels in an online environment according to one embodiment of the present invention will be described with reference to the attached drawings.

[0024] FIG. 1 is a configuration diagram of a user empathy level measurement system in an online environment according to one embodiment of the present invention.

[0025] As illustrated in FIG. 1, a user empathy level measurement system (10) in an online environment includes a data collection unit (100), a feature extraction unit (200), a learning unit (300), and a classification unit (400).

[0026] The data collection unit (100) collects the user's conversation data in real time as video, voice, text, and biosignal modalities.

[0027] The feature extraction unit (200) extracts features of each modality from the collected data.

[0028] The feature extraction unit (200) includes a video modality feature extraction module (210) that generates face embeddings using FaceNet512, analyzes emotional states through a Facial Emotion Recognition (FER) model, extracts gestures using Mediapipe, and analyzes gaze data through GazeTracking; a voice modality feature extraction module (220) that resamples voice signals to 16000Hz and extracts advanced voice features through HuBERT and Wav2Vec2.0 models; a text modality feature extraction module (230) that converts voice into text and extracts linguistic features from the converted text using a DistilKoBERT model; and a biosignal modality feature extraction module (240) that embeds biosignal data such as Electrodermal Activity (EDA), Blood Volume Pulse (BVP), body temperature, and Metabolism (MET) in sequence units to reflect temporal changes in biosignals.

[0029] The learning unit (300) learns the importance using cross-attention between each modality and adjusts the weights for each modality through a bi-directional MLP-Mixer and a weight-based fusion module.

[0030] This learning unit (300) includes a Bi-directional MLP-Mixer module (310) that generates richer feature representations by simultaneously learning temporal and contextual information through bidirectional processing of the input sequence of each modality; a Cross-attention module (320) that learns Cross-attention weights between each modality to reflect the importance of each modality and assigns higher weights to important modalities based on the association of interactions; and a weight-based fusion module (330) that dynamically learns the importance of each modality based on Cross-attention results and performs optimal empathy prediction by combining the features of all modalities based on weights.

[0031] The classification unit (400) combines learned features to finally classify the user's empathy level.

[0032] This classification unit (400) includes a feature fusion module (410) that combines features reflecting the weights of each modality extracted from the learning unit (300) to generate a single comprehensive feature vector, and an empathy level classification module (420) that classifies a predefined empathy level based on the comprehensive feature vector generated by the feature fusion module and finally determines the user's empathy state as one of seven empathy levels.

[0033] A user empathy level measurement system in an online environment according to one embodiment of the present invention is described in detail as follows.

[0034] FIG. 2 shows a multimodal-based empathy level measurement model of a user empathy level measurement system in an online environment according to an embodiment of the present invention.

[0035] For image and voice features, the Bi-directional MLP-Mixer model proposed in this invention is used, while for text and biosignal features, a Transformer model is implemented and used. For image and voice features, two or more features are used, and cross-attention is applied for feature fusion within each modality. Additionally, a module is used to fuse each modality by having the model learn the importance of each output and determining weight ratios based on that importance. The final output is classified into seven predefined empathy levels.

[0036] FaceNet512 and Facial Emotion Recognition (FER) are used to extract visual features from video data. Face embeddings are generated using FaceNet512. First, faces are detected in each frame using RetinaFace. Then, for each detected face, a 512-dimensional face embedding is generated using the FaceNet512 model.

[0037] This embedding is a feature vector useful for face identification. During the FER feature extraction process, the image is converted to RGB format, and facial emotions are analyzed using an FER emotion detector. The detected emotion values ​​are returned in the form of a list, extracted based on seven emotions (Happiness, Sadness, Anger, Surprise, Fear, Disgust, and Neutral). By extracting FaceNet512 features for face identification and FER features for emotion, and combining them, there is an advantage in being able to track changes in a person's emotions over time within a video. By analyzing the emotional change patterns of each speaker and training the model, it is possible to detect emotional exchanges during conversations. Through this, transient errors caused by changes in face orientation or lighting can be reduced, and temporal consistency can be improved.

[0038] In addition, it extracts features of pose and gaze based on the listener's behavior to assess the degree of empathy with the speaker. Mediapipe is used to analyze gestures such as head nodding and hand movements. First, Mediapipe is used to estimate the pose in each frame and extract values ​​as 3D coordinates. Landmarks on the listener's face and upper body can be used to detect head movements and shoulder movements. For gaze features, the GazeTracking library based on dlib is used to extract gaze direction and pupil position. The extracted features output the gaze direction value and the left / right pupil positions as x, y coordinate values. Interest or concentration on the conversation can be analyzed by observing how the gaze direction changes over time and variations in pupil size.

[0039] Before extracting features from the audio data, all audio data is resampled to 16,000 Hz. Hidden-Unit BERT (HuBERT) and Wav2Vec 2.0 are used as feature extractors for the resampled audio data. HuBERT and Wav2Vec 2.0 are self-supervised learning models for speech processing. Wav2Vec 2.0 learns features directly from speech data and pre-trains using a large amount of unlabeled speech data. Additionally, it uses contrastive learning to learn speech representations and performs the task of predicting quantized speech units. HuBERT is a successor model to Wav2Vec 2.0 and applies BERT's masking strategy to the speech domain. It is characterized by the use of pseudo-labels generated through clustering. When using speech features extracted by both HuBERT and Wav2Vec 2.0, different aspects of speech features can be captured because the two models represent speech in different ways. Furthermore, it is possible to reduce the bias of a single model and obtain a balanced feature representation. In other words, combining the features of two models has the advantage of improving the stability and accuracy of predictions.

[0040] Text data is generated by extracting text from audio files. To extract the text, the Google Web Speech API is configured to Korean. Features are extracted from the text data using DistilKoBERT.

[0041] DistilKoBERT is a lightweight BERT model similar to DistilBERT, characterized by being specialized for the Korean language.

[0042] DistilBERT compresses the knowledge of the BERT model using a knowledge distillation technique. This is also known as teacher-student learning, where knowledge is transferred from a pre-trained large-scale BERT model (teacher) to a small-scale BERT (student) through knowledge distillation. DistilBERT is known to be lighter and about 60% faster than BERT. Since text data is large in size in itself, lightweighting is necessary when using it in a multimodal manner. Therefore, in this embodiment, DistilKoBERT, which is suitable for this purpose, is used.

[0043] The biosignal data consists of 2-minute intervals and includes 3 data points per sample, with time point t0 at the start, time point t1 at 1 minute after the start, and time point t2 at 2 minutes after the start. Therefore, considering the sequence characteristics, the biosignal data is embedded in the form of number of samples x 3 x 4.

[0044] The model for text and biosignal features is designed and implemented based on a transformer encoder structure. This structure includes a feedforward layer, a transformer encoder layer, and a positional embedding layer. First, the feedforward layer is responsible for applying non-linear transformations within the model. The feedforward layer consists of two linear transformation layers and one dropout layer. The first linear transformation layer uses the Gelu activation function to convert the input data into medium-sized vectors. The second linear transformation layer converts the vectors back to their original size and passes them through a dropout layer to prevent overfitting. The transformer encoder layer combines self-attention and a feedforward network to learn the relationships between each element of the input sequence. The input data is first normalized, then learns its relationship with itself through multi-head attention. After normalization, it applies non-linear transformations through the feedforward layer. Positional embedding adds positional information to the input sequence, allowing the model to learn order information. It receives the maximum number of positional embeddings and the feature dimension as input to generate embedding vectors for each position. Finally, the implemented transformer encoder applies positional embeddings based on the sequence of input data and learns by sequentially passing through multiple transformer encoder layers. Through this process, the model learns the complex relationships between each element of the input sequence.

[0045] Figure 3 shows the Bi-directional MLP-Mixer structure of a user empathy level measurement system in an online environment according to one embodiment of the present invention.

[0046] This invention proposes a Bi-directional MLP-Mixer (Bi-Mixer). The Bi-LSTM model proceeds with training only in the forward direction, and adds a hidden layer at the last node to transmit information in the reverse direction, thereby processing information more flexibly. Based on this training method, the Bi-Mixer is designed, and its forward processing is identical to that of the existing MLP-Mixer method. By adding a hidden layer that transmits information in the reverse direction, the model is implemented as a bidirectional model. In the reverse mixer layer, input data is transposed in the reverse direction during the token and channel mixing process and processed in the reverse direction. This allows the model to process forward and reverse information simultaneously and transmit the combined output to the next layer.

[0047] The formula for the forward mixer layer of the Bi-Mixer is (1), and the formula for the reverse mixer layer is (2). The combined output is given by formula (3).

[0048] Equation (1)

[0049] , for

[0050] , for

[0051] This illustrates the process of processing input data X and intermediate output U in a layer of the Bi-directional MLP-Mixer model.

[0052] represents the data of the i-th channel of the input vector, and W1 and W2 are the result of performing layer normalization on the input X, and are learnable weight matrices. After performing a linear transformation on the layer-normalized X through W1, an activation function σ is applied to add non-linearity, and then W2 is multiplied again. Add the above results to finally Find .

[0053] is the i-th sequence position data of the intermediate output U obtained in the previous step. is the result of performing layer normalization on U. W3 and W4 are trainable weight matrices, and after performing a linear transformation on U, which has been layer normalized through W3, an activation function σ is applied to add non-linearity, and then multiplied by W4. Add the above results to get the final output It calculates the input data bidirectionally to obtain richer contextual information and enables the learning of complex patterns through non-linear activation.

[0054] (2)

[0055] , for

[0056] , for

[0057] This formula represents the reverse processing process of the Bi-directional MLP-Mixer model. It is an operation designed to process input data in reverse and integrate it with forward information.

[0058] This process processes input data in reverse to integrate bidirectional contextual information and enables the model to learn richer information by learning various patterns.

[0059] (3)

[0060] , for

[0061] This formula represents the process of combining forward and reverse processing results in a bi-directional MLP-Mixer model. is the final output of the forward processing, and is the final output of the reverse processing. By adding the two outputs It generates a combined final output. This combined output includes both forward and reverse information, allowing it to comprehensively reflect bidirectional contextual information. Through this, the model can learn more sophisticated and richer feature representations by integrating information and patterns obtained from both sides of the sequence.

[0062] In these equations, X represents the input tensor, U represents the intermediate output, Y represents the final output in each direction, and Z represents the combined final output. W1, W2, W3, and W4 are the trainable weight matrices, and σ is the non-linear activation function. LayerNorm represents layer normalization. i represents the channel index (from 1 to C), and j represents the sequence position index (from 1 to S). Forward processing consists of two stages.

[0063] In the first step, layer normalization is applied to the input X, followed by a transformation using weights W1 and W2. In the second step, a similar process is applied to the intermediate output U to generate the final output Y. Reverse processing has a structure similar to forward processing but processes the input sequence in reverse order. X{rev} represents the input in reverse order. This allows the model to consider the bidirectional context of the sequence. In the combined output, the output Y from forward processing and the output Y{rev} from reverse processing are added element-wise to generate the combined output Z. Through this process, bidirectional information is integrated.

[0064] Cross-modal transformers are used to fuse the features of each modality. In cross-modal transformers, the target modality and source modality are set first.

[0065] Then, Key, Query, and Value are generated through linear transformation. The scaled dot-product attention score of Key-Query is calculated, and the weighted sum of the values ​​is calculated using the attention score as the weight. In the baseline model used in this embodiment, the weighted sum calculation part, which is the final part of the crossmodal transformers, learns the weights of each output and performs the weighted sum based on them. The modified method can dynamically reflect the importance of each modality.

[0066] (4)

[0067]

[0068] Equation (4) relates to cross attention used in crossmodal transformers. C represents cross attention and is the Query (Q), Key (K), and Value (V). Here, dk represents the dimension of the key vector, and (Root dk) is a factor for scaling. It plays a role in increasing learning stability by preventing the inner product value from becoming too large.

[0069] It calculates the dot product of the Query and Key to determine the similarity between each Query and Key. Softmax is applied to the Query-Key similarity to normalize each value to between 0 and 1, ensuring that the total sum is 1. This generates weights indicating which Key is more important for a given Query. These calculated weights are multiplied by the Values ​​to generate the final output C. This result is a weighted sum of the Value information corresponding to the Query, representing the final attention value that reflects the association between the inputs.

[0070] (5)

[0071]

[0072] Formula (5) represents the weighted sum used in this embodiment.

[0073] Here, each input It is a method of summing up by assigning weights to it. The weights are It is calculated based on the raw weights. Is It serves the role of calculating the normalized weights for each It is multiplied by the corresponding normalized weights, and all multiplication results are added to obtain the final weighted sum.

[0074] WeightedSum calculates a weighted sum by assigning weights to each input vector. Here is a learnable initial weight vector. The initial weight vector is normalized through the softmax function. is the i-th input vector, and is the weight for the corresponding vector. The normalized weight w is for each input vector Calculate the weighted sum by multiplying by it to generate the final output.

[0075] FIG. 4 is an overall flowchart of a method using a user empathy level measurement system in an online environment according to one embodiment of the present invention.

[0076] As illustrated in FIG. 4, in a method using a user empathy level measurement system in an online environment (hereinafter referred to as the user empathy level measurement system), (a) the user empathy level measurement system collects conversation data between users in real time as video, voice, text, and biosignal modalities.

[0077] Next, (b) the user empathy level measurement system extracts the characteristics of each modality from the collected data.

[0078] Next, (c) the user empathy level measurement system learns the importance between modalities using cross-attention on the features of each extracted modality, and generates learned features by adjusting the weights for each modality through a bi-directional MLP-Mixer and a weight-based fusion module.

[0079] And (d) the user empathy level measurement system combines each learned modality feature to generate a single comprehensive feature vector, and predicts the empathy level based on this vector to classify the user's empathy state into one of seven predefined empathy levels.

[0080] The proposed invention can be used as a system capable of measuring the empathy levels of users conversing in the metaverse and online environments in real time. Furthermore, it is possible to measure whether and to what extent a listener (e.g., counselor, doctor, etc.) empathizes with the words of a speaker (e.g., client, patient, etc.) during psychological counseling or telemedicine in the metaverse and online environments.

[0081] Regarding the terms, they are listed as follows: FER: Facial Emotion Recognition, Q: Query, K: Key, V: Value, EDA: Electrodermal Activity, BVP: Blood Volume Pulse, MET: Metabolic Equivalent of Task, TF: Transformer, Bi-M: Bi-Mixer

[0082] As online interactions increase in modern society, the importance of measuring empathy in non-face-to-face environments is being highlighted. This system invents a model that accurately predicts the level of empathy between users by comprehensively utilizing multimodal data (video, audio, text, and biosignals). The key features of this invention are a bidirectional MLP-mixer layer and a weight-based fusion module.

[0083] The Bi-directional MLP-Mixer captures richer contextual information by processing input sequences bidirectionally. This enables more accurate analysis of emotional changes and interaction patterns over time. The weight-based fusion module dynamically learns the importance of each modality, allowing it to focus on the most relevant information depending on the situation. This system collects and analyzes various data from two users conversing in real time. For video data, it uses FaceNet512 and FER to extract facial features and emotions, and analyzes gesture and gaze information through Mediapipe and GazeTracking. For voice data, it extracts advanced speech features using the HuBERT and Wav2Vec2.0 models. Text data is processed efficiently using DistilKoBERT, and biosignals are subjected to special embedding techniques that consider sequence characteristics. This technology can be utilized in various fields and is expected to make interactions between users richer and more meaningful within the metaverse environment. In online psychological counseling, it can help to more accurately identify the level of empathy during the process of building rapport between the counselor and the client, and in telemedicine, it can be used to comprehensively evaluate how much a doctor empathizes with a patient.

[0084] This invention is a research project conducted with funding from the government (Ministry of Science and ICT) and supported by the National Research Foundation of Korea (RS-2023-00219107).

[0085] This invention was carried out as a result of the Artificial Intelligence Convergence Innovation Talent Development Project of the Ministry of Science and ICT and the Information and Communications Planning and Evaluation Institute (IITP-2023-RS-2023-00256629).

[0086] This invention was carried out as a result of research conducted by the University ICT Research Center Project of the Ministry of Science and ICT and the Information and Communications Planning and Evaluation Institute (IITP-2024-RS-2024-00437718).

Claims

1. A data collection unit (100) that collects user conversation data in real time as video, voice, text, and biosignal modalities, A feature extraction unit (200) that extracts features of each modality from the collected data above, A learning unit (300) that learns importance using cross-attention between each modality and adjusts modality-specific weights through a bi-directional MLP-Mixer and a weight-based fusion module, and A user empathy level measurement system in an online environment characterized by including a classification unit (400) that combines learned features to finally classify the user's empathy level.

2. In Paragraph 1, The above feature extraction unit A video modality feature extraction module (210) that generates face embeddings using FaceNet512, analyzes the emotional state through a FER (Facial Emotion Recognition) model, extracts gestures using Mediapipe, and analyzes gaze data through GazeTracking, A speech modality feature extraction module (220) that resamples a speech signal and extracts advanced features of speech through HuBERT and Wav2Vec2.0 models, A text modality feature extraction module (230) that converts speech into text and extracts linguistic features from the converted text using a DistilKoBERT model, A user empathy level measurement system in an online environment characterized by including a biosignal modality feature extraction module (240) that reflects temporal changes in biosignals by embedding biosignal data such as Electrodermal Activity (EDA), Blood Volume Pulse (BVP), body temperature, and Metabolism (MET) in sequence units.

3. In Paragraph 1, The above learning unit (300) A Bi-directional MLP-Mixer module (310) that generates richer feature representations by simultaneously learning temporal and contextual information through bidirectional processing of the input sequence of each modality, A Cross-attention module (320) that learns Cross-attention weights between each modality to reflect the importance of each modality and assigns higher weights to important modalities based on the association of interactions, and A user empathy level measurement system in an online environment characterized by including a weight-based fusion module (330) that dynamically learns the importance of each modality based on cross-attention results and performs optimal empathy prediction by combining the features of all modalities based on weights.

4. In Paragraph 1, The above classification unit (400) A feature fusion module (410) that combines features reflecting the weights of each modality extracted from the learning unit (300) to generate a single comprehensive feature vector, A user empathy level measurement system in an online environment characterized by including an empathy level classification module (420) that classifies a predefined empathy level based on a comprehensive feature vector generated in the above feature fusion module and finally determines the user's empathy state as one of seven empathy levels.

5. In a method using a user empathy level measurement system in an online environment, (a) A step in which the above-described user empathy level measurement system collects conversation data between users in real time as video, voice, text, and biosignal modalities; (b) A feature extraction step in which the above-described user empathy level measurement system extracts features of each modality from the collected data; (c) A step in which the user empathy level measurement system learns the importance between modalities using cross-attention on the features of each extracted modality, and generates learned features by adjusting the weights for each modality through a bi-directional MLP-Mixer and a weight-based fusion module; and (d) a step of combining each learned modality feature of the user empathy level measurement system to generate a single comprehensive feature vector, and predicting the empathy level based on this vector to classify the user's empathy state into one of seven predefined empathy levels; a method using a user empathy level measurement system in an online environment characterized by including

Images