Multi-modal emotion recognition brain-computer interface instrument based on contrastive learning gated network

Abstract

Provided in the present invention is a multi-modal emotion recognition instrument based on a contrastive learning gated network. Firstly, a hardware collection device is designed by means of an electroencephalogram and an eye-tracking signals, so as to ensure efficient and accurate data collection; and secondly, a multi-modal contrastive learning gated network is designed, so as to improve the accuracy of emotion recognition.

Description

A multimodal emotion recognition brain-computer interface instrument based on contrastive learning gating networks

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202510190843.0, filed on February 20, 2025, entitled "A Multimodal Emotion Recognition Brain-Computer Interface Instrument Based on Contrastive Learning Gated Network", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention belongs to the field of brain-computer interfaces, and in particular relates to a multimodal emotion recognition brain-computer interface instrument based on a contrastive learning gating network. Background Technology

[0004] With the development of computer and information technology, machines with emotion recognition capabilities can significantly improve the user experience of human-computer interaction, providing a smoother and more natural interface. Against this backdrop, emotion recognition has received widespread attention from academia and industry, and has been extensively applied in fields such as autonomous driving, healthcare, and intelligent robotics. Existing emotion recognition research generally uses input methods categorized into physiological signals and non-physiological signals. Physiological signals have the advantage of being unaffected by human subjective consciousness, and can more realistically and reliably reflect changes in human emotions.

[0005] Research on emotion recognition using EEG and peripheral physiological signals has a long history. However, traditional deep learning feature extractors typically focus only on specific feature scales, such as single spatial, temporal, or frequency domain features. This limitation results in an insufficiently comprehensive range of extracted features, potentially overlooking crucial multidimensional information in emotion recognition. Therefore, comprehensive multi-scale features are essential for improving the accuracy of emotion recognition. Effectively integrating these multi-scale features can help capture more subtle emotional changes, thereby improving the model's performance and robustness.

[0006] Traditional emotion recognition methods primarily rely on data from a single modality. However, human emotional expression is diverse, with different emotional information embedded in various forms of physiological signals. Therefore, it is necessary to combine multiple physiological signals to express emotions. By leveraging the differences and complementarities among different modalities, more comprehensive emotional information can be obtained.

[0007] In existing multimodal emotion recognition, a common approach is to simply concatenate or fuse features from different modalities. This simple feature fusion method may fail to fully capture the complex relationships between different modalities, thus affecting model performance. To effectively fuse features from multiple sources, some studies use contrastive learning methods to learn the similarities and differences between different modalities, thereby extracting and aligning semantic information from data from different modalities.

[0008] Furthermore, in multimodal tasks, each modality possesses unique advantages and information. A significant challenge lies in how to flexibly select the optimal modality based on specific circumstances and effectively switch between the dominant modality and multiple modalities. Existing multimodal emotion recognition methods lack flexibility in modality selection and switching, resulting in the failure to fully utilize all valuable information from different modalities in certain samples.

[0009] Therefore, in multimodal emotion recognition, we need to pay more attention to the diversity and complementarity of features and explore more efficient dynamic decision-making mechanisms to achieve a more accurate and natural human-computer interaction experience. Summary of the Invention

[0010] To overcome the inaccuracy of existing multimodal emotion recognition technologies, this invention provides a multimodal emotion recognition instrument based on a contrastive learning gated network. Through dual optimization of hardware and algorithms, the performance of emotion recognition is comprehensively improved. First, this invention combines electroencephalogram (EEG) and eye signals in its hardware acquisition design to ensure efficient and accurate data acquisition. Second, a multimodal contrastive learning gated network (MCGNet) is designed to improve the accuracy of emotion recognition.

[0011] This invention relates to a multimodal emotion recognition brain-computer interface instrument based on a contrastive learning gating network. The instrument comprises an EEG data acquisition device, an eye-tracking data acquisition device, and a multimodal data processing device. The multimodal data processing device includes an MCU, a neural network accelerator, a data bus, a data storage device, and an output interface. The EEG data acquisition device acquires EEG data, and the eye-tracking data acquisition device acquires eye-tracking data. The acquired EEG and eye-tracking data are stored in the data storage device under the control of the MCU. Furthermore, the MCU controls the neural network accelerator to retrieve the EEG and eye-tracking data from the data storage device for training the multimodal contrastive learning gating network.

[0012] Furthermore, the EEG data acquisition device consists of a power supply, a main control chip, and an analog-to-digital converter.

[0013] Furthermore, the eye-tracking data acquisition device consists of binocular infrared cameras, which can capture clear eye images in low-light or no-light environments, avoiding interference from visible light.

[0014] To verify the practical effect of the multimodal acquisition helmet in emotion recognition, the multimodal emotion recognition method MCGNet, which combines EEG signals and eye movement signals, was executed in the multimodal emotion recognition brain-computer interface instrument proposed in this invention. Specifically, the MCU controls the neural network accelerator to retrieve EEG data and eye movement data from the data storage for training the multimodal contrastive learning gating network. The training includes the following three modules: a single-modal feature encoder pre-training module, a multimodal contrastive learning training module, and a gating structure training module.

[0015] The unimodal feature encoder pre-training module inputs EEG primary features and eye-tracking primary features into the EEG feature encoder and eye-tracking feature encoder, respectively, and then inputs the encoding results into the classifier to predict the unimodal emotion category.

[0016] The multimodal contrastive learning training module inputs the primary EEG features and primary eye-tracking features into the feature encoder of the single-modal feature encoder pre-training module to obtain multimodal features EEG and Eye. Then, the EEG and Eye signals are decomposed using EEG similarity mappers and EEG dissimilarity mappers, and eye similarity mappers and eye dissimilarity mappers, respectively. The decomposition results are then fused and subjected to intra-sample contrastive learning and inter-sample contrastive learning.

[0017] The gating structure training module fuses primary EEG features and primary eye-tracking features and inputs them into the gating network. The gating network generates a two-dimensional vector as output to select and activate different expert network branches to complete the emotion classification task.

[0018] This invention utilizes contrastive representation learning and contrastive feature decomposition to enhance the representation of multimodal information. Furthermore, to address the issue that the fusion of multimodal information may sometimes introduce noise interference, leading to lower accuracy compared to single-modal methods, a gating structure is introduced. This structure selects either a single-modal or multimodal method and its corresponding expert network based on sample features to improve the accuracy of emotion recognition. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is a diagram of a multimodal emotion recognition brain-computer interface instrument system based on a contrastive learning gating network.

[0021] Figure 2 shows the pre-training module diagram of a single-modal feature encoder.

[0022] Figure 3 shows the multimodal contrastive learning training module.

[0023] Figure 4 shows the training module diagram of the gated structure.

[0024] Figure 5 shows the experimental results of this invention on the TJU-Emotion dataset.

[0025] Figure 6 shows the experimental results of this invention on the public dataset SEED.

[0026] Figure 7 shows the performance results of the present invention in the ablation experiment.

[0027] Figure 8 is a confusion diagram of the EEG mode and eye mode of the present invention.

[0028] Figure 9 shows the experimental results of the single-modal EEG, eye, and two multimodal fusion methods of the present invention.

[0029] Figure 10 shows the two-dimensional projection results of the four decomposition features of the test sample of the present invention. Detailed Implementation

[0030] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0031] As shown in Figure 1, the instrument consists of an EEG data acquisition device, an Eye data acquisition device, and a multimodal data processing device. The multimodal data processing device includes an MCU, a neural network accelerator, a data bus, a data memory, and an output interface. The EEG data acquisition device acquires EEG data, and the eye-tracking data acquisition device acquires eye-tracking data. After acquisition, the EEG and eye-tracking data are stored in the data memory under the control of the MCU. Furthermore, the MCU controls the neural network accelerator to retrieve the EEG and eye-tracking data from the data memory for training the multimodal contrastive learning gated network.

[0032] Furthermore, the EEG data acquisition device consists of a power supply, a main control chip, and an analog-to-digital converter.

[0033] Furthermore, the eye-tracking data acquisition device consists of binocular infrared cameras, which can capture clear eye images in low-light or no-light environments, avoiding interference from visible light.

[0034] To verify the practical effect of the multimodal acquisition helmet in emotion recognition, the multimodal emotion recognition method MCGNet, which combines EEG signals and eye movement signals, was executed in the multimodal emotion recognition brain-computer interface instrument proposed in this invention. Specifically, the MCU controls the neural network accelerator to retrieve EEG data and eye movement data from the data storage for training the multimodal contrastive learning gating network. The training includes the following three modules: a single-modal feature encoder pre-training module, a multimodal contrastive learning training module, and a gating structure training module.

[0035] The unimodal feature encoder pre-training module inputs EEG primary features and eye-tracking primary features into the EEG feature encoder and eye-tracking feature encoder, respectively, and then inputs the encoding results into the classifier to predict the unimodal emotion category.

[0036] The multimodal contrastive learning training module inputs the primary EEG features and primary eye-tracking features into the feature encoder of the single-modal feature encoder pre-training module to obtain multimodal features EEG and Eye. Then, the EEG and Eye signals are decomposed using EEG similarity mappers and EEG dissimilarity mappers, and eye similarity mappers and eye dissimilarity mappers, respectively. The decomposition results are then fused and subjected to intra-sample contrastive learning and inter-sample contrastive learning.

[0037] The gating structure training module fuses primary EEG features and primary eye-tracking features and inputs them into the gating network. The gating network generates a two-dimensional vector as output to select and activate different expert network branches to complete the emotion classification task.

[0038] The single-modal feature encoder pre-training module constructed in this invention is shown in Figure 2. This pre-training module includes an EEG modality pre-training module and an eye-tracking modality pre-training module. The EEG modality pre-training module consists of an EEG primary feature extraction module, an EEG feature encoder DSSTNet module, and a classifier module connected in sequence; the eye-tracking modality pre-training module consists of an eye-tracking primary feature extraction module, an eye-tracking feature encoder module, and a classifier module connected in sequence.

[0039] Specifically, in the EEG modality pre-training module, during the pre-training phase, given a sample, the EEG primary feature extraction module extracts the EEG features of the sample data and inputs these features into the EEG feature encoder DSSTNet module. The output of the EEG feature encoder DSSTNet module is then input into the classifier module to predict the category of the single-modality sentiment. The EEG feature encoder DSSTNet module includes a spatial feature extractor, a frequency band attention module, and a temporal feature extractor connected in sequence. The spatial feature extractor extracts spatial features of the EEG signal, the frequency band attention module extracts frequency domain features, and the temporal feature extractor extracts temporal features.

[0040] Specifically, the raw EEG signal is preprocessed and then input into a spatial feature extractor, which consists of a graph converter and an adaptive graph convolutional layer. In the graph converter, a 64×1 kernel with a stride of 1×1 is first used to convolve along the electrode channel dimension to obtain the spatial features of the EEG signal. The purpose of setting a larger kernel size is to obtain a wider receptive field, thereby fully extracting the spatial features. This layer uses a padding operation to maintain the input and output feature dimensions unchanged. The obtained spatial features are then represented as graph-structured data.

[0041] Since the distribution of EEG channels is irregular and discrete, an adaptive graph convolutional layer is used to extract the spatial features of the EEG signal channel dimensions. In graph theory, a graph can be defined as G = (v, e), where v and e represent the nodes and edges in the graph, respectively. The connection relationships between N distinct nodes in the graph can be represented by the adjacency matrix A ∈ R. N×N Let g be the filtering function. θ =diag(θ), for a given spatial signal x∈R N The graph convolution operation can be defined as: g θ * G x=Udiag(θ)U T x (1)

[0042] Where θ is a variable parameter, * G This represents a graph convolution operation, where U∈R N×N Let L∈R be the Laplace matrix in graph G. N×N The orthogonal matrix formed by the eigenvectors of g. In actual calculations, g θ It is difficult to obtain directly, so a K-order Chebyshev polynomial is used to approximate g. θ This adaptive graph convolutional layer can automatically update the adjacency matrix during backpropagation without requiring manual construction. First, the adjacency matrix is ​​randomly initialized. Then, when backpropagation is applied, the optimal adjacency matrix A is adaptively learned. * When calculating the loss function for A * The partial derivatives of A * Update according to the following rules:

[0043] Where ρ is a parameter used to control the update speed of the adjacency matrix, and in this invention, ρ is set to 0.001.

[0044] This spatial feature extractor solves the problem of effectively extracting spatial features of EEG signals and avoids interfering with the adjacency matrix, enabling adaptive updates of the adjacency matrix for specific subjects.

[0045] Frequency Band Attention Module: This module, located between the spatial and temporal feature extractors, comprises three parts: semi-global pooling, local cross-band interaction, and adaptive weighting. First, semi-global pooling is used for global average pooling in the spatial dimension while preserving temporal information. Next, frequency domain features are extracted using 7×1 convolutional kernels, with imputation operations maintaining the temporal features and electrode channel dimensions of the feature map. Compared to fully connected layers, convolutional layers reduce the number of parameters and promote local interaction. Finally, the selected features are adaptively weighted, and the frequency domain information is probabilistically converted using a softmax function. The softmax output is then multiplied by the input features using a Hadamard product.

[0046] Temporal Feature Extractor: The temporal feature extraction module is constructed by stacking multiple one-dimensional convolutional layers. The first layer consists of D convolutional kernels of size 1×2 and stride 1×1, convolving along the temporal dimension. The second convolutional layer consists of D kernels of size 32×1 and stride 1×1, aiming to reduce the spatial dimension of the features to 1 for better extraction of temporal information. Subsequently, batch normalization is used to alleviate the gradient vanishing problem and accelerate the model training process. Next, the ELU activation function is used to non-linearize the data. The third layer performs average pooling along the temporal dimension, with a kernel size of 1×1 and stride of 1×2. This pooling layer smooths temporal features while preventing overfitting and reducing computational complexity. The final layer further extracts temporal features using D kernels of size 1×2 and stride 1×1. In this invention, the hyperparameter D is set to 40.

[0047] Finally, the EEG feature map is passed through a classifier module consisting of two fully connected layers, and its output is used to calculate the predicted probability of each of the three emotions using the softmax function.

[0048] Specifically, in the eye-tracking modality pre-training module, during the pre-training phase, given a sample, the eye-tracking primary feature extraction module is used to extract the eye features of the sample data and input the features into the eye-tracking feature encoder module. The output of the eye-tracking feature encoder module is input into the classifier module to predict the category of unimodal emotion.

[0049] After preprocessing the original eye signal, primary feature differential entropy features are extracted in the frequency bands [0,0.2]Hz, [0.2,0.4]Hz, [0.4,0.6]Hz, and [0.6,1]Hz, respectively. The features extracted for the left and right eyes include the mean, standard deviation, and differential entropy features of the four frequency bands. These are used as inputs to the eye-tracking feature encoder module, denoted as F. Eye ∈R T×K The sliding window length is T, and the feature dimension is K.

[0050] The eye-tracking feature encoder module adopts the encoder structure design in the transformer architecture, including an input embedding layer, multi-head attention, layer normalization, and a feedforward layer. The advantage of the transformer is that it can simultaneously handle long-range dependencies and capture global contextual information, effectively learning the intrinsic structure and important features in the input sequence through a self-attention mechanism.

[0051] The multimodal contrastive learning training module constructed in this invention is shown in Figure 3. In this module, the primary EEG features and primary eye-tracking features are input into the encoder of the single-modal feature encoder pre-training module to obtain the multimodal features EEG and Eye. Then, the multimodal features EEG are decomposed into similar features EEG using an EEG similarity mapper and an EEG dissimilarity mapper. s and dissimilar features EEG d The multimodal feature Eye is decomposed into similar features Eye using an eye similarity mapper and an eye dissimilarity mapper. s and dissimilar features Eye d Then, the four decomposed features are fused using different splicing methods, and intra-sample contrastive learning and inter-sample contrastive learning are performed. This invention provides two splicing methods. The first method splices the features according to the feature dimension and outputs the splicing result to the MLP classifier to obtain the sentiment prediction label (i.e., y1 in Figure 3). The second method splices the features according to the batch dimension and inputs them into the weight-sharing MLP classifier to obtain four single-modal prediction labels represented by y2.

[0052] In the first approach, the dataset is denoted as M. For a given sample i∈M, the multimodal sentiment classification loss L is calculated using the cross-entropy (CE) loss. pred The calculation method is as follows:

[0053] Where [;] indicates concatenation according to feature dimensions, y i The true label for the emotion category.

[0054] In the second approach, the unimodal emotion classification loss L uni for:

[0055] Where [,] indicates concatenation according to batch dimension, y i This represents the true label predicted by a single modality, and the prediction is performed separately for each of the four decomposed features.

[0056] In intra-sample contrastive learning and inter-sample contrastive learning, the four decomposed features are used to construct positive and negative sample pairs, as shown in Figure 3. First, for a given sample pair (i,j) in dataset M, features are extracted by the feature encoder and the sample [EEG] is calculated. i Eye i ] and [EEG j Eye j Cosine similarity score between [ ]: C i,j =sim([EEG i Eye i [EEG] j Eye j (7)

[0057] Next, select similar and dissimilar samples for sample i. This will be compared with the label y. i Identical samples are sorted from low to high according to the cosine similarity score calculated in the previous step to construct a candidate similar sample set. At the same time, label non-y i The samples are divided into candidate dissimilar sample sets. From the set Two samples with high cosine similarity scores are randomly selected from the samples and paired with sample i to form a pair of samples, denoted as Neighbor. i From set Two samples with lower cosine similarity scores are randomly selected from the samples, denoted as . Two samples with high cosine similarity scores are randomly selected and denoted as . and Together with sample i, they form a negative pair between samples, Outlier i .

[0058] First, construct the in-sample positive alignment. and negative pair

[0059] where j∈Neighbor i ∪Outlier i Neighbor i and Outlier i Let i represent similar and dissimilar samples to sample i, respectively.

[0060] Then, construct the positive alignment between samples. and negative pair The structure is as follows:

[0061] where j∈Neighbor i ,k∈Outlieri

[0062] The loss functions for intra-sample and inter-sample contrastive learning are represented by the joint contrastive loss, which includes two aspects: comparison between similar and dissimilar samples between samples, and comparison between similar and dissimilar features within a sample. Given sample i, the contrastive learning loss L... c for:

[0063] The NT-Xent contrastive learning loss framework combines intra-sample and inter-sample contrastive learning for modal feature decomposition and modal representation learning. The loss for sample i is expressed as:

[0064] Where (a,p) and (a,n) represent a pair of decomposed feature vectors within a sample, for example... Or a pair of decomposed feature vectors between samples, for example P i For a positive set, it is represented as Including in-sample positive and between samples N i For a set of negative pairs, denoted as Including negative pairs within the sample Negative pairs between samples For P i In the positive pair, (a,n) is N i The negative pair in the middle.

[0065] The total loss function for this module is: L = L pred +λ uni L uni +λ c L c (14)

[0066] Where L pred For multimodal emotion classification loss, L uni For unimodal emotion classification loss, L c To compare the learning loss. λ uni and λ c This determines the degree to which each task contributes to the updating of model parameters during training.

[0067] The gated structure training module constructed in this invention is shown in Figure 4. This module concatenates the primary EEG feature x1 and the primary eye feature x2 and inputs them into a gated network G(x) composed of an MLP. This gated network generates a two-dimensional vector g as its output to select and activate different network branches to complete the emotion classification task. A set of expert networks was selected based on a hybrid expert model. Each expert specializes in studying a subset of all modalities. In a specific task, there can be a maximum of three expert networks for two modalities, denoted as E1(x1), E2(x1,x2), and E3(x2). While combining EEG with eye can provide useful cues, in practical applications, sometimes single-modal EEG is better at recognizing emotions. This invention uses the single-modal EEG emotion classification network model pre-trained by the single-modal feature encoder pre-training module as expert network one E1(x1), and the multimodal contrastive learning network model trained in the multimodal contrastive learning training module as expert network two E2(x1,x2). The final output predicted label is represented by y3 as: y3=g1E1(x1)+g2E2(x1,x2) (15)

[0068] This invention selects two expert network models with different complexities; generally, expert networks with higher model complexity have stronger representational capabilities. If the network is trained solely by minimizing the loss of the emotion classification task, the gating network will always choose the branch with higher model complexity. Therefore, an additional loss function β[g1C(E1)+g2C(E2)] is introduced when designing the loss function. Here, g1 and g2 represent the decision vectors output by the gating structure, C(E) represents the cost of executing the expert network (e.g., MAdds), and β is a hyperparameter used to adjust the relative importance between the two loss terms.

[0069] This invention jointly optimizes expert networks and gating networks in an end-to-end manner. For the logic vector output by the gating structure MLP, we use a softmax function with a temperature parameter to adjust the sensitivity of the decision boundary. Furthermore, this invention involves selecting different expert networks for subsequent training, allowing each network to focus on specific modalities of the data.

[0070] This invention uses a self-built dataset (TJU-Emotion) and the publicly available SEED dataset to evaluate the performance of the proposed method. Experimental results show that the proposed method outperforms a series of state-of-the-art comparative methods in emotion recognition tasks.

[0071] First, this invention constructs a three-class multimodal emotion dataset, named the TJU-Emotion dataset, which includes EEG and eye multimodal data. The experiment was conducted at Tianjin University, with 10 participants invited to participate. The EEG modality used a 32-electrode experiment, and the eye data collected included pupil diameter data from the participants' left and right eyes. The experiment used 15 film clips (which could evoke positive, neutral, and negative emotions) as stimuli, and comprised 15 trials.

[0072] The preprocessing steps for the EEG portion include filtering, reducing the sampling rate, rereference transformation, segmentation, and artifact removal. The preprocessed EEG data is then divided into samples with a sliding window length of T = 15 s and a step size ΔT = 1 s. Preprocessing of the eye portion utilizes principal component analysis to remove the first principal component, illumination reflection.

[0073] The publicly available SEED dataset contains EEG data from 15 participants while watching 15 Chinese film clips with three different emotional states: negative, positive, and neutral. The EEG electrode distribution follows the international 10-20 electrode localization scheme, with a total of 62 electrode channels.

[0074] Four types of quantitative and qualitative effect analyses were conducted on this invention: comparative experiments with state-of-the-art methods, the impact of each structure on model performance, multimodal information complementarity analysis, and comparative learning visualization analysis.

[0075] This invention conducts a participant-dependent experiment and calculates the mean classification accuracy, standard deviation, and F1-score for all participants. The experiment sets the learning rate of the Adam optimizer to 1e-5 and 1e-3 on two datasets, respectively, with parameter λ. uni , λ c Both β and β are set to 0.1.

[0076] This invention selected six baseline models to compare the performance of the present invention, including the simple decision-level fusion methods MAX and SUM, the fuzzy integral-based model fusion method Fuzzy, the DCCA-AM method with attention mechanism and deep canonical correlation analysis, the BAT method that uses bidirectional adapters to transfer complementary information between different modalities, and the CARAT method that uses contrastive learning and feature reconstruction.

[0077] First, we performed a unimodal emotion classification experiment on 10 subjects in the TJU-Emotion dataset using a unimodal feature encoder, and compared the performance of our invention with the six baseline models from the previous section. The experimental results are shown in Figure 5.

[0078] From the first two columns of the graph, we can see that the performance of unimodal emotion classification depends on the data type used, with the EEG modality performing best among unimodal models. Our invention, by incorporating multimodal information, significantly improves the performance of emotion classification. We then conducted comparative experiments on this dataset using six multimodal fusion methods and calculated the t-test significance p-values ​​of several models relative to our invention. The classification performance of our invention shows a significant difference compared to other methods. Experimental results demonstrate that our invention not only has a significant advantage in classification accuracy but also exhibits considerable superiority in model stability. Furthermore, our invention's average F1-score is improved compared to other methods.

[0079] We also validated the performance of our proposed method on the widely used public dataset SEED. The experimental results are shown in Figure 6, demonstrating that our proposed method is competitive on public datasets as well.

[0080] To investigate the impact of various structures on the performance of this invention, we set up several ablation experiments, the results of which are shown in Figure 7. These include the following ablation experiment models: a model without contrastive learning, a model using only intra-sample contrastive learning, a model using only inter-sample contrastive learning, and a model without gating structures.

[0081] For the TJU-Emotion dataset, as shown in Figure 7, the performance of the four ablation experiment models is inferior to that of the model in this invention. Compared to models that do not use contrastive learning methods, models using intra-sample or inter-sample contrastive learning methods show significant performance improvements. Furthermore, the model in this invention has a more flexible structure, allowing for dynamic selection of different modalities and expert networks based on the characteristics and needs of each sample. Similar conclusions were obtained for the SEED dataset.

[0082] Figure 8 shows the confusion diagram between the EEG modality and the eye modality. In the eye modality, the recognition accuracy for positive states is higher than in the EEG modality. The EEG modality is better at recognizing neutral and negative states. Therefore, it is hoped that these two modalities can complement each other to improve the performance in recognizing each emotional state.

[0083] Figure 9 shows the confusion matrices for single-modal EEG, eye, and two multimodal fusion methods: a model without gating structures and the model of this invention, demonstrating the advantages and disadvantages of different modalities and different modal fusion methods. We found that the EEG and eye modalities have important complementary features, and the multimodal fusion method significantly improves classification performance. Overall, effectively combining multiple modalities can leverage the characteristics and complementarity of each modality to improve the ability to identify three types of emotions.

[0084] A participant was randomly selected in TJU-Emotion, and the two-dimensional projections of the four decomposed features of all their test samples are shown in Figure 10. Figure 10(a) shows the sample without contrastive learning, while Figure 10(b) shows the sample with contrastive learning. As can be seen from the figures, after using contrastive learning, the similarity features (EEG)... s and Eye s The features become increasingly similar, while dissimilar features (EEG) become increasingly similar. d and Eye d The features they are moving further and further away from their corresponding similar features. This demonstrates the effectiveness of using contrastive learning methods to learn the consistency and inconsistency between modalities.

[0085] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0086] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0087] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0088] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0089] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0092] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0093] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multimodal emotion recognition instrument based on a contrastive learning gating network, characterized in that, The instrument consists of an EEG data acquisition device for acquiring EEG data, an eye movement data acquisition device for acquiring eye movement data, and a multimodal data processing device. The multimodal data processing device includes an MCU, a neural network accelerator, a data bus, a data memory, and an output interface. The MCU controls the neural network accelerator to retrieve EEG data and eye-tracking data from the data memory for training the multimodal contrastive learning gating network, specifically including: The single-modal feature encoder pre-training module inputs the primary EEG features and primary eye-tracking features into the EEG feature encoder and the eye-tracking feature encoder, respectively, and then inputs the encoding results into the classifier to predict the single-modal emotion category. The multimodal contrastive learning training module inputs the primary EEG features and primary eye movement features into the feature encoder of the single-modal feature encoder pre-training module to obtain multimodal features EEG and Eye. Then, the EEG and eye movement signals are decomposed using EEG similarity mappers and EEG dissimilarity mappers, and eye similarity mappers and eye dissimilarity mappers, respectively. The decomposition results are then fused and subjected to intra-sample contrastive learning and inter-sample contrastive learning. The gating structure training module fuses primary EEG features and primary eye-tracking features and inputs them into the gating network. The gating network generates a two-dimensional vector as output to select and activate different expert network branches to complete the emotion classification task.

2. The multimodal emotion recognition instrument based on contrastive learning gating network as described in claim 1, wherein the single-modal feature encoder pre-training module consists of an EEG modality pre-training module and an eye-tracking modality pre-training module; the EEG modality pre-training module consists of an EEG primary feature extraction module, an EEG feature encoder DSSTNet module, and a classifier module connected in sequence; the eye-tracking modality pre-training module consists of an eye-tracking primary feature extraction module, an eye-tracking feature encoder module, and a classifier module connected in sequence. The EEG feature encoder DSSTNet module includes a spatial feature extractor, a frequency band attention module, and a temporal feature extractor connected in sequence.

3. The multimodal emotion recognition instrument based on a contrastive learning gating network as described in claim 1, wherein the multimodal contrastive learning training module specifically includes: Multimodal features (EEG) are decomposed into similar features (EEG) using EEG similarity mappers and EEG dissimilarity mappers. s and dissimilar features EEG d ; The multimodal feature Eye is decomposed into similar features Eye using an eye similarity mapper and an eye dissimilarity mapper. s and dissimilar features Eye d ; The four decomposed features are fused along the feature dimension or batch dimension, and intra-sample comparison learning and inter-sample comparison learning are performed.

4. The multimodal emotion recognition instrument based on contrastive learning gating network as described in claim 1, wherein in the gating structure training module, the single-modal EEG emotion classification network model pre-trained by the single-modal feature encoder pre-training module is used as expert network one E1(x1), and the multimodal contrastive learning network model trained in the multimodal contrastive learning training module is used as expert network two E2(x1,x2). The final output predicted label is represented by y3 as: y3=g1E1(x1)+g2E2(x1,x2) (1) Where g1 and g2 represent the decision vectors output by the gating structure.

5. The multimodal emotion recognition instrument based on contrastive learning gating network as described in claim 2, wherein the spatial feature extractor is used to extract spatial features of EEG signals, the frequency band attention module is used to extract frequency domain features, and the time domain feature extractor is used to extract time domain features.

6. The multimodal emotion recognition instrument based on contrastive learning gating network as described in claim 2, wherein the eye-tracking feature encoder module adopts the encoder structure design in the transformer architecture, including an input embedding layer, multi-head attention, layer normalization, and feedforward layer.

7. The multimodal emotion recognition instrument based on a contrastive learning gating network as described in claim 3, when fusing according to feature dimensions, denotes the dataset as M, and for a given sample i∈M, calculates the multimodal emotion classification loss L using cross-entropy loss. pred The calculation method is as follows: Where [;] indicates concatenation according to feature dimensions, y i The true labels are for emotion categories; MLP stands for Multilayer Perceptron.

8. The multimodal emotion recognition instrument based on a contrastive learning gating network as described in claim 3, when fusion is performed according to the batch dimension, the single-modal emotion classification loss L uni for: Where [,] indicates concatenation according to batch dimension, y i This represents the true label predicted by a single modality; MLP stands for Multilayer Perceptron.

9. The multimodal emotion recognition instrument based on a contrastive learning gating network as described in claim 3, wherein in the intra-sample contrastive learning and inter-sample contrastive learning, the four decomposed features are used to construct positive and negative sample pairs, specifically including: For a given sample pair (i,j) in dataset M, features are extracted by the feature encoder and the sample [EEG] is calculated. i Eye i ] and [EEG j Eye j Cosine similarity score between ] C i,j =sim([EEG i ;Eye i ],[EEG j ;Eye j ]) (6) Select similar and dissimilar samples for sample i; compare with label y i The same samples are sorted from low to high according to the cosine similarity score calculated by equation (6) to construct a candidate similar sample set. At the same time, label non-y i The samples are divided into candidate dissimilar sample sets. From the set Two samples with high cosine similarity scores are randomly selected from the samples and paired with sample i to form a pair of samples, denoted as Neighbor. i From set Two samples with lower cosine similarity scores are randomly selected from the samples, denoted as . Two samples with high cosine similarity scores are randomly selected and denoted as . and Together with sample i, they form a negative pair between samples, Outlier i ; First, construct the in-sample positive alignment. and negative pair where j∈Neighbor i UOutlier i Neighbor i and Outlier i Let i represent similar and dissimilar samples to sample i, respectively. Constructing positive pairs between samples and negative pair The structure is as follows: where j∈Neighbor i ,k∈Outlier i .

10. The multimodal emotion recognition instrument based on contrastive learning gating network as described in claim 1, wherein the EEG data acquisition device consists of a power supply, a main control chip, and an analog-to-digital converter; and the eye-tracking data acquisition device consists of a binocular infrared camera.

Images