1466 results about "Emotion identification" patented technology

The invention relates to the field of data processing, in particular to an emotion recognition method based on a large model and a related device. The method comprises the following steps: firstly, acquiring to-be-processed multi-modal data; the multi-modal data at least comprises text information, audio information and image information input by a user; the image information at least comprises a facial expression image and / or a limb action image. And inputting the multi-modal data into an emotion recognition model for emotion state analysis to obtain a predicted emotion state corresponding to the user. And finally, performing confidence evaluation based on the predicted emotional state, and generating a user emotion recognition result based on the predicted emotional state and a corresponding evaluation result. According to the method, richer emotion details are captured from the multi-modal data through the emotion recognition model, cross-modal emotion intelligent recognition is realized, the accuracy of emotion recognition is further improved by combining confidence evaluation, the emotion recognition efficiency is improved, and the application range of emotion recognition is expanded.

The invention relates to the field of natural language understanding in an artificial intelligence technology man-machine dialogue system, in particular to a psychological accompanying dialogue method based on multi-agent collaboration and a storage medium. The psychological accompanying dialogue with individuation, high naturalness and accurate emotion recognition is provided, the user experience is effectively improved, and the psychological support requirement of the user is met. According to the main scheme, the method comprises the steps of performing multi-mode emotion evaluation on user input to obtain user emotion state description; retrieving related dialogue states and external knowledge from a shared memory module according to user questions and user emotional state description; according to the retrieved dialogue state and external knowledge, the strategy planning module generates a reply strategy to obtain a strategy group; the dialogue content generation module calls a corresponding vertical domain agent according to the generated reply strategy to generate specific reply content; and storing the user questions and model replies in the current round into the short-term memory of the shared memory module, and updating the long-term memory. The method is used for AI dialogues.

The invention relates to the technical field of online teaching management, and discloses an online teaching optimization method and system based on emotion recognition, and the method comprises the steps: collecting the real-time emotion data of a learner through a multi-modal sensor, carrying out the fusion of a graph convolutional neural network to generate a feature vector, and optimizing a teaching strategy parameter through a meta-reinforcement learning model, the dynamic course generation model combines an improved genetic algorithm and a knowledge graph to optimize a teaching content sequence, and the hierarchical teaching control model realizes knowledge path planning, interactive adjustment and learning state evaluation, and generates a teaching control instruction. According to the method, personalized online teaching optimization is realized, the state of a learner can be accurately grasped, the teaching interaction mode is optimized, the course content is dynamically adjusted, the learning effect is comprehensively and accurately evaluated, the problems that traditional online teaching lacks personalization, the state of the learner is difficult to grasp and the like are effectively solved, the online teaching quality and the learning experience are improved, and the learning experience is improved. And the online education development is promoted.

The invention relates to the technical field of emotion recognition, and discloses an electroencephalogram and synchronous physiological signal emotion recognition method based on cross-modal comparative learning and multi-scale characterization, in a pre-training stage, features are extracted from physiological signals of different modalities through encoders corresponding to the modalities, and modal characterization related to emotion stimulation is learned; the modal representations are mapped to a potential space through a mapping layer to calculate the similarity, then the contrast loss of each modal is calculated, and contrast learning in the modal is carried out; performing cross-modal comparative learning on the representation based on each modal extracted by an encoder, and aligning the representation of different modals to a shared potential space; according to the invention, the encoder is pre-trained at different time scales by using short-term and long-term emotion changes in each mode; in the fine tuning stage, emotion recognition training is carried out by using a pre-trained encoder, an initialized modal fusion module and a classifier, and the model weight is optimized based on the emotion label of the physiological signal.

The invention provides a system platform for psychological assessment and emotion feedback. The system platform comprises a multi-source data acquisition module, an emotion data analysis module, a psychological state assessment module, an intervention scheme generation module, an emotion change feedback module, a scheme optimization module and a data security protection module. The emotion recognition accuracy is improved by synchronously acquiring physiological-behavior-environment data and combining with a dynamic feature fusion method, a personalized intervention strategy is generated based on a reinforcement learning algorithm, real-time feedback is realized, and a data security protection system is constructed by adopting federal learning and homomorphic encryption technologies, so that the data security is ensured.

Aiming at the problems of insufficient language understanding depth, weak personalized dialogue generation ability, lack of continuous learning and long-term user state modeling and the like in the current emotion recognition and psychological intervention technology, the invention provides a user emotion recognition and psychological intervention method combined with a large language model (LLM). According to the method, the potential emotional state is identified by analyzing free text information input by a user by utilizing the powerful capabilities of a large language model in the aspects of natural language understanding, emotional modeling and text generation; constructing a multi-round dialogue context, and reasoning a psychological change trend of the user; in combination with a psychological knowledge base, personalized and mild psychological intervention dialogue content with a dredging effect is generated. The system supports recognition and classification of various emotional states such as depression, anxiety and alonity, is suitable for various interaction scenes (such as APPs, webpages and social robots), and can greatly improve the precision of emotion recognition and the timeliness and effectiveness of psychological intervention. The emotion recognition and psychological intervention method based on the large language model provides solid technical support for constructing an intelligent, continuous and personalized psychological health management system, and has wide application prospects and profound social significance.

The invention relates to the field of deep learning and emotion recognition, in particular to an EEG emotion recognition system and method based on multi-scale space-time diagram convolution and comparative learning, and the method comprises the steps: carrying out the preprocessing of an EEG signal; constructing positive and negative sample pairs based on the emotion categories and the identities of the subjects; extracting hidden space representation of the electroencephalogram signal by adopting a stack auto-encoder, and dividing the electroencephalogram signal into a plurality of functional brain regions to obtain a brain source signal after dimension reduction; constructing an undirected graph structure of the brain source signal; constructing a multi-scale space-time diagram convolutional network as a feature extraction network, inputting an undirected graph structure, dynamically learning and updating an adjacent matrix, expressing features of brain source signals in the undirected graph structure, and extracting emotion related features; based on a multi-scale space-time diagram convolution and comparative learning framework, an EEG emotion recognition model is built and trained to eliminate individual differences. According to the method, the multi-dimensional information of the EEG signals is fully utilized, and the accuracy, robustness and generalization performance of emotion recognition are remarkably improved.

The invention provides a dynamic calibration method and system for a vehicle-mounted emotion recognition system, and the method comprises the steps: S1, obtaining multi-source data which comprises a facial image, a voice signal and a physiological signal; the obtained multi-source data are preprocessed, and preprocessed multi-source data are obtained; s2, performing feature extraction based on the preprocessed facial image, the voice signal and the physiological signal to obtain a facial expression feature vector, an audio feature vector and a physiological state feature vector; s3, evaluating the current environment credibility based on an environment credibility evaluation function; s4, dynamically distributing the weight of the multi-source data according to the credibility of the current environment and the real-time scene; and S5, constructing a multi-modal fusion vector based on the dynamically distributed weight of the multi-source data, the facial expression feature vector, the audio feature vector and the physiological state feature vector, and performing emotion recognition by using the constructed emotion recognition model based on the multi-modal fusion vector.

The invention discloses a multi-modal signal fusion emotion recognition method based on an attention mechanism. The method comprises the following steps: firstly, preprocessing physiological signal data; constructing a channel attention module for the multi-channel electroencephalogram data; respectively extracting EEG and other physiological signal features by using the EEGNet; a shared-private encoder is introduced to decouple shared features among the multiple modes and private features of each mode; and finally, providing a cross-modal cross attention fusion mechanism to realize multi-modal feature interaction and effective fusion. According to the method, multi-modal signals are combined, the limitation of a single mode is overcome, and a sharing-private feature separation mechanism and a cross-modal cross fusion mechanism are introduced, so that modal collaborative modeling and effective integration are realized, and the accuracy and generalization ability of emotion recognition are improved.

A dog training method for intelligently correcting pet behaviors based on multi-modality comprises the steps that multi-modality perception data of a pet is obtained, the multi-modality perception data comprises acoustic signals, motion signals and track data, an acoustic feature tensor, a motion feature tensor and a position feature tensor are constructed, and a unified coupling tensor is formed; inputting the coupling tensor into a pre-trained behavior recognition model, and outputting a behavior tag; the emotion recognition module outputs an emotion label, and updates a character label in a sliding window mode; inputting the behavior label, the emotion label and the character label into an intervention strategy generation model and outputting an adaptive intervention strategy, wherein the intervention strategy comprises a pacifying type strategy and a stimulation type strategy; according to a historical feedback effect record, adjusting a feedback intensity parameter of the intervention strategy, calculating a feedback convergence index and an emotion recovery index, and dynamically correcting a feedback level and a delay strategy of the intervention strategy.

The invention discloses an AI digital human interaction system based on emotion recognition, relates to the technical field of emotion calculation, and improves the accuracy of emotion recognition through multi-modal emotion data acquisition and processing of an emotion primary recognition module and a secondary recognition module. The system effectively distinguishes complex emotions, such as fatigue and depression, of the trainee in the virtual education task by calculating the first emotion coefficient and the second emotion coefficient; secondary emotion recognition is carried out, so that misjudgment is reduced, and the emotion state recognition accuracy is ensured; through an emotion feedback module, the system dynamically adjusts the expression, intonation and interaction frequency of the AI digital human according to a second emotion coefficient, so that the emotion feedback is more natural, and the emotion recognition sense of the student is enhanced; besides, the culture adaptation analysis module collects cross-culture emotion data based on the culture background of the trainee, calculates and evaluates a culture adaptation coefficient, and automatically adjusts an emotion recognition and feedback strategy, thereby improving the adaptability of the system in a multi-culture environment and the accuracy of emotion expression.

The invention provides a self-learning multi-modal emotion recognition method based on multi-scale cavity attention, and solves the problem of low recognition precision caused by different importance of basic action units of a face and different distances between key action units, and the problem of different modality confidence during decision-level fusion. The method comprises the following steps: preprocessing a facial expression image, inputting the facial expression image into a multi-scale cavity attention convolution module, extracting features through a parallel three-branch convolution structure, splicing the features, calibrating through an attention mechanism to obtain an enhanced feature map, and sending the enhanced feature map to a full connection layer to recognize emotion; an original electroencephalogram signal is input into a time-frequency-space three-dimensional feature extraction network, the signal is decomposed, differential entropy features are calculated and processed by a global attention module comprising a frequency spectrum attention module, a space attention module and a time attention module, time-frequency-space multi-dimensional feature representation is output, and emotions are recognized by a full connection layer; and finally, inputting the emotion recognition result of the facial expression and the electroencephalogram signal into a self-learning weight module, and obtaining a final emotion recognition result through dynamic weighted fusion.

The invention relates to the technical field of pet emotion recognition, in particular to a pet dog emotion recognition method and system based on deep learning. Comprising the steps of collecting pet dynamic data, pet physiological data and scene data to obtain a structured data set; labeling the structured data set through a cross validation labeling mechanism to obtain a labeled data set; multi-modal features are extracted based on the labeled data set, and multi-modal feature integration is carried out through a cascade SEblock array to obtain multi-modal fusion features; adversarial sample data generated by a stress scene simulator is injected in a training stage, and deep learning model training is performed based on the adversarial sample data and the multi-modal fusion features to obtain a pet emotion recognition model; and performing emotion recognition on a to-be-recognized pet through the pet emotion recognition model to obtain a pet emotion recognition result. According to the method, the data quality and the model robustness of pet emotion recognition are improved, and then the human-pet interaction quality is improved.

The invention discloses an emotion recognition method based on a space-time multi-scale attention convolutional neural network, and the method comprises the steps: collecting electroencephalogram data of a subject, and carrying out the preprocessing of the electroencephalogram data, so as to obtain electroencephalogram (EEG) data containing a space dimension and a time dimension; constructing a lightweight convolutional neural network comprising a double-flow spatial-temporal feature construction layer, a mixed attention mechanism layer, a high-order fusion layer and a classification layer; wherein the double-flow spatial-temporal feature construction layer comprises a temporal feature extraction module and a parallel spatial feature extraction module; the mixed attention mechanism layer combines a channel attention mechanism, a space attention mechanism and a self-attention mechanism; the high-order fusion layer is used for carrying out re-learning from the learned global convolution kernel to the representation of the local hemisphere convolution kernel; and recognizing the EEG data by using the trained lightweight convolutional neural network to obtain an emotion recognition result of the subject. The precision and efficiency of emotion recognition driven by electroencephalogram signals are improved by constructing a lightweight model with a small number of parameters.

The embodiment of the invention provides a vehicle-mounted emotion interaction method and device based on multi-dimensional recognition, and the method and device achieve the precise judgment of the emotion of a driver through innovatively constructing an emotion fusion recognition model and integrating the facial expression, voice emotion, driving behavior and physiological state features. And designing a scene-based self-adaptive interaction strategy, and establishing an interaction triggering threshold value for intelligent matching in combination with external environment data and a danger level. An interaction effect evaluation mechanism is introduced, an interaction strategy model is continuously optimized through an online learning module, and dynamic adjustment of personalized interaction content is achieved. According to the method, the defects of the traditional technology in the aspects of emotion recognition, interaction strategies, effect evaluation and the like are effectively overcome, and the intelligent level and the user experience of vehicle-mounted emotion interaction are remarkably improved.

The invention relates to the technical field of electroencephalogram signal processing, and discloses an electroencephalogram emotion recognition method and system based on a deep neural network. The method comprises the following steps: collecting and preprocessing a multi-channel EEG signal; constructing a graph data structure, extracting multi-domain features by taking electroencephalogram channels as nodes, and constructing a self-adaptive dynamic adjacency matrix; constructing a graph convolution long and short-term memory network, learning spatial features by GNN, and extracting time dependence by LSTM; enhancing emotion capture by using a multi-scale time-frequency feature fusion method in combination with STF and CWT; constructing global topological information of an FCN brain extraction region in combination with brain network features; and outputting alertness and other emotion indexes by means of the classification model. According to the method, graph structure learning and time sequence modeling are combined, EEG signal emotion recognition is optimized, and personalized adaptation and emotion recognition accuracy is improved.

The invention discloses an academic emotion recognition method and device based on facial expressions and contexts. According to the method, accurate emotion modeling is realized by utilizing a visual language pre-training model CLIP and combining student facial expressions in a video sequence and contextual information of a learning situation. Specifically, according to the method, firstly, collaborative modeling is carried out on facial expressions and context information in a learning scene, so that the ability of recognizing the learning state (such as focusing on learning and distracting) of students is remarkably improved; and secondly, the strong language and visual feature alignment capability of the CLIP pre-training model is utilized, so that the dependence on large-scale video training data is effectively reduced, and the generalization capability and practicability of the model are improved. An accurate and feasible solution is provided for academic emotion recognition, and the method is suitable for monitoring and analysis of academic emotions of students in an education scene.

The invention discloses a pet pacifying method, and relates to the technical field of emotion recognition, and the pet pacifying method comprises the following steps: obtaining pet behavior data, and inputting the pet behavior data into a pet emotion recognition model to recognize the current emotion type of a pet; according to the current emotion type of the pet, determining a pacifying strategy corresponding to the current emotion type; and pacifying the pet based on the pacifying strategy. According to the method, the emotion of the pet is recognized from multiple dimensions, the accuracy of pet emotion recognition is improved, whether the pet is in an anger, happy or calm state is judged, corresponding strategies are selected for pacifying according to different emotional states of the pet, and the anxiety, tension and other emotions of the pet can be relieved more effectively.

The present application relates to the field of natural language processing or other related fields, and discloses a method and apparatus for text sentiment recognition, a computer storage medium, and an electronic device. The method comprises: obtaining text to be recognized, wherein said text at least comprises an attribute word and an emotion word, and the attribute word refers to a subject that the emotion word describes; and inputting said text to a sentiment task model for processing to obtain a sentiment recognition result, wherein the sentiment task model is configured to determine an attribute category to which the attribute word belongs and an emotion category to which the emotion word belongs, determine all permutation and combination results of the attribute word, the emotion word, the emotion category, and the attribute category, and determine a target permutation and combination result among all the permutation and combination results as the sentiment recognition result. The present application solves problems in the related art that accuracy of recognition results of text sentiment recognition is low and content of the recognition results is incomplete.

Disclosed are an affective haptic regulation system and method based on multimodal fusion, including a haptic optimal parameter adjustment module, a haptic generation module, a visual-auditory generation module, a multi-physiological signal acquisition module, a multi-sensory signal acquisition module, and a multimodal fusion emotion recognition module. The system can fuse multi-physiological signal features with audio and haptic modal features by acquiring a plurality of physiological signals of a user, accurately identify a current affective state of the user in real time through advanced data processing and analysis technology, seek for a haptic parameter with the help of an optimization theory, and achieve proactive regulation of affective state of the user; and the system can overcome the limitations of traditional subjective scale methods, effectively reduce the influence of unstable physiological signals on emotion recognition results, and significantly improve the accuracy of affective detection in the affective haptic regulation system.

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a service content generation method based on emotion recognition, and the method comprises the steps: collecting voice data, extracting voiceprint features, and building a user voiceprint file; extracting Mel frequency cepstrum coefficient features and Mel energy difference coefficient features, and inputting the extracted features into an emotion classification model for emotion classification recognition to obtain an emotion state of the user; the current emotional state is compared with a historical normal emotional state, emotional fluctuation is analyzed, and whether the emotional state of the user is abnormal or not is judged; and when it is detected that the emotional state is abnormal, target service content is determined from the personalized service library in combination with the emotional state and the user voiceprint file. According to the method, the voice features are extracted and the user voiceprint file is established, so that the emotion fluctuation of the user can be accurately analyzed, whether abnormity exists or not is judged, accurate recommendation service is provided in combination with the personalized features and historical behavior data of the user, and the intelligent and personalized degree of vehicle-mounted voice interaction is improved.

The invention discloses an emotion detection system based on facial recognition, and relates to the technical field of computer vision and emotion calculation. A video stream time sequence analysis module is used for extracting a facial micro-expression image sequence of continuous frames, a time sequence feature vector containing a micro-expression intensity gradient, an illumination robustness coefficient and a facial action unit cooperation feature is generated, and a multi-mode dynamic sensing module is combined to carry out real-time analysis on an emotion classification probability, voice emotion parameters and physiological signals. And the fusion decision module performs dynamic weighted fusion on the multi-modal data based on the scene adaptive weight, and finally generates a comprehensive emotion score. Through multi-modal time sequence modeling and a dynamic weight optimization mechanism, the accuracy and environmental adaptability of emotion recognition are remarkably improved, and real-time perception and accurate decision making of customer emotion are realized in a target scene.

The embodiment of the invention provides an emotion recognition method and device based on artificial intelligence, and identity verification is realized through voiceprint features by creatively integrating multi-modal data of facial images, voices and body actions. And a multi-level attention fusion network is designed, and intelligent fusion of modal interior and cross-modal features is realized by using a feature attention layer and a modal attention layer. An emotion change time sequence model is constructed in combination with medical record data, the physiological indexes are fused for evaluation and correction, and dynamic evaluation and accurate monitoring of the emotional state are achieved. According to the method, the defects of the traditional technology in the aspects of multi-modal fusion, time sequence modeling, medical monitoring and the like are effectively overcome, and the accuracy and the practical value of emotion recognition in the medical scene are remarkably improved.

The invention discloses an exercise rehabilitation assessment method and system based on limb posture and emotion recognition, and the method comprises the steps: collecting a motion video, recording the age and gender information of a patient, constructing a self-made data set, defining a candidate region containing the rehabilitation motion of the patient, constructing a basic motion posture data set, and carrying out the recognition of the rehabilitation motion of the patient based on a posture estimation algorithm. Personalized limb skeleton key points are extracted, and emotion features are obtained through face key point detection and face action unit analysis; analyzing the motion trail of the knee joint based on the personalized limb skeleton key points, and extracting limb posture information; and carrying out feature fusion on the limb posture information and the emotional features to form a quantitative rehabilitation evaluation result. According to the invention, high-precision capture of the key points of the human skeleton is realized through computer vision and pattern recognition technologies. In addition, in combination with analysis of the emotional state of the patient, the evaluation accuracy is enhanced, and the rehabilitation training effect evaluation is more comprehensive and accurate.

The invention belongs to the technical field of brain-computer interfaces, and particularly relates to a brain-computer interface driven emotion recognition and self-adaptive regulation and control system which comprises a multichannel nerve-peripheral coupling module, an emotion intensity probability mapping module and a closed-loop nerve regulation and control current module. The multi-channel nerve-peripheral coupling module is used for realizing overall quantification of central and peripheral emotional physiology; the emotion intensity probability mapping module is used for generating continuous emotion probabilities ranging from 0 to 1 through normalization and nonlinear mapping by utilizing emotion energy and combining eye movement fatigue and electroencephalogram entropy; and the closed-loop nerve regulation and control current module is used for dynamically adjusting the transcranial electrical stimulation intensity within the safety current upper limit according to the difference value between the emotion probability and the expected target. According to the invention, the recognition precision, the response speed and the use comfort are obviously improved.

A system for real-time analysis of emotional feedback during motivational speeches, consisting of: a series of multimodal sensors, including at least one visual sensor configured to capture facial expressions of spectators, at least one directional microphone configured to capture the audio responses of the audience, and optionally one or more physiological sensors configured to capture biometric signals from spectators; an edge-based processing unit that is communicatively coupled to the arrangement of multimodal sensors, wherein the edge-based processing unit comprises the following: (a) a feature extraction module configured to extract visual features from captured facial images, acoustic features from voice responses, and physiological features from biometric signals; (b) an emotion inference machine configured to process the features using a deep learning-based emotion recognition model comprising a convolutional neural network (CNN) for classifying facial expressions, a recurrent neural network (RNN) for classifying voice emotions, and a multimodal late fusion layer configured to compute a composite emotion state vector representing the aggregated emotions of the audience; (c) a timestamp and speech alignment module configured to correlate the calculated composite emotion state vector with segmented portions of a live motivational speech based on real-time speech-to-text transcription and semantic analysis; and (d) a session-based storage unit configured to log time-indexed emotional state vectors and corresponding speech segments for post-event analysis; A speaker feedback interface comprising a portable display or a podium-mounted visualization panel, wherein the interface is configured to display visual indicators of emotional feedback in real time, the indicators being derived from the emotional state vector and including at least emotional trend graphs, threshold alerts, or engagement indices.

The invention discloses an emotion recognition method based on electroencephalogram (EEG) and expression data, which combines fusion analysis of electroencephalogram (EEG) signals and facial expression data, and provides an innovative solution for the complexity and accuracy of emotion recognition. The method comprises the following specific implementation steps: 1, finding an optimal frame in an expression video by using multi-instance learning, extracting global visual features by adopting Swin Transform, and realizing dimension reduction and balance representation of cross-modal information through a cross attention mechanism; 2, preprocessing the electroencephalogram signal by using band-pass filtering and a fast ICA algorithm, and extracting effective features by using PSD; and 3, designing a fusion module based on Transform, combining the characteristics of the two modes, and giving full play to the complementary advantage of the modes. Experimental results show that 94.85% of classification accuracy is achieved in four types of emotion classification tasks, and compared with an existing method, the accuracy is remarkably improved.

The invention discloses a digital human posture adjustment method and device combined with image recognition, and relates to the technical field of digital human control, and the method comprises the steps: starting a data collection behavior of an interaction object; accumulative emotion recognition is synchronously carried out in the process of face image data acquisition, and real-time interaction emotion is obtained; performing interaction demand analysis to obtain an interaction appeal abstract; generating an adaptive response text according to the interactive appeal abstract; fusing and analyzing the adaptive response text and the real-time interaction emotion, and outputting an emotion posture sequence and a rhythm labeling sequence; and performing synchronous dynamic synthesis playing of the adaptive response voice according to the rhythm labeling sequence and the adaptive response text. The technical problem that in the prior art, it is difficult for a digital person to accurately perceive the emotion and appeal of a visitor and adjust the adaptive posture and voice is solved, and the technical effects that the recognition accuracy of the digital person on the emotion and appeal of the visitor is improved, and the adaptability of the posture and voice of the digital person on the emotion of the visitor is enhanced are achieved.

The invention relates to the technical field of emotion recognition, in particular to an electroencephalogram emotion recognition method based on an attention mechanism and multi-feature fusion, which comprises the following steps: acquiring electroencephalogram signal data, preprocessing the electroencephalogram signal data, and obtaining an electroencephalogram emotion recognition result through a constructed TSS-ENet network model; and carrying out time domain, frequency domain and space domain feature extraction and fusion on the preprocessed electroencephalogram signal data, and finally obtaining an emotion classification result of the electroencephalogram signal data. According to the method, features are directly extracted from electroencephalogram signal data in an end-to-end manner, and electroencephalogram signal emotion classification is adaptively performed in combination with the TSS-ENet network model, so that the accuracy of emotion recognition is effectively improved.

The invention discloses a robot interactive question-answering method and system based on a large service model, and belongs to the technical field of intelligent robots, and the method comprises the steps: carrying out the feature extraction and fusion of the original input of a user, obtaining a multi-modal vector, building a recognition model, outputting an emotion recognition tag, an intention recognition tag, a user state vector and a session history, fusing the user state vector with a domain knowledge graph, outputting a structure reasoning vector and a joint condition vector, and splicing a knowledge abstract vector with a condition fusion vector to obtain the user state vector as the input of a text generator; and outputting a natural language answer and a user state vector, mapping the natural language answer into a multi-modal behavior action, and finally outputting a voice answer signal and the multi-modal behavior action. According to the method, the answer strategy of the robot is dynamically adjusted according to the emotional state of the user by combining context awareness and knowledge reasoning technologies, so that intelligent interaction of emotional resonance and professional depth coexistence is realized.