821 results about "Emotionality" patented technology

The invention discloses a psychological health analysis method and apparatus, an electronic device and a storage medium, through the application, dynamic psychological assessment is generated through multi-modal data fusion analysis, and personalized scenarized dynamic interaction content is generated based on an assessment result and an interaction history, so that the psychological health analysis efficiency is improved. Meanwhile, continuous learning and correlation analysis are carried out on the psychological state of the user by utilizing a multi-dimensional memory architecture, so that the system can understand the change of the user state and adjust an interaction strategy like a human consultant, and therefore, the technical problems of insufficient emotional distraction and credibility of the user due to the adoption of a standardized reply mode in the prior art can be solved, and the user experience is improved. The technical effects of enhancing the emotion affinity of the system, establishing a continuous and credible interaction relationship and improving the durability of the psychological intervention effect are achieved.

The invention relates to the technical field of psychological health, and discloses a psychological health multi-modal evaluation method, system and device and computer equipment, and the method comprises the steps: obtaining multi-modal data, and carrying out the feature extraction of the multi-modal data, and obtaining multi-modal features; performing emotion-oriented cross-modal attention mechanism analysis on the multi-modal features to respectively obtain a preset-dimension emotion scale vector, an emotion semantic feature, a multi-dimensional emotion feature and an uncertainty quantitative feature; fusing the preset dimension emotion scale vector, the emotion semantic feature, the multi-dimensional emotion feature and the uncertainty quantitative feature to obtain a multi-modal fusion feature; and obtaining a mental health assessment result based on the multi-modal fusion features and a preset multi-task learning framework. Through a cross-modal attention mechanism, deep semantic fusion of five modals of vision, audio, physiology, text and behavior is realized, and intelligent mapping from original multi-modal data to accurate psychological state judgment is realized through a deep learning technology.

The invention discloses a cognitive disorder nostalgic audio-video intervention therapy method, which comprises the following steps of: building a scene containing nostalgic music art, classical movies and retro original sound, playing matched audio and visual materials to a patient, and guiding the patient to perform structured intervention activities such as puzzle, theme recall or emotion sharing. Therefore, memory is stimulated, cognition is exercised, and emotion expression is promoted; according to the invention, by integrating three layouts of audios of nostalgic music art, classical movies and retro original sounds with matched visual elements, a multi-scene nostalgic sensory environment is constructed, and auditory and visual channels of a patient can be synchronously stimulated; compared with single auditory or visual stimulation, the compound sensory stimulation can more effectively activate a plurality of brain areas related to memory and emotion in the brain, so that long-term memory of deep sleep is more intensively aroused, and the strength and awakening efficiency of reminiscence stimulation are greatly improved.

The invention relates to the field of multi-modal sentiment analysis, in particular to a multi-modal sentiment analysis method based on task association perception learning. According to the scheme, the method comprises the steps that multi-modal data including text, audio and visual modal data are obtained, feature extraction is conducted on the multi-modal data, a double-branch comparison module is constructed for each kind of modal data, each double-branch comparison module comprises a fusion branch and a comparison branch, and the fusion branch is connected with the comparison branch; input of a fusion branch and a comparison branch of each double-branch comparison module is an extracted text modal feature, an extracted audio modal feature and an extracted visual modal feature; and finally, feature fusion and prediction output are carried out, so that the accuracy and robustness of multi-modal sentiment analysis are improved. The method is suitable for multi-modal sentiment analysis.

The invention relates to the technical field of intelligent voice interaction, in particular to an AI mobile phone voice interaction method oriented to automatic marketing, which comprises the following steps of: providing a comprehensive basis for personalized marketing and improving the marketing pertinence by recognizing a user emotional state and a real-time user intention in parallel and mastering the user state; an emotion state transition network is dynamically constructed and updated, key indexes are extracted, and a user emotion change rule is analyzed, so that an interaction strategy is accurately adjusted; semantic fragments are recombined by combining emotional stability and emotional infection degree indexes, adaptive dynamic marketing verbal skill is generated, the attraction and persuasion of the verbal skill are enhanced, and the marketing effect is improved; the local NPU is utilized to synthesize emotional voices and superpose features, so that natural and vivid interaction experience is provided, and recognition of a user on marketing content is enhanced; through dynamic decision-making verbal skill strategy cross-scene switching, timely adjustment is carried out according to user changes, marketing strategy stiffness is avoided, and the marketing success rate is guaranteed.

The invention relates to the technical field of artificial intelligence and emotion calculation, and discloses an emotion state evaluation method based on multi-modal data, and the method comprises the steps: obtaining and decoupling an implicit signal, an explicit signal and situation data of an evaluated object; and respectively generating implicit features, explicit features and context vectors through double-flow decoupling coding and context knowledge graph coding. And the core engine determines the internal emotional state and the extrinsic emotional representation in parallel, and dynamically regulates and controls a bridging conversion process from the internal state to the predicted extrinsic representation by utilizing the situation vector. Finally, the emotion regulation intensity is calculated by comparing actual and predicted extrinsic representations, and the emotion regulation intensity, the internal state and the extrinsic representations are jointly used as a multi-dimensional evaluation result to be output. According to the method, the dynamic relationship and the adjustment strategy between the internal feeling and the external expression can be disclosed, a deeper and more three-dimensional analysis view angle is provided for the fields of mental health, human-computer interaction and the like, and the method has remarkable application value.

The invention relates to the technical field of computer vision and emotion calculation, in particular to an emotion real-time recognition and long-term monitoring method based on facial expressions. According to the method, an emotion recognition result is obtained by recognizing a high-definition facial image, and the emotion recognition result, environment information and physiological state data are fused to obtain time-space aligned multi-modal data; performing emotional causal analysis based on the multi-modal data, and judging emotional causes by combining a rule engine and a machine learning model: outputting a real-time emotional state recognition result and a periodic emotional report according to the emotional causes, and performing differentiated feedback according to the emotional causes. According to the method, through multi-source data fusion and a causal inference mechanism, the accuracy and interpretability of emotion recognition are effectively improved, and the technical span from passive recognition to personalized active intervention is realized.

The invention discloses a multi-mode emotion recognition method and system based on prompt enhancement, a medium and equipment, and belongs to the technical field of emotion analysis, and the multi-mode emotion recognition method based on prompt enhancement comprises the steps: obtaining input data of text, visual and audio modes; the modal-based prompt encoder performs prompt template-based feature enhancement on each modal input data to generate prompt enhanced feature representation; establishing an inter-modal information exchange channel by adopting a cross-modal adaptive alignment mechanism based on prompt enhanced feature representation, and generating alignment dominant features; based on the aligned dominant features, dynamically fusing the high-quality features by adopting a multi-level quality evaluation mechanism to generate enhanced dominant features; and performing multi-modal fusion on the enhanced dominant feature and the other two modal features, and outputting an emotion prediction result. According to the method, different data missing modes are effectively dealt with, the limitation of a rigid interaction mode of a traditional method is solved, the quality and reliability differences of different modes are considered, and robust multi-mode sentiment analysis is achieved.

The invention relates to the field of human-computer interaction, and discloses a human-computer interaction robot facial expression implementation control method, which comprises the following steps: dynamically capturing the face, sound and posture of a user in a human-computer interaction scene, and generating an initial emotion perception sequence; performing hierarchical cross mapping on the initial emotion perception sequence, and performing combinatorial analysis on facial expressions, voices and intonations and body movement features to form an emotion matching vector; based on the emotion matching vector, utilizing a priority regulation model to predict the emotion trend of the user at the next moment, and identifying an expression enhancement point and an emotion attenuation area; mapping the generated facial micro-expression adjusting instruction to a robot facial driving unit, and performing real-time correction and conflict resolution on an expression action sequence; and reversely fusing a user fixation point, facial muscle micro-motion and voice emotion which are acquired in real time into an emotion matching vector, and dynamically updating a perception weight and expression regulation and control parameters. The method has the advantage of improving the emotion matching degree of the robot expressions.

The invention discloses a multi-modal emotion calculation method and system, and mainly relates to the technical field of artificial intelligence and human-computer interaction. Comprising the following steps: synchronously collecting voice, visual and tactile multi-modal data of a user, and carrying out feature extraction on each modal data; performing cross-modal fusion on the extracted multi-modal features, including time sequence alignment and semantic association modeling, and generating a global emotion feature vector; performing dynamic emotion reasoning based on the global emotion feature vector, and outputting an emotion category and emotion intensity; generating a tactile feedback signal according to the emotion category and the emotion intensity, and driving an actuator to output; and updating the emotion memory graph based on user interaction data to complete personalized model self-adaption. The method has the beneficial effects that high-precision and low-delay emotion recognition and natural tactile feedback in a cross-culture scene are realized, and the core bottleneck of dynamic drift adaptation failure and emotion-behavior feedback decoupling in the prior art is solved.

The invention discloses an educational resource recommendation method and system based on artificial intelligence. The method comprises the following steps: acquiring audio data, interaction data and task data acquired by a user terminal; performing feature extraction on the audio data, the interaction data and the task data to obtain an emotion feature vector, a learning rhythm vector and a content feature vector; inputting the emotion feature vector and the learning rhythm vector into a pre-constructed emotion recognition model to obtain a psychological state vector; the cognitive load is calculated based on the learning rhythm vector and the content feature vector, and then the cognitive load is corrected through the psychological state vector; matching a state interval of the corrected cognitive load according to a preset threshold interval; and according to the state interval, adjusting a difficulty coefficient of the recommended course, rearranging a course content sequence and an auxiliary learning prompt, generating structured data, and outputting the structured data as an intelligent auxiliary learning recommendation result. According to the invention, online learning interactivity and teaching quality in rural and remote areas are effectively improved.

The invention discloses an intelligent communication content adaptation system and method based on user intention recognition. The system integrates and processes texts, voices, emoticons and unstructured behavior data through a multi-modal input analysis module; the deep learning intention recognition engine adopts a triple attention mechanism, current input and weighted historical interaction features are fused, and multi-level intention classification including basic operation, semantic targets and emotion driving is output; the real-time emotion state analysis module fuses acoustics, semantics and physiological indexes to generate a dynamic emotion matrix; the content adaptation decision engine is based on the intention confidence and the emotional state; and the multi-channel output optimization module performs cooperative adjustment on the speech synthesis rhythm, the text abstract and the visual interface according to the speech synthesis rhythm and the text abstract. According to the method, the perception precision of the deep intention and emotion of the user in a complex scene is remarkably improved, the individuation and multi-channel adaptive optimization of the response content are realized, and the communication efficiency and the user experience are effectively improved.

The invention discloses a knowledge enhancement and sentiment inconsistency-based multi-modal chaffy detection method, which comprises the following steps of: obtaining a to-be-detected sample containing a text and an image, and extracting image and text features and text embedding features by using a feature encoder; and obtaining image description and texts in the image based on the image, splicing the image description and the texts to form knowledge texts, and sequentially inputting the knowledge texts into the emotion dictionary and the feature encoder to obtain emotion features and knowledge embedding features. And carrying out feature interaction among the image, the text and the emotion features by adopting cross attention, and carrying out adaptive weighting on the image and text features through a gating mechanism to obtain image-text comprehensive features and emotion features processed by the cross attention. And inputting the text embedding features and the knowledge embedding features into an emotion inconsistency module, and calculating an emotion inconsistency expression. And based on the image-text comprehensive features, emotion features subjected to cross attention processing and emotion inconsistent representation, performing chiffon prediction, and outputting a detection result. According to the invention, the method achieves better prediction performance on a multi-mode anti-tech detection public data set, and can more accurately recognize an image-text sample with irony emotion.

The invention belongs to the technical field of multi-modal information processing, and particularly relates to a multi-modal emotion recognition method, which can give consideration to modal generality and modal difference at the same time, realizes adaptive fusion between modals, and is suitable for emotion understanding and analysis of video, voice and text multi-source data. Comprising the following steps: 1, preprocessing a CMU-MOSI data set and a CMU-MOSEI data set to generate training data; 2, processing data by adopting a pre-training encoder to obtain a high-quality initial feature sequence; 3, inputting each modal initial feature into a sharing and private coding unit through a feature decoupling module, and respectively obtaining a cross-modal public emotion sharing feature and a modal specific emotion private feature; according to the multi-modal emotion recognition method provided by the invention, the emotion recognition capability is remarkably improved, and effective data support can be provided for scenes such as psychological analysis, medical assistance and human-computer interaction.

The invention relates to the field of man-machine interaction, and discloses a man-machine interaction method and system based on a large language model, and the method comprises the steps: extracting original semantic features from the current input of a user; obtaining a context representation of the current dialogue; extracting a historical emotional state sequence from the multi-round dialogue historical record; generating a current user emotion representation vector based on the context representation of the current dialogue and the historical emotion state sequence; constructing a hierarchical memory structure based on the current user emotion representation vector and the context representation of the current dialogue; adjusting the attention degree of the historical dialogue content stored in the semantic memory component according to the hierarchical memory structure; and generating a response sequence through a large language model based on the adjusted attention degree, the hierarchical memory structure and the context representation of the current conversation, and outputting the response sequence to the user. According to the technical scheme, the problems of stiff switching and dialogue breakage during emotion turning of a traditional dialogue system are solved.

The invention relates to the technical field of artificial intelligence and man-machine interaction, and discloses a picture book co-reading method, system and device based on multi-modal emotion recognition and a medium, and the method comprises the following steps: obtaining multi-modal time sequence data of a user in picture book co-reading; establishing and updating a longitudinal emotion file for recording historical interaction data for the user; based on the data and the archive, personalized multi-dimensional internal state evaluation is carried out, and the evaluation comprises calculation of cognitive load indexes representing difficult understanding, recognition of instant emotional states and analysis of contextual emotional deviations of the emotional states and current plot expectations; and finally, based on the multi-dimensional evaluation result, determining and executing dynamic interaction strategies such as simplification, guidance or excitation. According to the method, deep personalization is achieved by constructing the multi-dimensional state model and combining the longitudinal archives, the user state can be more accurately judged, prospective and self-adaptive interaction is achieved, and the interaction fineness and effectiveness are improved.

The embodiment of the invention discloses a cerebral apoplexy upper limb rehabilitation training system and method based on dynamic reward feedback. A data acquisition module is used for acquiring physiological signals, motion signals, emotional state data and training performance data; the motion intention decoding module is used for performing motion intention feature extraction and reliability evaluation on the physiological signals according to an improved MREE-Net + + fusion algorithm, performing weighted fusion after feature weights are adjusted according to a reliability result, obtaining a unified motion intention feature vector, performing motion intention decoding, and obtaining a motion intention intensity index and a motion intention vector; the dynamic reward decision module is used for processing the facial micro-expression data according to an improved VGG-Face model to obtain an emotion titer, and obtaining an emotion awakening degree according to the voice signal; a reward action is generated according to an improved deep reinforcement learning algorithm; and the adaptive training regulation and control module is used for generating a personalized virtual training scene according to the generative adversarial network and regulating and controlling the training intensity according to the fatigue index. The rehabilitation training effect can be improved.

The present invention relates to a document analysis system that utilizes artificial intelligence to analyze documents written by group members in their daily lives, extract the key emotions and key keywords of the document writer, and determine relationships among group members. By utilizing the extracted key emotions and key keywords, the system analyzes and intuitively presents whether certain members are connected as a group or close companions, or whether certain members are isolated and not interacting with others.

The invention relates to the technical field of digital human voice interaction, and discloses a digital human live voice interaction system fused with emotion calculation. The system constructs an emotional response time window by acquiring a user voice input data stream, the starting point of the emotional response time window is the end moment of the user voice input data stream, and the end point is obtained by subtracting the necessary duration of voice response synthesis from the preset maximum response cut-off moment. The system obtains an emotional state vector of the user in real time and judges whether the emotional state vector reaches an emotional intensity threshold value or not; and if so, predicting the total generation duration of the digital human voice response. And when the residual duration of the emotional response time window is equal to the total generation duration, taking the window starting point as a voice response starting moment, and controlling the digital human to start voice response generation. According to the system, the voice response matched with the emotional state of the user is generated through refined time window management and emotional state perception, invalid response and interaction delay are reduced, and the naturalness of digital human live broadcast voice interaction and the user experience are improved.

The invention discloses a multi-modal sentiment analysis method combining dynamic sentiment knowledge and a sparse attention mechanism. The method comprises the following steps: acquiring and preprocessing text, image and audio multi-modal training samples; extracting preliminary features of each modal by using the pre-training model; emotion feature expression is enhanced through an emotion related feature extraction module; the enhanced features are input into a Transform Encoder, and an emotion knowledge vector is dynamically generated; guiding multi-modal feature semantic alignment by using the vector; reserving the first k maximum values of the attention score by adopting a sparse attention mechanism, and realizing effective fusion of modal features; and finally, outputting an emotion analysis result through the emotion classifier. According to the method, a multi-task joint loss function is designed, and end-to-end optimization is carried out in combination with classification loss, alignment loss and comparison loss. According to the method, the problems of difficulty in modal alignment, much fusion redundant information and insufficient cross-domain adaptability in multi-modal sentiment analysis are effectively solved, and the accuracy and generalization ability of sentiment analysis are remarkably improved.

According to the man-machine interaction-oriented multi-modal fine-grained emotion recognition method based on the large model provided by the invention, cross-modal alignment from coarse granularity to fine granularity is realized through an attention pairing interaction module (APIM) on the basis of an aspect-driven vision-text alignment and fusion network (AVTAF); emotion-related visual features (such as facial expressions and gestures) in a robot scene can be accurately captured, and environmental noise is inhibited; meanwhile, the RD-GAT is enhanced, and the reasoning ability of a large model on multi-modal emotion semantics is improved by integrating external emotion knowledge (such as SenticNet). The technology provides a new normal form for intelligent upgrading of robot emotion interaction and multi-modal understanding of a large model, and is expected to promote breakthrough application in the fields of family service robots, medical accompanying assistants, multi-modal content generation and the like.

The invention provides a group psychological data fluctuation early warning method based on VR emotion data processing, which comprises the following steps: acquiring multi-modal physiological signals such as electroencephalogram, heart rate variability and skin conductance of a user and behavior data such as voice, expression and virtual trajectory in real time, performing sliding window processing and feature extraction, fusing into multi-dimensional features, inputting the multi-dimensional features into an emotion recognition model, and performing emotion recognition on the emotion recognition model; generating an individual emotion dynamic sequence; a dynamic heterogeneous graph is constructed by combining the social relation and the emotion similarity, the emotion influence intensity between nodes is judged by using a dual-channel graph attention mechanism, and a propagation path is traced through an integral gradient method, so that interpretable group emotion diffusion mode recognition is realized; according to the method, a typical propagation mode and a time sequence convolutional network are combined, a group emotion intensity evolution trend is predicted, when fluctuation exceeds a threshold value, an emotion propagation path thermodynamic diagram is automatically generated and early warning is pushed, and the interpretability of an emotion propagation path and prediction and response efficiency of group psychological fluctuation are improved.

The invention discloses an emotion analysis method based on a multi-modal large model. The method comprises the following steps: S1, extracting multi-modal emotion features; respectively designing special emotional feature extractors for three modes of facial expression, voice and text; s2, carrying out cross-modal emotion alignment and fusion; mapping the emotion features of different modes to a unified emotion semantic space; s3, an emotion inconsistency detection mechanism; the method is specially used for detecting the emotion inconsistency phenomenon between different modes. S4, fine-grained sentiment classification is carried out; the sentiment classifier comprises three sub-tasks of basic sentiment classification, complex sentiment recognition and sentiment intensity regression; s5, a model fine tuning training strategy; and optimizing the performance of the model by adopting a multi-stage fine-tuning training strategy. According to the method, deep fusion and accurate analysis of facial expressions, voice acoustic features and text semantic information are realized, so that a complex emotional state and an emotional inconsistency phenomenon are effectively recognized.

An information processing method includes: obtaining, as text information, information related to communication by a person; performing emotion analysis on a plurality of words or phrases after breaking down the text information into the plurality of words or phrases by performing morphological analysis on the text information; and visualizing an analysis result of the emotion analysis according to each row and column of a matrix table by forming the matrix table by arranging, in a first direction, a plurality of emotional expression-related items that represent human emotions and arranging, in a second direction, an attribute category-related item that indicates an attribute of the person or an organization.

The invention discloses a dialect recognition and emotion feedback multi-mode interaction system based on large model driving, and relates to the technical field of intelligent man-machine interaction, the system comprises a dialect emotion interaction engine, an article situation analysis module and a situation self-adaptive response module, and the problems of dialect emotion transmission distortion and situation understanding splitting in the prior art can be solved. The dialect emotion interaction engine generates a target dialect acoustic parameter through a pronunciation decoupling architecture, and combines a thermodynamic diagram to dynamically compensate and optimize emotion expression; the article situation analysis module improves the emotion recognition robustness in a complex environment based on an article-emotion knowledge graph and cross-modal arbitration; the context adaptive response module generates adaptive dialect responses according to the emotions and the item semantics. According to the system, technologies such as end-to-end generation and adversarial learning are adopted, zero-sample dialect adaptation and progressive capacity upgrading are achieved, the emotion requirements of old dialect users can be accurately matched, and the interaction experience is improved.

The application discloses a recruitment interview evaluation method and system based on artificial intelligence and a storage medium, relates to the technical field of recruitment interview evaluation, and solves the technical problem that the prior art only completes morphological analysis and pragmatic reasoning through voice recognition and corpus comparison, does not deeply mine dynamic characteristics of voice signals, and does not perform correlation analysis on personality descriptions in resumes and real-time emotional performances in interviews, so that the evaluation dimension is single, and it is difficult to comprehensively and objectively reflect the language expression ability, psychological state and personality matching degree of a job seeker; the application obtains voice data and emotional data of a job seeker; the voice data is preprocessed to obtain a formant sequence corresponding to the voice data; a voice evaluation coefficient is calculated based on the formant sequence; an emotional similarity is calculated based on the emotional data; an interview evaluation coefficient is calculated based on the voice evaluation coefficient and the emotional similarity; and whether the job seeker is qualified is judged based on the interview evaluation coefficient, so that the above technical problem is solved.

The invention discloses a prompt-driven two-stage multi-modal emotion representation learning method. The method comprises the following steps: respectively collecting original video data from a plurality of public multi-modal emotion analysis data sets; preprocessing and feature extraction are carried out to obtain vectorized multi-modal feature representations of vision, audio and texts, and multi-source emotion clues are obtained; in a training stage, a prompt-driven two-stage multi-modal emotion representation learning model is constructed, inter-class separability is enhanced and intra-class intensity features are reserved through an emotion anchor point comparison and alignment stage, including prompt-based emotion anchor point learning and comparison and alignment between joint representation and emotion anchor points; capturing the dynamic change of emotion expression through an emotion intensity offset estimation stage; in the reasoning stage, emotion category prediction and final emotion state prediction are carried out. According to the method, shared emotion features among multiple modes can be stably captured, interference caused by individual differences is effectively suppressed, and the accuracy and robustness of emotion prediction are remarkably improved.

The invention discloses a multi-modal driving system based on streaming big language model output, which belongs to the technical field of multi-modal interaction and comprises the following steps: a streaming big language model module receives input information and then generates a text unit sequence in a streaming manner; when each text unit is generated, a multi-dimensional emotion label sequence containing emotion types, emotion intensity values and semantic association degrees is synchronously output; the speech generation module maps the text unit sequence and the corresponding multi-dimensional emotion label sequence into speech synthesis parameters, and drives a streaming text-to-speech model to generate an emotional speech stream; the label generation module generates an expression label sequence and an action label sequence of the virtual image in real time; the emotion synchronous control module performs synchronous interpolation and coordination control by taking the time step of the text unit as a reference to generate a synchronous multi-mode driving signal; and the rendering and driving module drives the virtual image based on the signal to realize collaborative output of voice, mouth shape, expression and limb movement, and realizes synchronous presentation of multi-modal emotion expression.

The embodiment of the invention discloses an accompanying robot control method and device, computer equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: collecting environment information, multi-mode emotion information of a user, and user instruction information; identifying an emotional state of the user based on the multi-mode emotional information; identifying a user intention based on the user instruction information; and performing multi-source information fusion analysis based on the environment information, the emotional state and the user intention, generating a control instruction, and controlling an accompanying robot to execute the control instruction. According to the method, the limitation of a single information source is broken through, the problems that emotion recognition is easily interfered and intention understanding is rigid are solved, and the crossing from passive response to active understanding is realized, so that the accompanying robot can be controlled to provide accurate service according to the actual state and the actual environment condition of the user.

The invention provides an emotional interaction decision-making method and system for a humanoid robot. The emotional interaction decision-making method comprises the following steps: acquiring multi-modal data, acquired by a multi-modal acquisition system, of a user interacting with the humanoid robot; preprocessing the collected multi-modal data to obtain a multi-modal emotion data set; constructing an improved multi-modal fusion network model, and training the improved multi-modal fusion network model by using the multi-modal emotion data set to obtain an emotion-behavior mapping decision model; performing sentiment analysis on the real-time input of the user through the sentiment-behavior mapping decision model, and generating a robot interaction behavior instruction or a service response strategy matched with the sentiment state of the user; the method has the following beneficial effects that the multi-modal emotion of the user can be accurately recognized, and the humanoid robot can be driven to make physical behavior feedback with temperature and emotional quotients in real time, so that the naturalness, affinity and emotion connection experience of man-machine interaction are remarkably improved.