1489 results about "Facial affect" patented technology

A method and system are introduced for recognizing cognitive engagement in classrooms by utilizing multimodal data. Associated modalities of cognitive engagement are learned from visual and audio data. This approach involves constructing a multidimensional representation model for cognitive engagement that includes behaviors, emotions, and speech. To identify engagement, three distinct deep learning models are employed: You Only Look Once version 8 (Yolov8) for analyzing body posture, Efficient Network (EfficientNet) for facial expressions, and Text Convolution Neural Network (TextCNN) for speech text. These models are trained and refined with the aid of student engagement surveys, leading to a decision-making process that integrates the results from the different modalities. Additionally, a dataset and a data annotation system are developed for engagement recognition. This innovative method aims to achieve detailed engagement recognition, addressing various perception needs in real-world applications.

The invention discloses an intelligent multi-modal virtual digital human interaction system based on an AI language large model. The system comprises a high-authenticity face generation module; the high-authenticity face generation module uses an AdaAN network, based on adaptive feature fusion and voice driving and time sequence modeling of voice features, feature information related to voice is extracted, the extracted voice features are processed through a deep neural network, it is ensured that the voice and facial expressions are highly aligned in time and space, and the face recognition accuracy is improved. Collecting a bio-electricity signal, mapping the signal to facial muscle movement, generating a final facial expression, and interacting with a user; the system further comprises an intelligent interaction module, a training optimization and efficient generation module, an efficient integration module, a multi-modal data acquisition module, an AI large model core processing module, a digital human image generation and driving module, an interaction scene adaptation module and a feedback optimization module. The invention further discloses a multi-mode digital human interaction method which has wide application value.

The invention discloses an intelligent real-time emotion evaluation method for social media and online text data based on a multi-modal knowledge graph, and the method comprises the steps: firstly obtaining and preprocessing multi-modal original data; secondly, designing single-mode feature extraction of dialogue texts, voices and facial expressions based on BERT-BiLSTM, AST and ResNet-50 pre-training models; furthermore, sharing among multiple modes and private emotion semantic information are considered, and complementation of cross-mode emotion information is promoted by enhancing repeated emotion information and utilizing uniqueness of private information. Meanwhile, a multi-mode knowledge graph method is provided from bottom to top to carry out emotion real-time evaluation. Analyzing user behaviors by using DeepFM and generating user features, and capturing external knowledge layer features in combination with K-BERT; finally, on the basis of a TransformerEncoder structure, cross-modal emotion information interaction and feature fusion are achieved, real-time recognition and classification of multi-modal emotion perception are completed through a Softmax layer, and therefore the purpose of intelligent real-time emotion evaluation is achieved.

The invention provides an intelligent pre-examination triage method based on multi-modal data and edge calculation, and relates to the technical field of multi-modal data, and the method comprises the steps: collecting the facial expression, wound part, sound and physiological signals of a patient through medical equipment; multi-modal features are extracted by using a deep learning network and dynamic weighted fusion is carried out to obtain a patient state evaluation result; performing data time sequence alignment and feature extraction based on edge nodes, dynamically adjusting calculation precision and realizing task migration; multi-source data are collected for time sequence alignment calibration and feature fusion, the emergency degree is calculated, and group damage distribution is predicted. According to the method, real-time processing and dynamic optimization of multi-modal data are realized, and the accuracy and efficiency of pre-examination triage are improved.

The invention relates to the field of data processing, in particular to an emotion recognition method based on a large model and a related device. The method comprises the following steps: firstly, acquiring to-be-processed multi-modal data; the multi-modal data at least comprises text information, audio information and image information input by a user; the image information at least comprises a facial expression image and / or a limb action image. And inputting the multi-modal data into an emotion recognition model for emotion state analysis to obtain a predicted emotion state corresponding to the user. And finally, performing confidence evaluation based on the predicted emotional state, and generating a user emotion recognition result based on the predicted emotional state and a corresponding evaluation result. According to the method, richer emotion details are captured from the multi-modal data through the emotion recognition model, cross-modal emotion intelligent recognition is realized, the accuracy of emotion recognition is further improved by combining confidence evaluation, the emotion recognition efficiency is improved, and the application range of emotion recognition is expanded.

The invention provides a multi-mode-based man-machine interaction method and system, and relates to the technical field of man-machine interaction. Voice input of a user is received, environment information is collected, facial expressions of the user are captured, and voice features, environment features and facial features are obtained based on the input; utilizing a pre-trained emotion recognition model to generate emotion features of the user, and utilizing a pre-trained user behavior model to generate behavior features according to the current operation information of the user; after the features are obtained, the first weights and the second weights of the features are dynamically adjusted, feature weighted fusion is carried out, and comprehensive features are obtained; the multiple candidate responses are generated based on the comprehensive characteristics, the final response is screened out through a preset scoring mechanism and output, comprehensive understanding of the intention and the emotional state of the user is achieved through comprehensive utilization of multiple data sources, and meanwhile the weight of each characteristic can be dynamically adjusted so that more intelligent and humanized interaction experience can be generated.

The invention discloses an emotion-enhanced digital human driving and presenting system and method, and belongs to the technical field of computers.The system comprises an audio decoupling module, a potential diffusion model module, a human motion prior module, a training module and a rendered image generation module; predicting and reconstructing the facial expression and gesture action sequence to obtain a final action sequence; and generating a neural rendering image based on the final action sequence. According to the method, through voice input, emotion and style features in voice are captured, a cross attention feature fusion mechanism is introduced to enhance expression of emotion and personal style features, a voice-driven gesture generation task is expanded to a facial expression and gesture action synchronous generation task, the integrity of generated digital human actions is realized, and the user experience is improved. The emotional expressive force of facial expressions and gestures of the digital human is greatly improved.

An autonomous environmental regulation and behavioral monitoring system is disclosed, configured to adapt temperature, lighting, and acoustic conditions based on real-time emotional and physiological data. The system includes a dual-redundant central processor, hierarchical communication networks, multi-angle visual acquisition units, infrared thermometers, and modular environmental subsystems. It detects posture, gestures, facial expressions, and thermal signals to classify user states and apply individualized airflow, light, and sound modulation without relying on external internet connectivity. The system also monitors connected appliances using voltage-based pressure analysis to forecast device degradation. With integrated gesture recognition, privacy-preserving data handling, and predictive adaptation, the invention enables multi-user personalization, long-term learning, and uninterrupted operation within residential, administrative, or healthcare infrastructures.

The invention provides an AI-based virtual digital human interaction system, and relates to the technical field of data processing, and the system is used for obtaining a user feature data set by performing feature extraction on an interaction data set, calculating an audio change rate and speech speed stability, obtaining speech emotion data, calculating syntactic complexity and counting the number of emotion words. Calculating a facial expression symmetry degree; obtaining facial emotion data; calculating body stability and identifying a visual staying area; obtaining attention state data; adjusting tone and speed of the virtual digital person according to the virtual digital person text, optimizing sentence pattern complexity and sentiment words in the virtual digital person text, adjusting changes of eyes, eyebrows and mouth corners of the virtual digital person, and adjusting information display sequence and visual focus guidance. According to the method, the interaction efficiency of the virtual digital human is improved by analyzing the facial expressions and the body actions of the user.

The invention provides a dynamic calibration method and system for a vehicle-mounted emotion recognition system, and the method comprises the steps: S1, obtaining multi-source data which comprises a facial image, a voice signal and a physiological signal; the obtained multi-source data are preprocessed, and preprocessed multi-source data are obtained; s2, performing feature extraction based on the preprocessed facial image, the voice signal and the physiological signal to obtain a facial expression feature vector, an audio feature vector and a physiological state feature vector; s3, evaluating the current environment credibility based on an environment credibility evaluation function; s4, dynamically distributing the weight of the multi-source data according to the credibility of the current environment and the real-time scene; and S5, constructing a multi-modal fusion vector based on the dynamically distributed weight of the multi-source data, the facial expression feature vector, the audio feature vector and the physiological state feature vector, and performing emotion recognition by using the constructed emotion recognition model based on the multi-modal fusion vector.

The invention provides a self-learning multi-modal emotion recognition method based on multi-scale cavity attention, and solves the problem of low recognition precision caused by different importance of basic action units of a face and different distances between key action units, and the problem of different modality confidence during decision-level fusion. The method comprises the following steps: preprocessing a facial expression image, inputting the facial expression image into a multi-scale cavity attention convolution module, extracting features through a parallel three-branch convolution structure, splicing the features, calibrating through an attention mechanism to obtain an enhanced feature map, and sending the enhanced feature map to a full connection layer to recognize emotion; an original electroencephalogram signal is input into a time-frequency-space three-dimensional feature extraction network, the signal is decomposed, differential entropy features are calculated and processed by a global attention module comprising a frequency spectrum attention module, a space attention module and a time attention module, time-frequency-space multi-dimensional feature representation is output, and emotions are recognized by a full connection layer; and finally, inputting the emotion recognition result of the facial expression and the electroencephalogram signal into a self-learning weight module, and obtaining a final emotion recognition result through dynamic weighted fusion.

The invention discloses a multi-modal large model identification and intelligent intervention method and system for mental disorder, and belongs to the technical field of intelligent identification and intervention of mental disorder, and the method comprises a multi-modal data acquisition stage, a data preprocessing stage, a feature extraction stage, a cross-modal feature fusion stage, a mental disorder identification stage, and a personalized non-drug intervention stage. In the multi-modal data acquisition stage, skin electricity, electrocardio, electroencephalogram and facial expression multi-modal data are acquired in real time, and a dialogue-feedback-adjustment closed-loop mechanism is constructed in combination with a cognitive behavior therapy theory. According to the multi-modal large model recognition and intelligent intervention method and system for mental disorder, accurate recognition and personalized intervention of mental disorder are achieved by integrating electroencephalogram, electrocardio, skin electricity, expression analysis and text data multi-modal physiological and behavior data and combining a large language model and a cross-modal attention mechanism.

The invention discloses a student sentiment analysis method and system based on a multi-modal dynamic memory large model, and belongs to the field of multi-modal sentiment analysis. The method comprises the following steps: firstly, acquiring facial expression images and corresponding real emotional state category labels when different students watch an online education course to form a training data set, training a multi-modal dynamic memory large model based on the data set, and in the training process, performing prompt fine tuning on an image encoder and a text encoder so as to obtain a multi-modal dynamic memory large model; visual aggregation representation generated in the historical training process is stored through a dynamic feature space, and model parameters are optimized through cross entropy loss; and finally, inputting a facial expression image to be classified into the trained multi-modal dynamic memory large model, and outputting an emotional state category prediction result of the facial expression image. The method can be effectively applied to the field of education, and is beneficial for teachers to know the emotional state of students, thereby improving the participation degree and learning effect of the students.

The invention provides a dynamic self-adaptive multi-modal sentiment analysis fusion method and system, and relates to the technical field of multi-modal sentiment analysis. The method comprises the following steps: synchronously acquiring voice, text, facial expression and limb movement data of a target user to form a multi-modal data set; the method comprises the following steps: firstly, extracting emotional characteristics of each mode, and constructing a cross-mode correlation model to capture a collaborative and complementary relationship among different modes; and calculating a real-time confidence score and a complementarity index of each modal based on the weight matrix of the cross-modal correlation model. Then, according to the scores and the indexes, a weighted average or maximum entropy algorithm is dynamically selected to fuse multi-modal emotion features, and a comprehensive emotion feature vector is generated; and finally, inputting the vector into a pre-training deep learning model, and outputting an emotional state category of the user. According to the method, the user emotion can be accurately and comprehensively captured, efficient emotion recognition and classification are realized, and the robustness and flexibility of an emotion analysis system in a complex scene are improved.

The invention discloses a touch and talk pen intelligent children education method based on emotion recognition. The method comprises the steps that S1, facial expression data, voice and intonation data and touch behavior data of children are collected; s2, multi-modal high-quality emotion data matched with the actual emotion state is generated; s3, generating deep emotional features related to facial expressions, voices and tones and touch behaviors; s4, generating an emotional state label corresponding to the current emotional state of the child; s5, the generation model generates voice feedback content, image animation content and interactive game content which are adaptive to the current emotional state; s6, generating a personalized recommendation content list conforming to the emotional state of the child; and S7, displaying the education content in the personalized recommendation content list through a display screen, a voice output unit and a touch interaction unit of the touch and talk pen. According to the method, the fuzzy classification problem of the emotional state is solved in a touch and talk pen emotion recognition scene, and the adaptability of the model to the complex emotional state is remarkably improved.

The invention discloses a psychological accompanying method based on multi-modal emotion recognition, and belongs to the technical field of psychological health services, and the method comprises the steps: S1, synchronously collecting physiological signals, voice features, facial expressions and interactive behavior data of a user through a multi-modal sensor; s2, performing fusion analysis on the multi-modal data by using a deep learning model, and identifying a current emotional state and an emotional intensity level of the user; and S3, dynamically generating an adaptive psychological accompanying intervention scheme based on a preset emotion-intervention strategy mapping rule in combination with historical emotion data and personalized preferences of the user. According to the method, the physiological signals, the voice features, the facial expressions and the interactive behavior data are synchronously collected through the multi-modal sensor, the deep learning model is used for fusion analysis, and compared with single-modal recognition, the emotion state and the intensity level of the user can be judged more comprehensively and accurately, the emotion misjudgment risk is reduced, and a reliable basis is provided for subsequent intervention.

The invention discloses an academic emotion recognition method and device based on facial expressions and contexts. According to the method, accurate emotion modeling is realized by utilizing a visual language pre-training model CLIP and combining student facial expressions in a video sequence and contextual information of a learning situation. Specifically, according to the method, firstly, collaborative modeling is carried out on facial expressions and context information in a learning scene, so that the ability of recognizing the learning state (such as focusing on learning and distracting) of students is remarkably improved; and secondly, the strong language and visual feature alignment capability of the CLIP pre-training model is utilized, so that the dependence on large-scale video training data is effectively reduced, and the generalization capability and practicability of the model are improved. An accurate and feasible solution is provided for academic emotion recognition, and the method is suitable for monitoring and analysis of academic emotions of students in an education scene.

The invention relates to a virtual image expression generation method and system based on a real feeling technology. The method comprises the steps of collecting facial expression data in a multi-angle mode, and performing triangulation and facial region marking through the collected facial expression data; component analysis is carried out on the facial region, main features of facial expression changes are extracted, and a facial model is established through weighted fusion; establishing a mapping relation between the expression features and the facial displacement, and calculating a geometric constraint function of the facial displacement; correspondingly generating different expression models according to the geometric constraint function, optimizing details of a specific area and realizing dynamic face rendering on the virtual image; by introducing multi-view data acquisition, a multi-level feature extraction and fusion strategy, expression displacement regression analysis, geometric constraint and detail optimization, the authenticity and personalized effect of virtual expression generation are greatly improved.

A method of three-dimensional reconstruction of human heads using a single photo in the form of polygonal mesh, with animation and realistic rendering capabilities for novel head poses is provided. The method includes encoding, by using a first convolutional neural network, a single source image into a neural texture; estimating, by a pre-trained detailed expression capture and animation (DECA) system, a face shape, a facial expression, and a head pose by using the single source image and a target image, and providing an initial mesh; providing a predicted mesh of a head mesh based on the initial mesh and the neural texture; rendering a human image by using the predicted mesh.

The invention discloses an intelligent customer service interaction method and system, and relates to the technical field of intelligent customer service interaction, and the method comprises the steps: obtaining a user request; calculating a comprehensive emotion score through multi-modal emotion analysis based on a user request; according to the comprehensive emotion score, classifying the user emotion state, according to a classification result, initializing a dialogue context, and constructing a dialogue context graph; dynamically adjusting the response mode of the intelligent customer service according to the emotional state and the user question in the dialogue context graph, and collecting the feedback information of the user; and dynamically adjusting the dialogue strategy based on the feedback information of the user and the dialogue context. According to the method, the emotion state of the user can be comprehensively captured from multiple aspects such as texts, voices and facial expressions through multi-modal emotion analysis, so that the accuracy and robustness of emotion recognition are improved, a reliable basis is provided for subsequent dialogue management and response modes, and the experience satisfaction degree and the service quality of the user are improved.

The invention relates to a vehicle control method and device, equipment and a medium, and the method comprises the steps: obtaining voice data and video data of a target user in a vehicle, inputting the voice data and the video data into an instruction generation model, and obtaining a target instruction text outputted by the instruction generation model; and then the vehicle is controlled through the target instruction text. Wherein the instruction generation model is used for fusing audio features corresponding to the voice data and image features corresponding to the video data so as to generate a target instruction text, the audio features comprise a first text sub-feature and a first emotion sub-feature, and the image features comprise a second text sub-feature and a second emotion sub-feature; the first emotion sub-feature is used for representing at least one of the tone, intonation and volume of speaking of the target user, and the second emotion sub-feature is used for representing the facial expression and / or lip change of the target user. According to the invention, the accuracy of vehicle control is improved, and the driving experience of a user is improved.

The invention relates to the technical field of robot perception, in particular to an intelligent accompanying human-type robot based on multi-mode emotion interaction and a perception method, and the method comprises the following steps: collecting facial expression and body posture data of a user through a visual perception unit, collecting intonation, speed, volume and keyword data through a voice perception unit, and carrying out the recognition of the speech, speed, volume and keyword data; data preprocessing and feature extraction are carried out; secondly, carrying out time sequence modeling on the extracted visual and voice features based on a recurrent neural network, and generating visual and voice time sequence feature vectors; the method comprises the steps that firstly, time sequence features are extracted, then, a multi-mode fusion model based on an attention mechanism is adopted, the time sequence features are fused into a comprehensive emotion feature vector used for emotion state classification, and finally, personalized interaction content such as voice, expressions and limb actions matched with the emotion state of a user is generated according to an emotion classification result. According to the method, the accuracy of robot emotion perception and the naturalness of man-machine interaction are remarkably improved, and the method is suitable for multi-scene application of education, medical treatment, old-age care and the like.

The invention discloses an employment and entrepreneurship support system based on artificial intelligence, relates to the technical field of occupational planning, and aims to solve the problem that a traditional support mode is insufficient in individuation and accuracy. The system comprises a multi-modal data acquisition module used for acquiring facial expressions, whole body dynamics and voice data of a user in at least 30 minutes of video dialogue in real time; a dialogue text is processed through a large language model based on a Transform architecture, continuous time sequence modeling is carried out on multi-modal dynamic behaviors by applying a liquid time constant network, and deep fusion reasoning is carried out in combination with multiple groups of multi-head Transform attention mechanisms. The core analyzes the real thought, psychological state, behavior pattern and core ability of the user through consistency verification, generates a structured dynamic user insight abstract, and customizes personalized vocational development or entrepreneurship planning according to the structured dynamic user insight abstract. According to the method, the potential of the user can be deeply informed, high-precision personalized planning is provided, the decision-making quality and success rate are improved, and the method has dynamic adaptation and continuous learning capabilities.

The invention discloses a text-driven advertisement video generation method based on emotion feedback, relates to the technical field of digital advertisement and man-machine interaction, and aims to dynamically adjust advertisement content and improve individuation, interactivity and conversion rate by collecting emotion data such as facial expression, voice features and eye movement tracks of a user in real time. A text generation video model and a multi-mode emotion recognition technology are combined, and feature fusion and feedback processing are utilized, so that the advertisement video better conforms to the emotion of the user; multi-round feedback and self-adaptive optimization ensure that advertisements transmit core information and flexibly respond to user emotional states at the same time, attraction and participation degree are greatly improved, an innovative feature fusion algorithm further improves individuation and real-time response capability of generated videos, a brand new solution is provided for the field of digital advertisements, and the method is suitable for popularization and application. And the development of individuation and emotional interaction is promoted.

A system for real-time analysis of emotional feedback during motivational speeches, consisting of: a series of multimodal sensors, including at least one visual sensor configured to capture facial expressions of spectators, at least one directional microphone configured to capture the audio responses of the audience, and optionally one or more physiological sensors configured to capture biometric signals from spectators; an edge-based processing unit that is communicatively coupled to the arrangement of multimodal sensors, wherein the edge-based processing unit comprises the following: (a) a feature extraction module configured to extract visual features from captured facial images, acoustic features from voice responses, and physiological features from biometric signals; (b) an emotion inference machine configured to process the features using a deep learning-based emotion recognition model comprising a convolutional neural network (CNN) for classifying facial expressions, a recurrent neural network (RNN) for classifying voice emotions, and a multimodal late fusion layer configured to compute a composite emotion state vector representing the aggregated emotions of the audience; (c) a timestamp and speech alignment module configured to correlate the calculated composite emotion state vector with segmented portions of a live motivational speech based on real-time speech-to-text transcription and semantic analysis; and (d) a session-based storage unit configured to log time-indexed emotional state vectors and corresponding speech segments for post-event analysis; A speaker feedback interface comprising a portable display or a podium-mounted visualization panel, wherein the interface is configured to display visual indicators of emotional feedback in real time, the indicators being derived from the emotional state vector and including at least emotional trend graphs, threshold alerts, or engagement indices.

The invention discloses an emotion recognition method based on electroencephalogram (EEG) and expression data, which combines fusion analysis of electroencephalogram (EEG) signals and facial expression data, and provides an innovative solution for the complexity and accuracy of emotion recognition. The method comprises the following specific implementation steps: 1, finding an optimal frame in an expression video by using multi-instance learning, extracting global visual features by adopting Swin Transform, and realizing dimension reduction and balance representation of cross-modal information through a cross attention mechanism; 2, preprocessing the electroencephalogram signal by using band-pass filtering and a fast ICA algorithm, and extracting effective features by using PSD; and 3, designing a fusion module based on Transform, combining the characteristics of the two modes, and giving full play to the complementary advantage of the modes. Experimental results show that 94.85% of classification accuracy is achieved in four types of emotion classification tasks, and compared with an existing method, the accuracy is remarkably improved.

The invention discloses a digital character real-time generation method and device, and the method comprises the steps: obtaining audio content and a two-dimensional digital character image, generating a model through a pre-trained whole body motion, processing the audio content, obtaining a three-dimensional whole body posture sequence, mapping the three-dimensional whole body posture sequence into a two-dimensional whole body posture sequence, and carrying out the real-time generation of a digital character. Performing real-time posture driving on the two-dimensional whole body posture sequence and the two-dimensional digital character image through a pre-trained real-time posture driving generation model to obtain a corresponding posture video, and adjusting a face image of a digital character in the corresponding posture video in real time through a pre-constructed lip synchronization model and audio content to obtain a lip synchronization result. Enabling the lip shape in the face image to be consistent with the audio content, and obtaining a target digital figure video which is a two-dimensional digital figure video in which whole body actions, facial expressions and voice content semantically are consistent.

The invention belongs to the technical field of vehicle-mounted human-computer interaction, and particularly relates to a vehicle-mounted music recommendation system based on context awareness, which covers the technologies of computer vision, recommendation systems, human-computer interaction and the like. The whole system is divided into two main parts: the first part is a context information collection module, and the module realizes real-time recognition of the emotion of a driver by collecting facial images and voice information of the driver; and the second part is a feedback module used for processing interaction between the driver and different emotion analysis results, including mapping rule reasoning and music recommendation. The system generates a personalized music recommendation list by analyzing the facial expression of the driver and the emotional characteristics extracted from the voice information, so as to reduce the music selection burden of the driver and prevent the occurrence of extreme emotions, thereby having important significance for guaranteeing the personal safety of the driver and the road traffic safety.

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.

A method of generating facial representations using a machine learned model is provided. A text encoder generates a representation of an input text segment. An aligner determines a time alignment between the input text segment and an input audio signal. A decoder generates a facial representation based at least in part on the representation of the input text segment, the time alignment, and on target style data, the target style data representing a target audio style, wherein the facial representation comprises a sequence of facial expressions corresponding to the input text segment. A method of training such a machine learning model is also provided.