366 results about "Facial expression recognition" patented technology

The invention discloses a facial expression recognition method fusing spatial channel features and fast regional convolution. The method comprises a fast regional convolution module, a spatial channel feature fusion module and a multi-scale attention module. The method comprises the following steps: firstly, extracting multi-scale local features by using depth separable convolution, and constructing a feature extractor by stacking a fast regional convolution module; then, on the basis of a space channel feature fusion module, feature fusion is realized by using deep convolution and a cross-channel self-attention mechanism, so that the calculation complexity and the memory requirement are remarkably reduced; finally, through a multi-scale attention module, feature extraction and spatial information aggregation of different scales are utilized, and the feature representation capacity of the expression change key area is enhanced. According to the invention, a lightweight facial expression recognition network is designed by using respective advantages of a full convolution model and a fast regional convolution model. A large number of experiments show that compared with other methods, the method provided by the invention has better performance, and meanwhile, the calculation cost and the network scale are far smaller than those of the same type of methods.

The invention provides a student attention assessment method and system based on big data analysis, and the method comprises the steps: capturing a class image of a student through a camera, collecting the facial expression of the student from the class image of the student, and recognizing the expression type of the student according to the facial expression of the student; acquiring student posture images from the student class images, and judging whether the students raise hands to answer questions or not according to the student posture images; attention scores are calculated from changes in the facial expression types and the body features of the students, and whether the attention of the students is concentrated is determined based on the attention scores. Through the method and the corresponding system, the attention of the student can be evaluated by using a plurality of sequences recorded by the video, and a more real result can be obtained.

The invention discloses a face emotion perception recognition method based on rPPG signals and three-dimensional normal vector assistance, and relates to the technical field of face recognition, and the method comprises the steps: carrying out the extraction of a face three-dimensional normal vector of a face video frame through employing a Medipe module, and then extracting a feature vector fnv of the face three-dimensional normal vector; meanwhile, an MTTS-CAN network architecture module is adopted to carry out rPPG signal extraction, and then an rPPG signal feature vector frPPG is extracted; and inputting the face three-dimensional normal vector feature vector fnv and the rPPG signal feature vector frPPG into a feature fusion classification model to obtain a final emotion perception result. According to the invention, the two different sources of face signals are ingeniously integrated and are applied to facial expression recognition of the face, so that the accuracy of face emotion perception is improved.

The invention discloses a facial expression recognition method based on an intelligent accompanying robot, relates to the technical field of intelligent recognition, and aims to recognize mask-type expressions (namely expressions inconsistent with real emotions) by analyzing emotional contradiction degree Qmd, effectively avoid the problem of emotion misjudgment, and improve the recognition efficiency especially when facing complex or camouflage emotions. The robot can distinguish the real state of the emotion and adjust the feedback intensity according to the information, so that the emotion demand of the accompanying object can be accurately perceived and properly responded. According to the dynamic feedback adjustment mechanism, a feedback response intensity value Fqdz is constructed, the feedback intensity is dynamically adjusted according to the emotional contradiction degree, and the interaction intensity of the robot can be accurately adjusted according to emotional states in different situations. By combining initial setting and historical data, the intelligent accompanying robot can optimize the behavior mode of the intelligent accompanying robot according to long-term interaction feedback.

The invention relates to the technical field of medical electronic equipment, and discloses a face emotion recognition method and system based on generative adversarial separation and expression exchange. The training process of the adopted facial expression recognition model comprises the following steps: pre-training a visual converter based on a self-supervised mask auto-encoder; extracting face feature vectors corresponding to the two face expression images through an image encoder in a visual converter; performing feature decoupling on the face feature vector to separate emotion-related features and emotion-unrelated features, and performing feature exchange to generate exchanged face features; fusing the face key point features with the exchange face features to generate fused features; generating a reconstructed facial expression image and a synthesized exchange facial expression image by using a generator based on the fusion features, and constraining identity consistency of the reconstructed image and the original facial expression image; a discriminator is used for discriminating the authenticity of a reconstructed image, and meanwhile, a classifier is used for performing facial emotion classification on the basis of emotion related features, so that parameters of a facial expression recognition model are optimized. The core problems of poor model generalization, low generation quality and difficulty in non-paired data training in the prior art are effectively solved, and the training cost is reduced.

The invention discloses an emotion recognition method based on a global feature and local feature fusion network, and relates to the technical field of face recognition. Comprising the following steps: constructing an emotion recognition model based on a global feature and local feature fusion network GLFNet; inputting a face image to be recognized into the emotion recognition model, extracting an intermediate feature map of the face image through the backbone network ResNet-18, and respectively sending the intermediate feature map into the global module and the local module to obtain a global feature map and a local feature map; the feature fusion module fuses the global feature map and the local feature map to obtain fused features; residual connection is established between the intermediate feature and the fusion feature to obtain a final fusion feature; and performing full connection layer FC processing on the final fusion feature to obtain an emotion recognition result. According to the method, the accuracy of a facial image emotion recognition result can be improved, richer fusion feature information is obtained, and higher performance is provided for a facial expression recognition task.

The invention discloses a multi-modal dynamic facial expression recognition method based on CLIP, and the method comprises the following steps: constructing a label enhancement module, generating positive-negative text supervision, and obtaining positive text features and negative text features; constructing a multi-modal data mining module, and mining different levels of feature information from the video; fusing the facial expression features, the audio features and the fine-grained text description features by using an adaptive fusion strategy to obtain fused feature representation; and performing cosine similarity calculation on the fused feature representation, the positive text features and the negative text features to obtain final emotion classification. According to the method, class label enhancement is introduced, class labels are converted into positive-negative text supervision, and label enhancement is performed through P-N descriptors, so that fuzzy categories which are originally difficult to distinguish can be distinguished; and the similarity between correct image-text pairs is maximized by utilizing a CLIP contrast learning mechanism, so that the classification and retrieval precision is improved.

The invention discloses a facial action unit recognition method and system based on multi-modal fusion and text enhancement. The method comprises the following steps: extracting a facial image attribute set by a visual language model, constructing an adaptive text cue word based on the facial image attribute set, and processing the adaptive text cue word by adopting CLIP to obtain a text semantic feature; extracting facial key point features by the facial key point detection model; the facial expression recognition model extracts facial expression features; the image semantic feature fusion module processes the facial key point features and the facial expression features to obtain multi-modal image semantic features; inputting the multi-modal image semantic features and the text semantic features into a multi-modal feature fusion module to obtain multi-modal fusion features after text information enhancement; the recognition of the facial action unit of the facial image is realized. According to the method, the complementarity of cross-modal features and the robustness of joint representation can be enhanced, and the recognition precision of facial actions in a complex scene is effectively improved.

The embodiment of the invention provides a model training method, a facial expression recognition method and device and electronic equipment, and relates to the technical field of video processing. The model training method comprises the following steps: acquiring a sample video and a first sample label; extracting a time feature and a space feature of the sample video by using a time-space feature extraction network in the facial expression recognition model of the initial structure; calculating an attention weight representing the correlation between the spatial feature and the time feature by using a mapping network; performing weighted aggregation on the time features by using the attention weight to obtain fused spatio-temporal features; inputting the fused spatial-temporal features into a classification network to obtain a first recognition result; and performing model training based on the difference between the first recognition result and the first sample label to obtain a trained facial expression recognition model with higher accuracy.

The present invention provides a facial expression recognition method and system based on multi-cue associative learning, belonging to the technical field of computer vision. The recognition method comprises: inputting a pre-recognized facial image into a student model and / or a teacher model for facial expression recognition. A training method comprises: cropping a global facial sample image to obtain an upper half facial sample image and a lower half facial sample image; extracting cue features; acquiring adjacency matrices corresponding to the upper half facial sample image, the lower half facial sample image and the global facial sample image; fusing associated semantics by using a feature-level attention mechanism, so as to acquire the teacher model; supervising training of the teacher model by using a cross-entropy loss; and supervising training of the student model by using label distillation, KL divergence and a cross-entropy loss.

The invention discloses a medical training evaluation method and system based on an intelligent simulation patient, and relates to the field of artificial intelligence technology and medical simulation, and the method comprises a virtual patient model construction step, a multi-modal perception step, an intelligent interaction step and an intelligent evaluation step. By integrating voice recognition, natural language understanding, facial expression recognition, gesture recognition and virtual reality technologies, a high-simulation and high-interactivity intelligent patient model is constructed for medical students or clinical medical staff to perform diagnosis and treatment training and skill operation assessment. The virtual patient model supports custom information such as age, gender, race, medical history, living habits and the like, different virtual patient images and different pathophysiological states are generated, and personalized learning requirements of users are met. In addition, targeted training suggestions are provided, the training difficulty and content can be adjusted according to the performance of the user, and different training requirements are met.

A facial expression recognition method based on grid attention and pyramid segmentation attention belongs to the technical field of deep learning image processing and emotion calculation, and comprises the following steps: introducing a grid attention mechanism and a pyramid segmentation attention mechanism on a ResNet101 large model, capturing local expression detail features through a grid attention module, establishing multi-scale global feature association by using a pyramid segmentation attention mechanism; hierarchical segmentation is carried out on the backbone network, and information from multiple hierarchies is effectively fused; and dynamically balancing the contribution degree of each level of features through learnable parameters, and integrating the extracted features to judge the expression category. Experiments show that the model achieves the recognition accuracy superior to that of a traditional model on a public data set in natural scenes such as complex illumination and posture change. According to the method, a solution with high robustness is provided for facial expression recognition in a complex environment, and the method has important application value in the fields of intelligent human-computer interaction, mental health assessment and the like.

The invention discloses a science popularization education knowledge recommendation method and system, and relates to the technical field of recommendation service, and the method comprises the following steps: building an emotion recognition mechanism, generating emotion feedback data through an algorithm, and collecting the emotion feedback data of a user in real time; establishing a self-adaptive learning path recommendation mechanism, and dynamically adjusting learning contents and paths according to the collected emotion feedback data of the user; and based on the adjusted learning path, collecting feedback data of the user in the learning process, and evaluating the validity of the learning path and the learning effect of the user. According to the emotion feedback data of the user, personalized learning paths and content recommendation can be dynamically generated, personalized requirements of different learners are met, the learning effect is improved, the emotion feedback data of the user are collected and analyzed in real time through facial expression recognition and voice emotion analysis technologies, and the user experience is improved. The learning content and path are dynamically adjusted according to the emotional state of the user, and more adaptive learning experience is provided.

This invention belongs to the field of facial expression recognition and provides a micro-expression recognition method based on convolutional neural networks and optical flow features, which is used to improve the adaptability and accuracy of micro-expression recognition. The method first constructs a face detection module to perform face detection on each image frame in a video segment, outputs facial key point information, and extracts facial regions of interest (ROIs) based on the facial key point information: nose, mouth, left eye and eyebrow, right eye and eyebrow. Then, optical flow information and optical strain information are introduced to characterize the spatiotemporal information of facial movement and the intensity information of facial deformation. The optical flow eigenvalue of each image frame is calculated, and a binary search algorithm is used to search for the image frame with the maximum optical flow eigenvalue, which is used as the vertex frame for micro-expression recognition. Finally, a micro-expression recognition model based on a convolutional neural network is constructed, which uses the horizontal and vertical components of the optical flow and optical strain of the vertex frame as input. The micro-expression recognition model outputs a micro-expression category prediction result.

The invention relates to the technical field of image processing, in particular to a mental patient auxiliary analysis method and device based on facial expression recognition, and the method comprises the steps: collecting a plurality of face images when a patient communicates with a doctor; obtaining a plurality of suspected Chinese character chuan-shaped pattern regions in the eyebrow region and a suspected Chinese character chuan-shaped pattern region sequence; acquiring a pupil area of each face image to obtain a plurality of facial expression change moments; obtaining the possibility that the expression of the patient is angry at each facial expression change moment; obtaining a plurality of anger stages of the patient; obtaining the time point when the doctor menses the keyword of the patient every time, obtaining the anger initial stage of each anger stage of the patient, and further obtaining the control ability of the patient on the own emotion in the conversation process; and the severity of the mental problem of the patient is obtained. The face change of the patient is analyzed, a doctor is assisted to judge the mental state of the patient, and the diagnosis and treatment effect is improved.

The invention discloses a high-precision facial expression emotion detection method based on improved YOLOv8n, which optimizes a facial expression recognition task while retaining a basic framework of the YOLOv8n, and comprises the following steps of: firstly, replacing a standard convolution module in a backbone network and a neck structure of the YOLOv8n with a self-adaptive multi-branch convolution module; a large-scale separable nuclear attention module is integrated to the seventh layer and the tenth layer of a YOLOv8n backbone network, dynamic attention on key facial features is enhanced, interlayer feature injection connection is applied to the YOLOv8n backbone network and a neck structure, cross-layer connection between the sixth layer and the seventeenth layer and cross-layer connection between the ninth layer and the nineteenth layer are established, and therefore the dynamic attention on the key facial features is enhanced. By establishing the direct connection between the shallow and deep features, the information loss is reduced, the multi-scale feature fusion is optimized, and the precision and efficiency of facial expression emotion detection are significantly improved.

The invention discloses a face anonymous method based on a StyleGAN encoder and a skin color matching generative adversarial network. The method comprises the following steps: converting an original face image into a semantic segmentation map by using a pre-trained face semantic segmentation model BiSeNet; utilizing a translation encoder to extract posture potential representation of the semantic segmentation map, and utilizing an identity encoder to extract identity potential representation information of the reference face image; fusing the posture potential characterization information and the identity potential characterization information to obtain a new potential characterization which keeps the original face posture and is different from the original face identity, and converting the injected new potential characterization into a preliminary anonymous face image through a StyleGAN generator; and fusing the generated anonymous face image and the background of the original image by using a skin color matching generative adversarial network to realize final face anonymity. The anonymity of the anonymous picture, the face detection performance, the image generation quality, the diversity, the facial expression recognition performance and the picture integrity are improved.

The invention provides an emotion regulation music therapy system based on biological feedback and a method thereof, and aims to realize personalized emotion regulation and psychological rehabilitation by monitoring the emotion state of a user in real time and automatically regulating music rhythm and melody. The system collects physiological and behavioral data of a user by integrating a heart rate monitor, a skin electric response sensor, a facial expression recognition camera and other biosensors. By utilizing an advanced emotion analysis engine, the system can accurately identify the emotion change of the user and recommend appropriate music from a music database containing various styles and emotion tags. In addition, the system further has a personalized learning algorithm, music recommendation can be continuously optimized according to preference and feedback of the user, and customized music experience is provided. A user can interact with the system through an intuitive user interface, adjust settings and provide feedback. The system also pays attention to privacy protection of the user data, and ensures that all the collected data accord with privacy protection laws and regulations.

The invention relates to the technical field of expression recognition, and particularly discloses an AI virtual human shopping guide system and method based on facial expression recognition, and the system comprises a face data obtaining module which is used for obtaining a face image sequence in the interaction process of a shopping guide target user and an AI virtual human; the expression information recognition module is used for recognizing facial expressions in the interaction process of the shopping guide target user and the AI virtual human based on the face image sequence to obtain real-time facial expression information of the shopping guide target user, and the real-time facial expression information comprises a real-time facial expression type of the shopping guide target user and a corresponding real-time expression index; the shopping guide information generation module is used for generating latest shopping guide information based on the real-time facial expression information of the shopping guide target user and the latest shopping guide information given by the AI virtual human; the method and the device are used for improving the interactive experience of a user and an AI virtual human, enhancing the satisfaction degree and the acceptability of the user for shopping guide services, and improving the sales conversion rate and the service quality.

The invention discloses a human-computer interaction method based on expression recognition, an interactive intelligent toy and a robot. The man-machine interaction method comprises the following steps: preprocessing a user face image collected in real time, and determining a face effective area; preset key point information in the face effective area is extracted; determining facial features based on preset key point information, and analyzing the facial features based on the constructed classification model to obtain user expression information; determining an expression intensity level based on a preset expression library and the user expression information; if the expression intensity grade is smaller than a preset threshold value, triggering a first strategy to carry out multi-target grid deformation adjustment on the to-be-deformed target object; if the expression intensity grade is not smaller than the preset threshold value, a second strategy is triggered to conduct three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the target object obtained after deformation adjustment is matched with the expression of the user. According to the method, the intelligent degree of the target body can be effectively improved, and intelligent, rich and personalized interaction experience is provided for a user.

In a facial expression recognition method, facial key points are identified as graph nodes. A facial graph structure for a first image is constructed with edges between pairs of the graph nodes based on relationships between the facial key points corresponding to the graph nodes. A first feature of a facial texture is extracted from color information of pixels in the first image. A second feature of the first image is extracted by processing the facial graph structure using a graph neural network (GNN). The first feature and the second feature are combined, to obtain a fused feature. A first expression type of a face in the first image that corresponds to the fused feature is determined. The first expression type is determined from a plurality of facial expression types.

The invention discloses a man-machine interaction method and device, equipment and a medium, and relates to the technical field of natural language processing, and the method comprises the steps: responding to a dialogue instruction triggered by a user, and collecting the voice of the user and an image of the user; the method comprises the following steps: analyzing voice of a user through a voice emotion analysis technology to obtain a first result, recognizing an image of the user through a facial expression recognition technology to obtain a second result, and obtaining an emotion state of the user according to the first result and the second result; judging whether the intention of the user is matched with the intention in the candidate intention set or not according to the voice of the user and the emotional state of the user; if the intention of the user is matched with the first intention in the candidate intention set, reply content corresponding to the first intention is replied to the user, and the method can improve the accuracy of a machine to understand user requirements, so that the accuracy of reply of the machine to the user is improved, and the interaction experience of the user is improved.

The invention provides a facial expression recognition optimization method and device based on deep learning and electronic equipment. The method comprises the following steps: acquiring facial expression data, and constructing a facial three-dimensional model by adopting a 3D facial reconstruction technology according to the facial expression data; extracting facial features from the facial three-dimensional model by adopting an SIFT algorithm, determining expression tags corresponding to the facial features, constructing a deep learning model, and training the deep learning model by adopting the facial features and the expression tags as training samples to obtain a facial expression recognition model; and at least adopting the facial expression recognition model to recognize the facial expression to be recognized, and generating a recognition result. By integrating 3D face reconstruction and SIFT algorithm technologies, the accuracy, robustness, fine expression recognition capability, dynamic expression processing capability and anomaly detection and processing capability of facial expression recognition are improved. The problem that an existing facial feature extraction algorithm is limited in fine expression recognition capacity is solved.

The invention discloses a sign language synthesis service method based on multi-modality, and the method comprises the following steps: S10, carrying out end-cloud collaborative rendering setting: deploying an edge end as a lightweight model to generate a basic action, and deploying a cloud end as operating a MoMask and 3D rendering engine; s20, performing multi-modal data acquisition: acquiring a voice signal through a microphone, acquiring a 48 * 48 pixel face grayscale image through a camera, and acquiring action posture data through an IMU sensor; s30, performing emotion feature extraction on the collected multi-modal data: performing voice emotion recognition to output six types of emotion probability distributions, and performing facial expression recognition to output seven types of expression probability distributions; s40, performing cross-modal feature fusion: dynamically fusing the voice and facial features based on a confidence weighting strategy, and generating a three-dimensional emotion intensity vector; and S50, sign language action generation is carried out, and a 3D skeleton sequence matched with emotion is generated through RVQ layering quantification and MoMask Transformer.

The invention discloses a facial expression recognition and simulation algorithm for a virtual character or a robot, and belongs to the field of image recognition and simulation, and the algorithm mainly comprises the steps: inputting a cut face image into a depth model, recognizing face key points in the face image through the depth model, and carrying out the recognition and simulation of the face key points; and obtaining a head pose parameter, and predicting a value of a facial expression according to the facial key points. Calculating the head pose of the robot: calculating a rotation matrix through the head pose parameters, and converting the rotation matrix into an Euler angle of the head pose of the robot; mapping the predicted facial expression value into a rotation angle of a robot facial control steering engine; therefore, the robot is controlled to operate. According to the method, the micro-expression robot expression simulation data with relatively small dynamic amplitude can be obtained, a large amount of robot identification simulation data can be conveniently and accurately mapped into a robot entity from a large amount of video expressions, and the overall head pose is linked, so that the expression expression is more accurate.

The invention relates to a dynamic facial expression recognition method for modeling a complex space-time relationship, and belongs to the field of computer vision. The method comprises the steps of dividing facial expression data into a training set and a verification set, and performing data preprocessing; constructing a multi-view space-time network; and mapping expression spatio-temporal features captured by the multi-view spatio-temporal network to a label system of basic expressions through a full connection layer and a softmax scoring function to complete recognition and classification of facial expressions. According to the method, key spatial features of expressions can be concerned from global and local perspectives, and time sequence features of the expressions are captured from short-term and long-term perspectives, so that a complex space-time relationship generated by individual differences of the expressions is jointly modeled, and the accuracy of a facial expression recognition task is improved.

The invention discloses an autism diagnosis method for performing cross-domain facial expression recognition based on clustering. The method comprises the following steps: S1 to S4 are source domain model training parts; s1, acquiring a source domain facial expression image; s2, using a large model CLIP to initialize a pre-training model, and preprocessing an input face image; s3, initializing deep learning training conditions, freezing large model parameters, and only training Adapter and a classification module; s4, after the model is trained, facial expression images can be classified, and a trained source domain model weight is also obtained; s5-S10 are target domain model training and autism diagnosis parts. S5, inputting a face video (image sequence) of each child; s6, using the trained source domain model weight to initialize the model, and preprocessing a non-label face image; s7, training a deep learning model, freezing large model parameters, and only training Adapter and a classification module; and S8, according to expressions and facial features classified by the model, domain self-adaption is carried out through a clustering-based dynamic pseudo tag generation module. S9, classifying a facial expression sequence by using the model subjected to domain self-adaption, and performing autism classification by using a long-short-term memory network; and S10, obtaining a diagnosis result.

The invention provides a classroom student concentration degree monitoring system and method based on computer vision. The system comprises a vision collection module, an edge calculation unit and a cloud analysis platform. A multi-dimensional sensing mode combining head posture analysis, eye state tracking, facial expression recognition and body movement detection is adopted to evaluate the concentration degree, through a dynamic feature fusion mechanism, the system can intelligently adjust the weight ratio of different modal features, the limitation of single-dimensional analysis in a traditional method is effectively solved, and the accuracy of the system is improved. The system adopts edge calculation and cloud analysis, ensures high efficiency of data processing, ensures reliability of complex algorithm operation, and has important value for improving classroom teaching quality and learning efficiency. According to the invention, through multi-modal visual analysis, behavior characteristics and physiological reactions of students in class are monitored in real time, the concentration level of the students is comprehensively evaluated, and objective teaching feedback and improvement basis are provided for teachers.

Embodiments of the present disclosure disclose a dynamic facial expression recognition method, a device, and product. The method includes: segmenting a video to be recognized into video segments; extracting and aggregating, for each video segment, features of face image frames in the video segment by using a 3D convolutional network, to obtain target feature data of the video segment; aggregating target feature data of the plurality of video segments, to obtain target feature data of the video to be recognized; and determining an expression label of the video to be recognized based on the target feature data of the video to be recognized, to obtain an expression label recognition result of the video to be recognized.

The invention provides a facial expression recognition system and method based on a multi-dimensional mixed attention mechanism, and relates to the technical field of facial data processing, and the method comprises the steps: obtaining a human face image; inputting the preprocessed facial image into a facial expression recognition model, and outputting to obtain a facial expression category; wherein the facial expression recognition model is an enhanced MobileNet V3 neural network based on a feature pyramid and a mixed attention mechanism, after a facial image is subjected to depth separable convolution operation, multi-scale features of the facial image are extracted through the feature pyramid network, high-level semantic features and low-level detail features are obtained and fused, and the facial expression recognition model is obtained. And through a mixed attention mechanism of frequency domain and time domain fusion and space channel fusion, guiding a model to pay attention to effective features in an image by analyzing energy distribution in a frequency domain and extracting space information and channel information, obtaining a final feature vector, and realizing facial expression classification based on the final feature vector.