306 results about "Facial expression recognition" patented technology

The invention discloses a facial expression recognition method fusing spatial channel features and fast regional convolution. The method comprises a fast regional convolution module, a spatial channel feature fusion module and a multi-scale attention module. The method comprises the following steps: firstly, extracting multi-scale local features by using depth separable convolution, and constructing a feature extractor by stacking a fast regional convolution module; then, on the basis of a space channel feature fusion module, feature fusion is realized by using deep convolution and a cross-channel self-attention mechanism, so that the calculation complexity and the memory requirement are remarkably reduced; finally, through a multi-scale attention module, feature extraction and spatial information aggregation of different scales are utilized, and the feature representation capacity of the expression change key area is enhanced. According to the invention, a lightweight facial expression recognition network is designed by using respective advantages of a full convolution model and a fast regional convolution model. A large number of experiments show that compared with other methods, the method provided by the invention has better performance, and meanwhile, the calculation cost and the network scale are far smaller than those of the same type of methods.

The invention relates to the technical field of medical electronic equipment, and discloses a face emotion recognition method and system based on generative adversarial separation and expression exchange. The training process of the adopted facial expression recognition model comprises the following steps: pre-training a visual converter based on a self-supervised mask auto-encoder; extracting face feature vectors corresponding to the two face expression images through an image encoder in a visual converter; performing feature decoupling on the face feature vector to separate emotion-related features and emotion-unrelated features, and performing feature exchange to generate exchanged face features; fusing the face key point features with the exchange face features to generate fused features; generating a reconstructed facial expression image and a synthesized exchange facial expression image by using a generator based on the fusion features, and constraining identity consistency of the reconstructed image and the original facial expression image; a discriminator is used for discriminating the authenticity of a reconstructed image, and meanwhile, a classifier is used for performing facial emotion classification on the basis of emotion related features, so that parameters of a facial expression recognition model are optimized. The core problems of poor model generalization, low generation quality and difficulty in non-paired data training in the prior art are effectively solved, and the training cost is reduced.

The invention discloses a multi-modal dynamic facial expression recognition method based on CLIP, and the method comprises the following steps: constructing a label enhancement module, generating positive-negative text supervision, and obtaining positive text features and negative text features; constructing a multi-modal data mining module, and mining different levels of feature information from the video; fusing the facial expression features, the audio features and the fine-grained text description features by using an adaptive fusion strategy to obtain fused feature representation; and performing cosine similarity calculation on the fused feature representation, the positive text features and the negative text features to obtain final emotion classification. According to the method, class label enhancement is introduced, class labels are converted into positive-negative text supervision, and label enhancement is performed through P-N descriptors, so that fuzzy categories which are originally difficult to distinguish can be distinguished; and the similarity between correct image-text pairs is maximized by utilizing a CLIP contrast learning mechanism, so that the classification and retrieval precision is improved.

The invention discloses a facial action unit recognition method and system based on multi-modal fusion and text enhancement. The method comprises the following steps: extracting a facial image attribute set by a visual language model, constructing an adaptive text cue word based on the facial image attribute set, and processing the adaptive text cue word by adopting CLIP to obtain a text semantic feature; extracting facial key point features by the facial key point detection model; the facial expression recognition model extracts facial expression features; the image semantic feature fusion module processes the facial key point features and the facial expression features to obtain multi-modal image semantic features; inputting the multi-modal image semantic features and the text semantic features into a multi-modal feature fusion module to obtain multi-modal fusion features after text information enhancement; the recognition of the facial action unit of the facial image is realized. According to the method, the complementarity of cross-modal features and the robustness of joint representation can be enhanced, and the recognition precision of facial actions in a complex scene is effectively improved.

The embodiment of the invention provides a model training method, a facial expression recognition method and device and electronic equipment, and relates to the technical field of video processing. The model training method comprises the following steps: acquiring a sample video and a first sample label; extracting a time feature and a space feature of the sample video by using a time-space feature extraction network in the facial expression recognition model of the initial structure; calculating an attention weight representing the correlation between the spatial feature and the time feature by using a mapping network; performing weighted aggregation on the time features by using the attention weight to obtain fused spatio-temporal features; inputting the fused spatial-temporal features into a classification network to obtain a first recognition result; and performing model training based on the difference between the first recognition result and the first sample label to obtain a trained facial expression recognition model with higher accuracy.

The present invention provides a facial expression recognition method and system based on multi-cue associative learning, belonging to the technical field of computer vision. The recognition method comprises: inputting a pre-recognized facial image into a student model and / or a teacher model for facial expression recognition. A training method comprises: cropping a global facial sample image to obtain an upper half facial sample image and a lower half facial sample image; extracting cue features; acquiring adjacency matrices corresponding to the upper half facial sample image, the lower half facial sample image and the global facial sample image; fusing associated semantics by using a feature-level attention mechanism, so as to acquire the teacher model; supervising training of the teacher model by using a cross-entropy loss; and supervising training of the student model by using label distillation, KL divergence and a cross-entropy loss.

The invention discloses a medical training evaluation method and system based on an intelligent simulation patient, and relates to the field of artificial intelligence technology and medical simulation, and the method comprises a virtual patient model construction step, a multi-modal perception step, an intelligent interaction step and an intelligent evaluation step. By integrating voice recognition, natural language understanding, facial expression recognition, gesture recognition and virtual reality technologies, a high-simulation and high-interactivity intelligent patient model is constructed for medical students or clinical medical staff to perform diagnosis and treatment training and skill operation assessment. The virtual patient model supports custom information such as age, gender, race, medical history, living habits and the like, different virtual patient images and different pathophysiological states are generated, and personalized learning requirements of users are met. In addition, targeted training suggestions are provided, the training difficulty and content can be adjusted according to the performance of the user, and different training requirements are met.

A facial expression recognition method based on grid attention and pyramid segmentation attention belongs to the technical field of deep learning image processing and emotion calculation, and comprises the following steps: introducing a grid attention mechanism and a pyramid segmentation attention mechanism on a ResNet101 large model, capturing local expression detail features through a grid attention module, establishing multi-scale global feature association by using a pyramid segmentation attention mechanism; hierarchical segmentation is carried out on the backbone network, and information from multiple hierarchies is effectively fused; and dynamically balancing the contribution degree of each level of features through learnable parameters, and integrating the extracted features to judge the expression category. Experiments show that the model achieves the recognition accuracy superior to that of a traditional model on a public data set in natural scenes such as complex illumination and posture change. According to the method, a solution with high robustness is provided for facial expression recognition in a complex environment, and the method has important application value in the fields of intelligent human-computer interaction, mental health assessment and the like.

This invention belongs to the field of facial expression recognition and provides a micro-expression recognition method based on convolutional neural networks and optical flow features, which is used to improve the adaptability and accuracy of micro-expression recognition. The method first constructs a face detection module to perform face detection on each image frame in a video segment, outputs facial key point information, and extracts facial regions of interest (ROIs) based on the facial key point information: nose, mouth, left eye and eyebrow, right eye and eyebrow. Then, optical flow information and optical strain information are introduced to characterize the spatiotemporal information of facial movement and the intensity information of facial deformation. The optical flow eigenvalue of each image frame is calculated, and a binary search algorithm is used to search for the image frame with the maximum optical flow eigenvalue, which is used as the vertex frame for micro-expression recognition. Finally, a micro-expression recognition model based on a convolutional neural network is constructed, which uses the horizontal and vertical components of the optical flow and optical strain of the vertex frame as input. The micro-expression recognition model outputs a micro-expression category prediction result.

The invention discloses a high-precision facial expression emotion detection method based on improved YOLOv8n, which optimizes a facial expression recognition task while retaining a basic framework of the YOLOv8n, and comprises the following steps of: firstly, replacing a standard convolution module in a backbone network and a neck structure of the YOLOv8n with a self-adaptive multi-branch convolution module; a large-scale separable nuclear attention module is integrated to the seventh layer and the tenth layer of a YOLOv8n backbone network, dynamic attention on key facial features is enhanced, interlayer feature injection connection is applied to the YOLOv8n backbone network and a neck structure, cross-layer connection between the sixth layer and the seventeenth layer and cross-layer connection between the ninth layer and the nineteenth layer are established, and therefore the dynamic attention on the key facial features is enhanced. By establishing the direct connection between the shallow and deep features, the information loss is reduced, the multi-scale feature fusion is optimized, and the precision and efficiency of facial expression emotion detection are significantly improved.

The invention discloses a face anonymous method based on a StyleGAN encoder and a skin color matching generative adversarial network. The method comprises the following steps: converting an original face image into a semantic segmentation map by using a pre-trained face semantic segmentation model BiSeNet; utilizing a translation encoder to extract posture potential representation of the semantic segmentation map, and utilizing an identity encoder to extract identity potential representation information of the reference face image; fusing the posture potential characterization information and the identity potential characterization information to obtain a new potential characterization which keeps the original face posture and is different from the original face identity, and converting the injected new potential characterization into a preliminary anonymous face image through a StyleGAN generator; and fusing the generated anonymous face image and the background of the original image by using a skin color matching generative adversarial network to realize final face anonymity. The anonymity of the anonymous picture, the face detection performance, the image generation quality, the diversity, the facial expression recognition performance and the picture integrity are improved.

The invention provides an emotion regulation music therapy system based on biological feedback and a method thereof, and aims to realize personalized emotion regulation and psychological rehabilitation by monitoring the emotion state of a user in real time and automatically regulating music rhythm and melody. The system collects physiological and behavioral data of a user by integrating a heart rate monitor, a skin electric response sensor, a facial expression recognition camera and other biosensors. By utilizing an advanced emotion analysis engine, the system can accurately identify the emotion change of the user and recommend appropriate music from a music database containing various styles and emotion tags. In addition, the system further has a personalized learning algorithm, music recommendation can be continuously optimized according to preference and feedback of the user, and customized music experience is provided. A user can interact with the system through an intuitive user interface, adjust settings and provide feedback. The system also pays attention to privacy protection of the user data, and ensures that all the collected data accord with privacy protection laws and regulations.

The invention discloses a human-computer interaction method based on expression recognition, an interactive intelligent toy and a robot. The man-machine interaction method comprises the following steps: preprocessing a user face image collected in real time, and determining a face effective area; preset key point information in the face effective area is extracted; determining facial features based on preset key point information, and analyzing the facial features based on the constructed classification model to obtain user expression information; determining an expression intensity level based on a preset expression library and the user expression information; if the expression intensity grade is smaller than a preset threshold value, triggering a first strategy to carry out multi-target grid deformation adjustment on the to-be-deformed target object; if the expression intensity grade is not smaller than the preset threshold value, a second strategy is triggered to conduct three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the target object obtained after deformation adjustment is matched with the expression of the user. According to the method, the intelligent degree of the target body can be effectively improved, and intelligent, rich and personalized interaction experience is provided for a user.

In a facial expression recognition method, facial key points are identified as graph nodes. A facial graph structure for a first image is constructed with edges between pairs of the graph nodes based on relationships between the facial key points corresponding to the graph nodes. A first feature of a facial texture is extracted from color information of pixels in the first image. A second feature of the first image is extracted by processing the facial graph structure using a graph neural network (GNN). The first feature and the second feature are combined, to obtain a fused feature. A first expression type of a face in the first image that corresponds to the fused feature is determined. The first expression type is determined from a plurality of facial expression types.

The invention discloses a man-machine interaction method and device, equipment and a medium, and relates to the technical field of natural language processing, and the method comprises the steps: responding to a dialogue instruction triggered by a user, and collecting the voice of the user and an image of the user; the method comprises the following steps: analyzing voice of a user through a voice emotion analysis technology to obtain a first result, recognizing an image of the user through a facial expression recognition technology to obtain a second result, and obtaining an emotion state of the user according to the first result and the second result; judging whether the intention of the user is matched with the intention in the candidate intention set or not according to the voice of the user and the emotional state of the user; if the intention of the user is matched with the first intention in the candidate intention set, reply content corresponding to the first intention is replied to the user, and the method can improve the accuracy of a machine to understand user requirements, so that the accuracy of reply of the machine to the user is improved, and the interaction experience of the user is improved.

The invention provides a facial expression recognition optimization method and device based on deep learning and electronic equipment. The method comprises the following steps: acquiring facial expression data, and constructing a facial three-dimensional model by adopting a 3D facial reconstruction technology according to the facial expression data; extracting facial features from the facial three-dimensional model by adopting an SIFT algorithm, determining expression tags corresponding to the facial features, constructing a deep learning model, and training the deep learning model by adopting the facial features and the expression tags as training samples to obtain a facial expression recognition model; and at least adopting the facial expression recognition model to recognize the facial expression to be recognized, and generating a recognition result. By integrating 3D face reconstruction and SIFT algorithm technologies, the accuracy, robustness, fine expression recognition capability, dynamic expression processing capability and anomaly detection and processing capability of facial expression recognition are improved. The problem that an existing facial feature extraction algorithm is limited in fine expression recognition capacity is solved.

The invention discloses a sign language synthesis service method based on multi-modality, and the method comprises the following steps: S10, carrying out end-cloud collaborative rendering setting: deploying an edge end as a lightweight model to generate a basic action, and deploying a cloud end as operating a MoMask and 3D rendering engine; s20, performing multi-modal data acquisition: acquiring a voice signal through a microphone, acquiring a 48 * 48 pixel face grayscale image through a camera, and acquiring action posture data through an IMU sensor; s30, performing emotion feature extraction on the collected multi-modal data: performing voice emotion recognition to output six types of emotion probability distributions, and performing facial expression recognition to output seven types of expression probability distributions; s40, performing cross-modal feature fusion: dynamically fusing the voice and facial features based on a confidence weighting strategy, and generating a three-dimensional emotion intensity vector; and S50, sign language action generation is carried out, and a 3D skeleton sequence matched with emotion is generated through RVQ layering quantification and MoMask Transformer.

The invention discloses an autism diagnosis method for performing cross-domain facial expression recognition based on clustering. The method comprises the following steps: S1 to S4 are source domain model training parts; s1, acquiring a source domain facial expression image; s2, using a large model CLIP to initialize a pre-training model, and preprocessing an input face image; s3, initializing deep learning training conditions, freezing large model parameters, and only training Adapter and a classification module; s4, after the model is trained, facial expression images can be classified, and a trained source domain model weight is also obtained; s5-S10 are target domain model training and autism diagnosis parts. S5, inputting a face video (image sequence) of each child; s6, using the trained source domain model weight to initialize the model, and preprocessing a non-label face image; s7, training a deep learning model, freezing large model parameters, and only training Adapter and a classification module; and S8, according to expressions and facial features classified by the model, domain self-adaption is carried out through a clustering-based dynamic pseudo tag generation module. S9, classifying a facial expression sequence by using the model subjected to domain self-adaption, and performing autism classification by using a long-short-term memory network; and S10, obtaining a diagnosis result.

The invention provides a classroom student concentration degree monitoring system and method based on computer vision. The system comprises a vision collection module, an edge calculation unit and a cloud analysis platform. A multi-dimensional sensing mode combining head posture analysis, eye state tracking, facial expression recognition and body movement detection is adopted to evaluate the concentration degree, through a dynamic feature fusion mechanism, the system can intelligently adjust the weight ratio of different modal features, the limitation of single-dimensional analysis in a traditional method is effectively solved, and the accuracy of the system is improved. The system adopts edge calculation and cloud analysis, ensures high efficiency of data processing, ensures reliability of complex algorithm operation, and has important value for improving classroom teaching quality and learning efficiency. According to the invention, through multi-modal visual analysis, behavior characteristics and physiological reactions of students in class are monitored in real time, the concentration level of the students is comprehensively evaluated, and objective teaching feedback and improvement basis are provided for teachers.

Embodiments of the present disclosure disclose a dynamic facial expression recognition method, a device, and product. The method includes: segmenting a video to be recognized into video segments; extracting and aggregating, for each video segment, features of face image frames in the video segment by using a 3D convolutional network, to obtain target feature data of the video segment; aggregating target feature data of the plurality of video segments, to obtain target feature data of the video to be recognized; and determining an expression label of the video to be recognized based on the target feature data of the video to be recognized, to obtain an expression label recognition result of the video to be recognized.

The invention provides a facial expression recognition system and method based on a multi-dimensional mixed attention mechanism, and relates to the technical field of facial data processing, and the method comprises the steps: obtaining a human face image; inputting the preprocessed facial image into a facial expression recognition model, and outputting to obtain a facial expression category; wherein the facial expression recognition model is an enhanced MobileNet V3 neural network based on a feature pyramid and a mixed attention mechanism, after a facial image is subjected to depth separable convolution operation, multi-scale features of the facial image are extracted through the feature pyramid network, high-level semantic features and low-level detail features are obtained and fused, and the facial expression recognition model is obtained. And through a mixed attention mechanism of frequency domain and time domain fusion and space channel fusion, guiding a model to pay attention to effective features in an image by analyzing energy distribution in a frequency domain and extracting space information and channel information, obtaining a final feature vector, and realizing facial expression classification based on the final feature vector.

The invention relates to a method for analyzing user emotion experience based on facial expressions and a subjective emotion scale. Specifically, the method comprises the following steps: firstly, capturing and recording a face image of a user in a product (or interaction) using process in real time by using a camera, and continuously detecting a human face in the face image in real time by using a RetinaFace method; then, for the human face detected in each frame, various expression intensities, including surprising, afraid, dislike, happy, sadness, vigor and neutral expressions, are calculated by using a ResmaskNet facial expression recognition method; and finally, combining objective facial expressions of the user and a subjective emotion scale to comprehensively analyze the emotion experience of the user in a product using or interaction process. The facial expression of the user is captured in real time through the camera, the cost is low, the experience of the user in the interaction process is not affected, and the method is suitable for evaluating the user experience in most interaction scenes.

The invention belongs to image processing and deep learning in the field of artificial intelligence, and particularly relates to a facial expression recognition method which is realized by utilizing a deep learning algorithm and has high transparency and interpretability and application of the facial expression recognition method. The objective of the invention is to solve the problems that an existing facial expression and emotion recognition technology is low in accuracy in a complex environment, and a model is opaque and does not have interpretability. The method comprises the following steps: firstly, acquiring a facial video stream through a camera or other equipment, enhancing data to improve the robustness and generalization ability of a model, then, carrying out facial detection by using an MTCNN model to obtain facial bounding box coordinates and key point information, then, carrying out expression classification by using an Xception and YOLOv5-cls double deep learning model, and finally, carrying out facial expression classification by using a facial expression classification model. And generating a saliency map through an RISE algorithm. According to the method, the robustness and generalization ability of recognition are improved through learning of the double deep learning models, the recognition accuracy is remarkably improved, the method is particularly excellent in performance in a complex environment, the significance graph is generated through the RISE algorithm, the decision process of the models is explained, and the transparency and the user trust degree of the models are improved.

The invention discloses a multi-mode mental health assessment system based on artificial intelligence, and belongs to the technical field of mental health assessment. A multi-modal mental health assessment system based on artificial intelligence comprises a data acquisition module, a multi-modal fusion module and a health assessment module. The psychological health assessment method and device solve the problem that in the prior art, in the process of collecting user information, the collection dimension is single, so that the user information is not comprehensive, and then the accuracy of psychological health assessment is affected, and the accuracy of psychological health assessment is improved by integrating multi-source data such as facial expressions, voices, texts and videos. The method effectively improves the objectivity, accuracy and personalized intervention capability of evaluation, greatly improves the overall evaluation accuracy through the fusion of facial expression recognition, voice pressure signal detection, text emotion feature recognition and physiological parameter monitoring technologies, effectively reduces the single-modal error through multi-modal data complementation, enables the user information to be more comprehensive, and improves the user experience. And the accuracy of mental health assessment is improved.

The invention provides a facial expression recognition method based on multi-module collaborative optimization, which has an efficient network structure and can realize high-precision and strong-robustness facial expression recognition in a complex environment. The method comprises the following steps: receiving an RGB face image and preprocessing the RGB face image to obtain a preprocessed image; inputting the preprocessed image into a mixed feature network module, and outputting to obtain a first feature map; inputting the first feature map into an efficient local attention mechanism module, and outputting to obtain a second feature map; and inputting the second feature map into a classifier module to realize facial expression recognition.

The invention relates to the technical field of expression recognition, in particular to a virtual digital human expression recognition method and system based on multi-scale expansion convolution. The method comprises the following steps: preprocessing acquired image data; constructing a neural network model based on multi-scale expansion convolution based on the preprocessed data, and training the neural network model by using the preprocessed image data; applying the trained model to image processing, and outputting an expression recognition result; and driving the virtual digital human to generate an expression animation according to an identification result. According to the method, efficient extraction and fusion of multi-level expression features are realized through a multi-scale expansion convolution fusion attention module (MDFA) and a coordinate attention collaboration mechanism, and through global average pooling and a 1 * 1 convolution dimension reduction strategy, the recognition accuracy of the surprising expression with low-resolution input is improved by 7.8%.

The invention relates to the technical field of detection, in particular to a self-supervision-based lightweight dynamic facial expression detection method and system, and the method comprises the steps: obtaining a training image from a video set with a facial image, and constructing and training a self-supervision neural network model based on the training image, thereby achieving the detection of the facial expression. The facial expression recognition method comprises the steps of obtaining a facial recognition self-supervised neural network model, processing a dynamic image, and inputting the processed dynamic image into the facial recognition self-supervised neural network model, thereby outputting a facial expression recognition result, and realizing the recognition and detection of the dynamic facial expression.

The invention provides a GAN-based vehicle-mounted expression data construction method, system and device and a vehicle, and relates to the technical field of intelligent driving. A key point detection technology is utilized to analyze and extract a facial action unit and expression intensity and duration thereof to generate a structured action description. And then, calling a generative adversarial network model to generate a synthetic video corresponding to the target expression type, and carrying out dynamic cutting and frame extraction according to a preset interval on the videos to obtain a multi-frame synthetic image. And finally, performing dual-model filtering on the synthesized image to ensure data quality, and executing privacy anonymization processing, thereby generating a high-quality vehicle-mounted expression data set. According to the method, the data acquisition cost is reduced, the diversity and accuracy of a data set are improved, and the performance and reliability of an expression recognition algorithm in an actual vehicle-mounted environment are effectively enhanced.

The invention discloses a multi-modal cross-domain small sample facial expression recognition method based on relational distillation self-paced learning, and relates to a computer vision technology. Constructing a multi-modal semantic enhancement module, generating semantic descriptions of expressions by using a large language model, performing CLIP coding, and performing alignment and fusion with image visual features in the multi-modal semantic enhancement module to construct a multi-modal prototype; a self-paced learning mechanism based on relational distillation is designed, visual and semantic structural errors are calculated, and progressive training from easy to difficult is realized through a soft and hard mixed sample selection strategy and a mixed sample selection mechanism regulated and controlled by a dynamic threshold value. And cross-domain migration of emotional knowledge from basic expressions to fine-grained composite expressions can be effectively realized. And under the condition that only a small number of labeled samples are provided, rapid adaptation and accurate recognition of new expressions can be realized. The method is remarkably superior to a traditional supervised learning method, has higher practicability and expansibility, and can better meet the requirement for efficient recognition of new expressions in practical application.

The invention discloses a facial expression recognition method and system based on a noise robust graph model, and the method comprises the steps: inputting a to-be-recognized facial image into the noise robust graph model, and carrying out the recognition of a facial expression, and the noise robust graph model comprises a feature extraction and pre-recognition network model, which is used for extracting the depth feature of a sample image and an expression pre-recognition tag; the adjacency matrix initialization model is used for constructing an initialized adjacency matrix; the adjacency matrix regularization model is used for regularizing the initialized adjacency matrix and outputting a regularized adjacency matrix; the classification network model is used for outputting an expression identification tag; and calculating a cross entropy loss function according to the expression pre-identification tag, the initialized adjacency matrix, the regularized adjacency matrix and the expression identification tag, and performing supervision training on the noise robust graph model by using the cross entropy loss function. According to the method, the feature similarity and learning difficulty difference between samples are concerned at the same time, and the expression recognition problem in a natural scene containing difficult samples and noise labels can be solved.