88 results about "Facial motion" patented technology

The invention belongs to the technical field of man-machine interaction and robot control, and particularly relates to a man-machine physical twin system based on multi-modal video analysis and adaptive mapping and a control method, and the system comprises a multi-modal collection unit which is used for collecting original video streams of human body actions, expressions and voices, and compensating and repairing dynamic shielding; the semantic analysis unit comprises an action analysis module used for extracting a human skeleton key point set object interaction track from the original video stream, and an expression / voice analysis module used for outputting a facial action unit coefficient and a voice text with an emotion label; the self-adaptive mapping unit is used for establishing human body-robot joint kinematics mapping, distributing control bandwidth in real time according to task types and degrading non-key modes based on network states to guarantee action continuity, and non-contact action capture is achieved through a pure vision scheme by eliminating dependence on a wearable sensor; and the problems of failure and the like of a traditional visual scheme under shielding and illumination variation are solved.

The invention provides a facial action unit recognition and model training method and device and electronic equipment. The method comprises the steps of obtaining a training sample set and inputting the training sample set into an initial facial action unit recognition model; wherein the training sample set comprises multiple frames and face action unit (AU) real labels corresponding to each frame; performing multi-level feature extraction on each frame by the facial action unit recognition model, generating a visual token according to the extracted features, performing semantic reasoning based on the visual token and a prompt text of a facial action unit recognition task, and generating an AU prediction probability corresponding to each frame; determining a loss value of the loss function according to the AU real label and the AU prediction probability, and reversely adjusting model parameters of the facial action unit recognition model based on the loss value; when a preset training condition is met, training is stopped, and a trained facial action unit recognition model is obtained. According to the invention, semantic reasoning can be carried out by using visual features to complete AU identification, and the accuracy of a facial action unit detection result can be improved.

The invention relates to the technical field of face recognition, and discloses a multi-modal alignment training-based face sentiment analysis method and system, and the method comprises the steps: obtaining multi-source data, carrying out the preprocessing of the multi-source data, obtaining a training set, constructing a basic model for the face sentiment analysis, selecting a sample with an inference text to generate a small number of high-quality sentiment analysis samples, and carrying out the recognition of the high-quality sentiment analysis samples; supervising and finely adjusting the model; selecting a sample with a facial action unit label and an emotion label, and performing reinforcement learning training on the model in combination with the predicted accuracy of the facial action unit label, the emotion label and the reasoning text; and training the basic model by using the training set, expanding the original training set by using the output of the trained basic model to obtain a new training set, continuously training the model, stopping training until a preset condition is met to obtain a final basic model, and performing facial sentiment analysis by using the final basic model. According to the method, illusion can be controlled, the accuracy of a facial emotion analysis result is improved, and the data set construction cost is reduced.

The invention belongs to the technical field of image restoration and computer vision, and particularly relates to a multi-mode face restoration and expression recognition system and method based on semantic guidance of a facial action unit. According to the system, multi-scale features are extracted through visual coding, and an AU activation probability is detected by using a graph neural network; the semantic conversion module converts the numerical probability into an interpretable biomechanical structured text; the multi-modal reasoning module fuses vision and text information, introduces the common sense reasoning ability of a multi-modal large language model, and improves the student network performance through knowledge distillation; and finally, the conditional generation module realizes image restoration by taking the semantic features as guidance. The facial action unit is used as a biomechanical medium, the multi-modal reasoning ability is converted into restoration constraint, the defects that in the prior art, restoration of physiology is distorted, recognition depends on image quality, and two tasks are isolated are overcome, collaborative enhancement of face restoration and expression recognition is achieved, and it is ensured that the restoration result is clear in vision and conforms to the physiological law of facial muscles.

The invention discloses a multi-modal behavior data processing system, which comprises a data acquisition module, a feature extraction layer, a dynamic attention weight layer, a multi-modal fusion layer and a downstream task decision-making layer, the data acquisition module guides human-computer interaction through an international neurological and mental interview tool matched with a DSM-5 standard and acquires audio and video stream data; the feature extraction layer extracts a video feature vector (including facial action unit activation intensity and the like), an audio feature vector (including Mel frequency cepstrum coefficient and the like) and a text feature vector (generated by a deep language model after automatic speech recognition transcription) in parallel; the dynamic attention weight layer is combined with data quality, symptomatic priori knowledge and cross-modal correlation to generate a dynamic fusion weight; weighting, splicing and dimensionality reduction are carried out on the multi-modal fusion layer to obtain a fusion feature vector; and the downstream task decision-making layer completes evaluation and generates a multi-modal behavioral index evaluation report. The system is deployed in a non-intrusive manner, the risk is controllable, and the evaluation robustness and accuracy can be improved.

For each avatar, testing images are rendered for different facial expressions that each have ground truth facial action units. An instance of a machine learning model is applied to the testing images to generate predicted facial action units for each testing image. A predictive performance of the instance is calculated for each avatar based on the predicted and ground truth facial action units for the testing images of the avatar. A first set of features common to the avatars for which the predictive performance was better than a first threshold, and a second set of features common to the avatars for which the predictive performance was worse than a second threshold, are identified. The features present only in the second set are identified, as difference features. New avatars having the difference features are generated.WO

The application discloses a driver fatigue behavior detection method and system based on video posture invariance, and relates to the technical field of computer vision. The application proposes a key frame selection model based on facial geometric information and a head and face action information fusion space-time network. First, the driver video captured by the vehicle-mounted camera is subjected to sequential processing, and image preprocessing is performed. Then, the key frame selection model based on facial geometric information is constructed based on the geometric features of the facial key points and a two-stage decision mechanism, and the key frames in the video sequence are extracted. Finally, the facial action modalities under any posture are extracted based on the facial forward processing, and the head posture attributes obtained based on the head posture estimation are combined to construct the head and face action information fusion space-time network, which is used for detecting the yawning, speaking, normal and other driver states. The application fully considers the head posture attributes, has high posture robustness, and can effectively distinguish the yawning and other fatigue behaviors from other driver states.

This application provides a method, apparatus, and electronic device for detecting attacks on identity authentication, relating to the field of identity authentication technology. The method for detecting attacks on identity authentication includes: performing specified segmentation processing on target audio and target video to obtain multiple data segments, wherein the multiple data segments include various audio segments and various video segments, and the target audio and target video are content recorded during user authentication based on reading specified verification text; extracting speech features from each audio segment to obtain feature vectors for each audio segment, and extracting facial motion features from each video segment to obtain feature vectors for each video segment; determining the deviation result corresponding to each target data segment of a specified media type based on the obtained feature vectors; and determining the detection result based on the deviation result corresponding to each target data segment. Therefore, this solution can improve the accuracy of attack detection.

The application discloses a speech-driven lip shape generation method, device and equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: extracting facial motion parameters and face identification features based on target image frames of a video sequence; encoding a driving audio sequence to obtain a time sequence audio feature sequence; performing prediction based on the facial motion parameters and the time sequence audio feature sequence to obtain target implicit key point expression coefficients; and generating target lip shape video frames based on the target implicit key point expression coefficients and the time sequence audio feature sequence. The application further generates target lip shape video frames by calculating target implicit key point expression coefficients, simplifies the overall process of lip shape generation, and improves the efficiency of speech-driven lip shape generation.

The application discloses an ADHD multi-feature extraction and fusion classification method based on original videos, and comprises the following steps: S1, a network camera is used to collect video records of a subject watching videos, the videos are preprocessed, and preprocessed image frames are obtained; S2, the preprocessed image frames are analyzed to obtain behavior modes including facial actions, eye movements and head movements; S3, feature components of the behavior modes are extracted and fused; and S4, a deep learning network is constructed to classify the fused features. The application further discloses an ADHD multi-feature extraction and fusion classification system based on original videos. The application classifies ADHD patients based on video sequences, avoids invasive influence, reduces cost and is easy to popularize. Through multi-modal feature fusion, the limitation of single modal data is reduced, better accuracy and effectiveness are achieved, and in addition, the application can also be used for classifying autism cases.

The invention provides a peripheral facial paralysis rehabilitation device. The peripheral facial paralysis rehabilitation device comprises a host and at least one stimulation patch electrically connected with the host, the stimulation patch is prefabricated into a special-shaped geometric structure matched with the anatomical direction of facial muscles, and at least integrates a myoelectricity acquisition part, an electrical stimulation part and a heating part; the host is configured to collect a surface electromyographic signal sequence of a user under a specified facial action through the electromyographic collection part of the worn stimulation patch, and the stimulation patch worn by the user is in one-to-one correspondence with target facial muscles involved in the specified facial action; determining at least one muscle function state index according to the surface electromyogram signal sequence; determining a working mode according to the muscle function state index; and controlling the electrical stimulation part and / or the heating part of the stimulation patch worn by the user to work according to the working parameters corresponding to the determined working mode.

The invention discloses a deep learning-based face attribute editing method for a generative adversarial network, and the method comprises the steps: 1, extracting a source face image, 2, generating a coding potential vector w, 3, obtaining a face action unit vector of a target attribute, 4, carrying out the fusion generation of a new potential vector w, and 5, carrying out the face attribute editing of the target attribute. 5, generating an edited face image and constructing a face database; according to the method, the generative adversarial network based on deep learning is combined with multi-scale content compensation, the resolution of the generated image is effectively improved, facial details are clear, artifacts are remarkably reduced, emotion attributes are accurately edited through the facial action unit, high-fidelity retention of identity features is achieved in combination with an age transformation model, and the user experience is improved. High-fidelity emotional face samples under the conditions of different ages, genders and races are generated, and a multi-age and multi-races static and dynamic emotional face database is constructed based on the high-fidelity emotional face samples.

The invention discloses a face state recognition method for face video sequence analysis, and belongs to the field of computer vision. The method takes a face video sequence as input, and comprises the following steps: firstly, performing frame extraction and preprocessing on video data; detecting and screening action key areas related to the face state change through a multi-scale face action key area positioning module; on this basis, constructing a full-connection weighted graph, and fusing a spatial cooperation relationship among the key regions based on a graph convolutional network to obtain a graph embedding feature in a single-frame state; inputting the image embedding features of each frame into a bidirectional long-short-term memory network according to a time sequence, and extracting global time sequence features; and finally, through a full connection layer and a Softmax layer, outputting probability distribution of categories or grades related to the face state. The method can automatically position the facial action key area and model the collaborative change mode of the facial action key area, and can be widely applied to various application scenes such as expression recognition, driver state monitoring, human-computer interaction and facial action capture.

Provided in the present application are a video generation method and a related apparatus. The embodiments of the present application can be applied to various scenarios such as artificial intelligence and computer vision. An embodiment of the present application comprises: first acquiring a face image and a video frame sequence comprising facial motion information of a talking face; then, extracting facial feature information from the face image, and extracting from the video frame sequence head pose features and facial expression features of the talking face; then, performing rendering on the basis of the facial feature information, head pose feature information and facial expression feature information, so as to obtain target frame images; and finally, on the basis of the target frame images, generating a target synthetic video, facial features of the target synthetic video being the same as facial features of the face image. The method provided by the embodiments of the present application introduces a video frame sequence comprising facial motion information of a talking face, so as to add to a static face image dynamic information missing during talking, thereby improving the quality of a target synthetic video.

This invention relates to the field of intelligent recognition, and more particularly to an intelligent micro-expression recognition system based on facial images. The invention includes a detail extraction module, which retrieves image data corresponding to the observed object in a video image to extract corresponding facial action features; an object analysis module, which combines facial action features and the frequency of contour jitter of the observed object to calculate a representation value of the observed object's action state, thereby classifying the observed object's action abnormality and ambiguity category; a labeling module, which selects to call a conventional labeling module or a dynamic evaluation module based on the action abnormality and ambiguity category; a conventional labeling module, which marks video images as forged videos; and a dynamic evaluation module, which evaluates and labels the observed object. This invention captures facial features and relies on precise quantitative analysis of dynamic details of micro-expressions to improve recognition accuracy and precision. In public security work, it can identify AI-generated facial image videos and combat online and telecommunications fraud.

The invention relates to the technical field of artificial intelligence and human-computer interaction, and discloses a rehabilitation training adjustment method and system based on multi-modal feature fusion and a storage medium, and the method comprises the steps: obtaining voice, face and physiological signals, constructing a multi-modal data stream, extracting features, and outputting a speech error classification identifier and a physiological signal quality index; splitting the facial action net displacement into healthy and affected sides, correcting affected side features based on healthy side features, and constructing visual representation; fusing each feature to output a comprehensive state vector; updating a dynamic baseline in a task gap, mapping a state vector, and screening to obtain an effective action set; calculating a composite reward and storing the composite reward in an experience playback pool to update the strategy network; and based on the action set, re-weighting the large model output probability, and generating a target interaction corpus. According to the method, the state sensing precision is improved by correcting the deviation of the affected side through the uninjured side, and a closed loop of self-adaptive adjustment of the rehabilitation difficulty and safe interaction text generation is realized by combining the dynamic baseline and penalty mechanism optimization reinforcement learning.

The invention provides a multi-source motion fusion speaker video generation system and method, belongs to the technical field of video generation, and aims to solve the problems of lip shaking and motion blurring in traditional speaker video generation. The system comprises an input processing and static modeling module, an initial and final motion conversion module, a multi-source adaptive fusion module and a dynamic rendering and synthesizing module. The method comprises the following steps: establishing a unique figure three-dimensional representation and an audio and video feature representation of a figure identity through an input processing and static modeling module, and analyzing a processing result through an initial and final motion conversion module to generate a control signal; the control signals are input into a multi-source adaptive fusion module for intelligent fusion to generate a high-fidelity control instruction for finally driving the three-dimensional face to move, and finally the high-fidelity control instruction for driving the three-dimensional face to move is efficiently and vividly converted into continuous high-fidelity speaker video frames for output through a dynamic rendering and synthesis module.

The invention provides an AI-based model generation method for driving and controlling facial actions, and the method comprises the steps: obtaining a plurality of pieces of audio and video data, each piece of audio and video data comprising audio data of a target object, facial action data corresponding to the audio data, and depth information corresponding to the facial action data; for each piece of audio and video data, utilizing a tool for resolving facial action data of a virtual object to generate actual controller data of each controller in the tool, the actual controller data being used for driving and controlling the facial action of the virtual object; and preprocessing the actual controller data, and training a deep learning model used for generating the facial action of the virtual object by taking the preprocessed actual controller data and the corresponding audio data as training data. The problem that the facial action of the virtual object generated by voice is not fine is solved.

This invention relates to the field of image processing technology, and more particularly to a method, apparatus, device, and medium for region segmentation of facial action units. The application first acquires a region segmentation model and a target face image to be segmented; simultaneously, it acquires a manually labeled first face image and an unlabeled training set of expression images. Based on the labeling result of the first face image, it performs region labeling on the images in the expression image training set to obtain the target labeling result for the corresponding training image; then, it uses the region segmentation model to predict the region segmentation of the images in the expression image training set to obtain the predicted labeling result for the corresponding training image; furthermore, based on the target labeling result and the predicted labeling result, it obtains a trained region segmentation model, and uses the trained region segmentation model to segment the target face image to obtain the region segmentation result; finally, with minimal manual annotation, the accuracy of region segmentation for facial action units is improved.

The invention discloses a digital human video generation method and device, equipment, a storage medium and a product, and relates to the technical field of artificial intelligence. The method comprises the following steps: inputting an original face image into an appearance and motion extractor of a target video generation model to obtain an appearance feature map and a face motion parameter; inputting the original audio into a facial expression generator of the target video generation model to obtain facial expression parameters; inputting the original audio into a head posture generator of a target video generation model to obtain head posture parameters; and inputting the appearance feature map, the facial motion parameter, the facial expression parameter and the head posture parameter into a video generation network of the target video generation model to obtain a digital human video generated by the video generation network. Decoupling is carried out through the determination process of the facial expression parameter and the head posture parameter, and the synchronism of the facial expression and the audio content in the generated video is improved.

A two-stage approach for learning and generating an expressive text-to-motion animation from partially annotated datasets (T2M-X). In an example implementation, T2M-X builds a unified motion dataset based on partially annotated datasets. In the first stage, T2M-X uses the unified motion dataset to train three vector-quantized variational autoencoders (VQ-VAE) for body, hand, and face, respectively, and generate high-quality motion outputs. In the second stage, T2M-X uses the high-quality motion outputs to train a multi-indexing generative pre-trained transformer (GPT) model that includes motion consistency loss and sequence length consistency for learning and then generating coordinated and expressive animations.

The present disclosure provides a video generation method, a model training method and related products. The video generation method comprises: performing feature extraction on an object image of a target object to obtain a first image feature; performing semantic feature extraction on a target audio to obtain a first audio feature; performing sound event feature extraction on the target audio to obtain a second audio feature; determining a facial motion feature of the target object according to the first audio feature and the second audio feature; and generating a target video corresponding to the target object according to the first image feature and the facial motion feature. According to the embodiments of the present disclosure, the sound events contained in the audio can be extracted and processed, so that the digital person can respond to the sound events accurately.

The invention provides a video generation method and device based on style migration, equipment and a medium, relates to the technical field of artificial intelligence, and is suitable for the fields of financial science and technology and medical health. The method comprises the following steps: performing facial action detection on an image frame of an original video to obtain an original facial action; performing action description on the original facial action through a large language model to obtain a candidate facial action description text; screening the image frames according to the image frames to obtain a target facial action description text, and then performing emotion recognition to obtain target text emotion features; performing feature mapping according to the target audio to obtain audio mapping visual features; performing face parameter mapping according to the target text emotion feature, the audio mapping visual feature and the target identity feature to obtain a target three-dimensional face parameter; and performing video rendering according to the target three-dimensional face parameters to obtain a target video. According to the invention, micro-expression-level expression control can be realized, facial distortion is reduced, and the video generation accuracy is improved.

The invention discloses a micro-expression recognition method based on optical flow guidance and local-global representation fusion, and belongs to the technical field of computer vision and emotion calculation. Comprising the following steps: step 1, designing an optical flow coding module, and extracting micro-expression motion information through TV-L1 optical flow and strain features; and step 2, designing an optical flow guided double-attention Transform mechanism, using optical flow motion information to significantly modulate an image feature learning process, highlighting a face motion area, and inhibiting identity and background interference. 3, designing a local-global representation learning module, dividing a human face into 9 AU semantic regions, extracting local features through a local Transform sharing a weight, modeling a cooperative relationship between motion regions through a global Transform, and realizing collaborative optimization of local-global representation, and 4, designing a multi-objective loss function, promoting double-branch deep collaboration, and realizing collaborative optimization of local-global representation. And the complementarity and discrimination of the features and semantic alignment of the expression labels are enhanced, so that the recognition performance of micro-expression recognition is further improved.

The invention relates to the technical field of facial expression analysis, in particular to a facial expression analysis method based on deep learning, which comprises the following steps: acquiring facial feature point coordinates, analyzing displacement and density distribution, obtaining a facial activity intensity index, calling muscle action data, analyzing synchronization rate and direction consistency, and constructing muscle group interaction data. The method comprises the following steps: tracking facial actions, identifying action time, analyzing an action change rate, calculating an action change contribution ratio, analyzing a ratio variable amplitude, adjusting a weight, obtaining an action contribution adjustment result, and comparing a facial region action collaboration degree to obtain a facial coordination output index. According to the method, through fine comparison of facial feature point displacement and density distribution, the measurement precision of facial activity intensity is improved, emotion information extraction is more accurate, muscle group interaction analysis particularly strengthens measurement of synchronous activity rate and direction consistency, and the recognition capability of micro expressions and fast change expressions is improved.

The present disclosure provides an upper limb movement rehabilitation system based on emotional interaction, comprising: a sensing unit for acquiring physiological signals and behavioral signals of a patient; in the central controller, a data processing module determines the physical state and psychological state of the patient in real time, an interaction strategy generation module simulates the professional knowledge and thinking mode of a rehabilitation physician to understand and integrate the current state of the patient for reasoning and decision-making based on a large language model, and outputs a multi-dimensional interaction strategy with the patient; in the execution unit, a loudspeaker is used to play a voice with a familiar timbre feature and a positive emotional style of the patient according to the voice interaction content during training; a visual interaction device generates a virtual character according to the visual interaction content, and makes it produce corresponding facial movements and expressions with the voice generated by the loudspeaker; the upper limb rehabilitation robot guides the patient to complete the upper limb rehabilitation training action according to the upper limb rehabilitation training content. The present disclosure provides emotional support for patient training, enhancing the rehabilitation effect.

The invention discloses a vehicle-mounted digital human interaction method and device, equipment, a storage medium and a product, and relates to the technical field of digital humans, and the vehicle-mounted digital human interaction method comprises the steps: when the local is in a preset resource limited scene, sending a digital human interaction instruction to a cloud; receiving a facial expression video stream fed back by the cloud, the facial expression video stream being obtained by rendering a facial action data frame; and decoding the facial expression video stream, and performing corresponding digital human interaction based on the decoded facial expression video stream. According to the method and the device, the decoding pressure of the vehicle-mounted terminal is reduced, the fluency of corresponding digital human interaction of the decoded facial expression video stream can be improved, and the digital human interaction quality is further improved.

The application discloses a virtual anchor real-time driving system based on facial motion capture, aims to solve the problems of insufficient precision, lack of naturalness and weak scene adaptation of virtual anchor facial motion in the prior art, and through multi-thread parallel collection of audio and video streams, optimizes data quality through adaptive preprocessing; adopts multi-dimensional facial feature collaborative extraction, double-branch phoneme recognition, face function area semantic segmentation and confidence dynamic weight adjustment technology, realizes accurate representation and collaborative fusion of expression and lip feature; finally, through the lip-expression collaborative driving mechanism, the real-time driving characteristics of the adaptive virtual image are output; thereby effectively improving the precision and naturalness of virtual anchor facial motion, enhancing the complex scene adaptability, giving consideration to real-time response and low-cost deployment, and being applicable to virtual live broadcast, online education and other scenes, and having good application value.

The application provides a long video micro-expression detection and recognition model construction and application method and an electronic device. A pupil dynamic feature is introduced to cooperatively modulate a facial motion feature, and a long video micro-expression detection and recognition model is constructed, so as to solve the problems that in the prior art, long video micro-expression analysis mainly depends on a single facial motion mode, a weak micro-expression key moment is not sensitive enough, cross-modal response is not synchronous, and video level interval recovery stability is poor.

The invention relates to the field of intelligent recognition, in particular to a micro-expression intelligent recognition system based on a face image, and the system is provided with a detail extraction module which is used for calling picture data corresponding to an observation object in a video image so as to extract corresponding facial action features; the object analysis module is used for calculating an action state characterization value aiming at the observation object in combination with the facial action characteristics and the contour jitter frequency of the observation object so as to divide an action anomaly fuzzy category of the observation object; the calling labeling module is used for selectively calling the conventional labeling module or the dynamic evaluation module based on the action abnormity fuzzy category; the conventional labeling module is used for labeling the video image as a forged video; and the dynamic evaluation module is used for evaluating and marking the observation object. According to the method, facial features are captured, recognition accuracy and precision are improved by means of precise quantitative analysis of micro-expression dynamic details, and AI face image videos can be discriminated and network telecommunication fraud can be attacked in public security work.