195 results about "Facial motion" patented technology

Systems, methods, and computer program products are disclosed for generating a common record based on differing source inputs. Generating a common record based on differing source inputs may include receiving via at least one sensor first non-audible signals indicative of verbalization of an individual; interpreting first words of the individual at least in part using the first non-audible signals received from the at least one sensor; receiving second signals generated by a source other than the individual; performing speech recognition on the second signals to interpret second words from the source; and using the first words and the second words to generate a record.

The invention discloses an emotion detection system based on facial recognition, and relates to the technical field of computer vision and emotion calculation. A video stream time sequence analysis module is used for extracting a facial micro-expression image sequence of continuous frames, a time sequence feature vector containing a micro-expression intensity gradient, an illumination robustness coefficient and a facial action unit cooperation feature is generated, and a multi-mode dynamic sensing module is combined to carry out real-time analysis on an emotion classification probability, voice emotion parameters and physiological signals. And the fusion decision module performs dynamic weighted fusion on the multi-modal data based on the scene adaptive weight, and finally generates a comprehensive emotion score. Through multi-modal time sequence modeling and a dynamic weight optimization mechanism, the accuracy and environmental adaptability of emotion recognition are remarkably improved, and real-time perception and accurate decision making of customer emotion are realized in a target scene.

Provided in the embodiments of the present disclosure are a video generation method and a parameter generation model training method. The video generation method comprises: acquiring speech to be processed; inputting an emotion feature of a target object and said speech into a parameter generation model, so as to obtain an expression parameter, wherein the expression parameter is used for describing facial motion information of the target object under the influence of the emotion feature, the parameter generation model is obtained by means of performing training on the basis of a sample emotion feature, sample speech and an expression parameter label that corresponds to the sample speech, and the sample emotion feature and the sample speech are obtained on the basis of a sample video; and inputting an object image of the target object and the expression parameter into a video generation model, so as to obtain a target video of the target object. By means of generating an expression parameter on the basis of an emotion feature and speech to be processed, and further generating a target video on the basis of the expression parameter, diversified emotion information is fused into the target video while ensuring the synchronization between speech and expression in the target video, thereby improving the accuracy and vividness of the target video.

The invention discloses an interpretable multi-modal information system for postoperative chronic pain assessment, which is characterized by comprising a facial expression algorithm framework for acquiring dynamic change of a facial expression facial action unit (AU) and a physiological signal algorithm module for receiving physiological signals, and the multi-modal fusion framework is used for fusing the facial expression algorithm framework and the physiological signal algorithm module. The method has the effects of accuracy, individuation and interpretability.

The invention discloses an audio and video depth forgery detection method based on quality perception and multi-scale alignment, and the method comprises the following steps: coding a synchronous audio and video sequence, and obtaining a frame-level visual feature, a facial action unit and a phoneme-level voice representation; a visual quality evaluation module is introduced to generate a spatial reliability mask, and quality weighting is carried out on the visual features; designing a global-local multi-scale cross-modal alignment mechanism, performing bidirectional cross-attention modeling on voice and face dynamic synchronization globally, and performing physiological coupling alignment on phonemes and face action units locally; and an uncertainty perception reasoning and calibration scheme is provided, adaptive temperature scaling is carried out according to quality and consistency, and uncertainty calibration is carried out by self-supervision loss. According to the method, the problems of insufficient robustness and excessive self-confidence misjudgment of an existing method in a low-quality video and high-synchronization counterfeit scene are solved, and the cross-dataset generalization capability and the actual deployment reliability are remarkably improved.

The invention relates to the technical field of data processing, in particular to a method for driving emotion interaction of an intelligent device based on multi-modal understanding, and aims to eliminate illumination and noise interference and output a standardized face video stream, an effective voice segment and a touch thermodynamic diagram through an environment adaptive acquisition module. The feature extraction module extracts facial action optical flow features, voice Mel-frequency cepstral coefficient vectors and tactile pressure gradient parameters. The cross-modal correlation model adopts a tensor decomposition algorithm to calculate a space-time correlation matrix of visual and voice features, and the tactile feature weight is dynamically adjusted in combination with environmental parameters. According to the response strategy, an intervention scheme is retrieved based on a graph database, emotion confirmation statements, guide statements and behavior suggestions are fused to generate multi-mode response, and PID adjustment of the temperature control device and tactile pulse output of the vibration device are synchronously driven. And the feedback evaluation module verifies the emotion recognition consistency through a Pearson's correlation coefficient, triggers conflict sample separation storage and model increment training, and realizes closed-loop optimization.

The invention discloses a facial action unit recognition method and system based on multi-modal fusion and text enhancement. The method comprises the following steps: extracting a facial image attribute set by a visual language model, constructing an adaptive text cue word based on the facial image attribute set, and processing the adaptive text cue word by adopting CLIP to obtain a text semantic feature; extracting facial key point features by the facial key point detection model; the facial expression recognition model extracts facial expression features; the image semantic feature fusion module processes the facial key point features and the facial expression features to obtain multi-modal image semantic features; inputting the multi-modal image semantic features and the text semantic features into a multi-modal feature fusion module to obtain multi-modal fusion features after text information enhancement; the recognition of the facial action unit of the facial image is realized. According to the method, the complementarity of cross-modal features and the robustness of joint representation can be enhanced, and the recognition precision of facial actions in a complex scene is effectively improved.

The invention belongs to the technical field of man-machine interaction and robot control, and particularly relates to a man-machine physical twin system based on multi-modal video analysis and adaptive mapping and a control method, and the system comprises a multi-modal collection unit which is used for collecting original video streams of human body actions, expressions and voices, and compensating and repairing dynamic shielding; the semantic analysis unit comprises an action analysis module used for extracting a human skeleton key point set object interaction track from the original video stream, and an expression / voice analysis module used for outputting a facial action unit coefficient and a voice text with an emotion label; the self-adaptive mapping unit is used for establishing human body-robot joint kinematics mapping, distributing control bandwidth in real time according to task types and degrading non-key modes based on network states to guarantee action continuity, and non-contact action capture is achieved through a pure vision scheme by eliminating dependence on a wearable sensor; and the problems of failure and the like of a traditional visual scheme under shielding and illumination variation are solved.

The invention discloses a cognitive impairment screening method and system based on face analysis. The cognitive impairment screening method comprises the following steps: selecting a video clip with a preset duration for a user to watch, wherein the video clip comprises a plurality of smile points; setting a corresponding time window according to the appearing time of each laughing point; based on spontaneous facial expressions after the user watches the video, facial motion features of the user in each time window are collected, and the facial motion features comprise values of a plurality of facial motion units of the user; inputting the facial motion features of the user in each time window into a preset model to output a cognitive impairment screening result; wherein the preset model carries out model training based on historical big data, so as to form a mapping relation from the facial movement features to the cognitive impairment screening result. Therefore, by utilizing the method, the cognitive impairment can be screened in a diversified scene, the dependence on professionals is reduced, a user does not need to be wounded, and the method is particularly suitable for early screening of the cognitive impairment.

The invention relates to the field of digital people, and discloses a method for driving a digital population pattern by voice, which comprises the following steps: receiving a voice signal and extracting a time-frequency feature, generating a personalized voiceprint feature based on the voice signal, generating a digital population pattern synchronous with the voice signal by using nonlinear dynamic modeling, and generating a digital population pattern synchronous with the voice signal. And precise synchronization of the mouth shape and the voice signal is ensured through a time sequence modeling method. Furthermore, facial expression generation is physically constrained through a Hamiltonian system, and it is ensured that the generated facial action conforms to the biomechanical law. Finally, a digital population type video is generated and output according to a synchronization result, and personalized, high-precision synchronous and natural facial expressions are achieved. The method is suitable for the fields of medical science popularization, virtual assistants and the like, and can improve the video generation efficiency and enhance the user experience.

The invention provides a facial action unit recognition and model training method and device and electronic equipment. The method comprises the steps of obtaining a training sample set and inputting the training sample set into an initial facial action unit recognition model; wherein the training sample set comprises multiple frames and face action unit (AU) real labels corresponding to each frame; performing multi-level feature extraction on each frame by the facial action unit recognition model, generating a visual token according to the extracted features, performing semantic reasoning based on the visual token and a prompt text of a facial action unit recognition task, and generating an AU prediction probability corresponding to each frame; determining a loss value of the loss function according to the AU real label and the AU prediction probability, and reversely adjusting model parameters of the facial action unit recognition model based on the loss value; when a preset training condition is met, training is stopped, and a trained facial action unit recognition model is obtained. According to the invention, semantic reasoning can be carried out by using visual features to complete AU identification, and the accuracy of a facial action unit detection result can be improved.

The present invention provides an emotion recognition method and device for road and bridge engineering scenarios, relating to the field of emotion recognition technology. The present invention pre-processes the acquired user's facial video and physiological information, then performs feature extraction to obtain the user's facial motion units and physiological features; then selects and fuses the optimal feature subset of the user's facial motion units and physiological features; and finally, based on the multi-channel feature inverse reasoning of a dynamic Bayesian network and an interpretable emotion recognition model, obtains the mapping relationship between the fused optimal feature subset and the emotion component, thereby obtaining the user's emotional state recognition result. Compared with the existing technology, the emotion recognition system of the present invention is more reliable and robust, and can achieve multi-level, all-round, and stable emotion recognition of users in the complex environment of road and bridge engineering.

The invention relates to a human body state detection method, system and device based on facial actions and a medium, and the method comprises the steps: obtaining an image sequence of a human face, carrying out the human face detection and mark point positioning of the image sequence, and obtaining the coordinates of a face mark point; performing micro-expression action extraction on the image sequence based on the facial mark point coordinates to obtain facial micro-expression action features; performing time sequence analysis on the facial action features to obtain time sequence facial action features; performing multi-scale illumination adjustment on the time sequence facial action features to obtain illumination facial action features; performing multi-modal analysis on the illumination facial action features through a pre-trained long-short term memory network to obtain comprehensive human body state features; and performing real-time state analysis based on the comprehensive human body state characteristics to obtain a human body state detection result. According to the method, different types of facial action features can be comprehensively processed, and the reliability of a detection result is enhanced.

The invention relates to the technical field of movie and television animation production, in particular to a movie and television animation character expression generation method and system combined with motion capture, and the method comprises the steps: firstly collecting a real-time motion data set of a target actor through motion capture equipment, including face and body motion data subsets; the facial action data is composed of facial key point displacement data with multiple continuous timestamps, then calling a preset expression mapping model to decouple the facial action data to obtain a basic and personalized expression feature set, then adjusting the basic expression feature set according to preset emotion parameters of an animation role, generating an emotion adaptive expression feature set, and generating a personalized expression feature set according to the emotion adaptive expression feature set; the method comprises the steps of obtaining a personalized expression feature set, fusing with the personalized expression feature set to obtain a target expression feature set, finally driving a three-dimensional face model to generate an expression animation by using the target expression feature set, and binding body action data to a three-dimensional body model to generate a complete character animation, thereby improving the authenticity and expressive force of animation character expressions.

The invention discloses an emotion recognition method and system based on facial expression semi-supervised learning, and the method comprises the steps: converting video data into a video frame sequence, and carrying out the preprocessing of the video frame sequence; performing facial key point labeling on the video frame sequence by using OpenPose, and constructing an expression data set with key point labeling; selecting a part of key frames from the expression data set with key point labels, performing manual labeling and auditing of the action units in combination with the facial action units and the relationship between the facial key points and the action units, and establishing an expression and emotion data set; constructing a semi-supervised learning expression recognition network based on manual labeling and auditing of a part of facial action unit expert group on the basis of an expression data set with key point labeling and an expression and emotion data set; and inputting the newly collected video data into the semi-supervised learning expression recognition network to obtain an emotion recognition result. According to the method, the labeling difficulty is reduced, and the precision of infant facial emotion recognition is improved.

A method for generating a talking head video includes: obtaining a text and an image containing a face of a user; determining a phoneme sequence that corresponds to the text and includes one or more phonemes; determining acoustic features corresponding to the text according to the phoneme sequence, and obtaining synthesized speech corresponding to the text according to the acoustic features; determining a first mouth movement sequence corresponding to the text according to the phoneme sequence, and determining a second mouth movement sequence corresponding to the text according to the acoustic features; creating a facial action video corresponding to the user according to the first mouth movement sequence, the second mouth movement sequence and the image; and processing the synthesized speech and the facial action video synchronously to obtain a talking head video corresponding to the user.

The invention relates to the technical field of finance, medical health and artificial intelligence, and provides a TTS voice and 3D mouth shape synchronous generation method, device, equipment and medium, a TTS voice and 3D mouth shape synchronous generation model is obtained by using a small sample adaptive training engine and based on transfer learning framework training, and the generalization ability and training efficiency of the model are improved; multi-modal joint coding and a dynamic weight distribution module are used for joint coding, so that the precision of a generated result is improved; a 3D mouth shape dynamic sequence is obtained based on a diffusion-autoregression hybrid generation module, the detail generation capability of a diffusion model and the time sequence coherence of an autoregression model are fused, and the generation precision and real-time performance are improved; and the real-time synchronization module based on cross-modal attention guidance obtains the synchronously adjusted 3D mouth shape parameters and the facial action sequence matched with the emotion and generates a target video, so that multi-modal dynamic fusion is realized, the lip movement-voice synchronization precision is improved, and the authenticity of emotion expression is enhanced.

The invention provides a GAN-based vehicle-mounted expression data construction method, system and device and a vehicle, and relates to the technical field of intelligent driving. A key point detection technology is utilized to analyze and extract a facial action unit and expression intensity and duration thereof to generate a structured action description. And then, calling a generative adversarial network model to generate a synthetic video corresponding to the target expression type, and carrying out dynamic cutting and frame extraction according to a preset interval on the videos to obtain a multi-frame synthetic image. And finally, performing dual-model filtering on the synthesized image to ensure data quality, and executing privacy anonymization processing, thereby generating a high-quality vehicle-mounted expression data set. According to the method, the data acquisition cost is reduced, the diversity and accuracy of a data set are improved, and the performance and reliability of an expression recognition algorithm in an actual vehicle-mounted environment are effectively enhanced.

The invention relates to a face forgery identification method, a face forgery identification device, equipment and a medium. The method comprises the following steps: determining a first frame image from a to-be-identified video; wherein the first frame image is a frame image of which the face action amplitude change exceeds a threshold value; acquiring first optical flow motion information of the first frame image and second optical flow motion information of the second frame image; wherein the sampling time of the second frame image is before the sampling time of the first frame image; performing gradient calculation on the first frame image to obtain first gradient information corresponding to the first frame image; and based on the first optical flow motion information, the second optical flow motion information and the first gradient information, obtaining a face forgery identification result of the to-be-identified video.

The invention relates to the technical field of face recognition, and discloses a multi-modal alignment training-based face sentiment analysis method and system, and the method comprises the steps: obtaining multi-source data, carrying out the preprocessing of the multi-source data, obtaining a training set, constructing a basic model for the face sentiment analysis, selecting a sample with an inference text to generate a small number of high-quality sentiment analysis samples, and carrying out the recognition of the high-quality sentiment analysis samples; supervising and finely adjusting the model; selecting a sample with a facial action unit label and an emotion label, and performing reinforcement learning training on the model in combination with the predicted accuracy of the facial action unit label, the emotion label and the reasoning text; and training the basic model by using the training set, expanding the original training set by using the output of the trained basic model to obtain a new training set, continuously training the model, stopping training until a preset condition is met to obtain a final basic model, and performing facial sentiment analysis by using the final basic model. According to the method, illusion can be controlled, the accuracy of a facial emotion analysis result is improved, and the data set construction cost is reduced.

The invention belongs to the technical field of image restoration and computer vision, and particularly relates to a multi-mode face restoration and expression recognition system and method based on semantic guidance of a facial action unit. According to the system, multi-scale features are extracted through visual coding, and an AU activation probability is detected by using a graph neural network; the semantic conversion module converts the numerical probability into an interpretable biomechanical structured text; the multi-modal reasoning module fuses vision and text information, introduces the common sense reasoning ability of a multi-modal large language model, and improves the student network performance through knowledge distillation; and finally, the conditional generation module realizes image restoration by taking the semantic features as guidance. The facial action unit is used as a biomechanical medium, the multi-modal reasoning ability is converted into restoration constraint, the defects that in the prior art, restoration of physiology is distorted, recognition depends on image quality, and two tasks are isolated are overcome, collaborative enhancement of face restoration and expression recognition is achieved, and it is ensured that the restoration result is clear in vision and conforms to the physiological law of facial muscles.

The invention provides an intelligent expression generation method and system based on a facial motion unit, and belongs to the technical field of intelligent generation, and the method comprises the steps: determining a unit data set of each facial motion unit of a corresponding individual based on a multi-modal facial sample; inputting the phoneme codes into an expression predictor to obtain a prediction data set of each facial motion unit; establishing a mapping relationship based on the unit data set of each facial movement unit according to different individuals in various expression states, and establishing a neutral relationship between each facial movement unit and the corresponding expression state in combination with a deviation set determined based on the prediction data set and the unit data set; establishing a facial motion unit expression model; and searching required motion features and parameter settings from the established facial motion unit expression model, and performing dynamic adjustment based on physical constraints of facial muscle motion and natural transition constraints of expression changes to generate expression animations for interaction. And a high-precision, high-reality and controllable expression generation effect is realized.

The invention belongs to the technical field of facial action driving, and particularly relates to a facial action driving method and device, a computer readable storage medium and a robot. The method comprises the following steps: acquiring a target sound for driving a facial action; processing the target sound through a preset three-dimensional facial action model to obtain a first facial action corresponding to the target sound; wherein the first facial action is a human three-dimensional facial action based on hybrid deformation expression; performing mapping processing on the first facial action to obtain a second facial action corresponding to the first facial action; wherein the second facial action is a robot three-dimensional facial action based on hybrid deformation expression; and controlling a face motor of the robot to execute the second face action. According to the method and the device, the three-dimensional facial action based on hybrid deformation expression is adopted, and the human three-dimensional facial action is mapped into the robot three-dimensional facial action, so that the method and the device can be better applied to an entity bionic robot.

The invention discloses a digital human video rendering method and device and a medium, and relates to the crossing field of computer graphics and generative adversarial networks, and the method comprises the steps: constructing a generative adversarial model based on a generative adversarial network of a single fusion architecture; performing multi-modal preprocessing on the original audio and video data based on the digital human reference image; extracting dual-granularity speech features through a speech feature extraction module of a generative adversarial model, and fusing the dual-granularity speech features; determining a local deformation field of the digital human reference image in a UV parameterized space based on a reference key point corresponding to the digital human reference image and the fused speech features; sampling the identity texture of the digital human reference image to generate a target digital human face image; and verifying the digital human face image based on multiple scales through a discriminator of a generative adversarial model. Through dual-granularity speech feature fusion and generator multi-resolution injection, deep alignment of speech semantics and facial actions is realized.

The invention discloses a multi-modal behavior data processing system, which comprises a data acquisition module, a feature extraction layer, a dynamic attention weight layer, a multi-modal fusion layer and a downstream task decision-making layer, the data acquisition module guides human-computer interaction through an international neurological and mental interview tool matched with a DSM-5 standard and acquires audio and video stream data; the feature extraction layer extracts a video feature vector (including facial action unit activation intensity and the like), an audio feature vector (including Mel frequency cepstrum coefficient and the like) and a text feature vector (generated by a deep language model after automatic speech recognition transcription) in parallel; the dynamic attention weight layer is combined with data quality, symptomatic priori knowledge and cross-modal correlation to generate a dynamic fusion weight; weighting, splicing and dimensionality reduction are carried out on the multi-modal fusion layer to obtain a fusion feature vector; and the downstream task decision-making layer completes evaluation and generates a multi-modal behavioral index evaluation report. The system is deployed in a non-intrusive manner, the risk is controllable, and the evaluation robustness and accuracy can be improved.

For each avatar, testing images are rendered for different facial expressions that each have ground truth facial action units. An instance of a machine learning model is applied to the testing images to generate predicted facial action units for each testing image. A predictive performance of the instance is calculated for each avatar based on the predicted and ground truth facial action units for the testing images of the avatar. A first set of features common to the avatars for which the predictive performance was better than a first threshold, and a second set of features common to the avatars for which the predictive performance was worse than a second threshold, are identified. The features present only in the second set are identified, as difference features. New avatars having the difference features are generated.WO

Utilization of windowing to set a region of interest (ROI) of a camera used for tracking facial expressions. In one embodiment, a system includes an inward-facing head-mounted camera that captures images of a region on a user's head utilizing a sensor that supports changing of its ROI. The system also includes a computer that detects, in a first subset of the images, a first sub-region in which changes due to a first facial movement reach a first threshold and reads from the camera a first ROI that covers at least a portion of the first sub-region. The computer detects, in a second subset of the images, a second sub-region in which changes due to a second facial movement reach a second threshold, and then reads from the camera a second ROI that covers at least a portion of the second sub-region, with the first and second ROIs being different.

The invention relates to the technical field of three-dimensional face animation generation, in particular to a voice-driven dynamic three-dimensional face and dynamic texture generation method and system, and the method comprises the following steps: S1, training two independent auto-encoders to respectively learn change primitives of face geometry and wrinkles, the facial motion map and the wrinkle map are compressed to low-dimensional implicit features; s2, based on implicit features zw and zf generated by an encoder, training a collaborative generation network based on a diffusion model, and generating clean implicit features from noise input under the guidance of audio features; and S3, constructing an implicit facial motion feature by using the speaking style axis feature pf, i of the style i, and constructing an implicit texture change feature by using the wrinkle style axis feature pw, j of the style j. The method can effectively solve the problems that an existing voice-driven face animation method cannot generate dynamic texture maps which are consistent and coordinated with face movement at the same time, the sense of reality of the face animation is reduced due to the lack of important face details, even the terrorist valley effect is caused, and the animation effect is affected.

The invention belongs to the technical field of facial action driving, and particularly relates to a facial action driving method and device, a computer readable storage medium and electronic equipment. The method comprises the following steps: acquiring a target sound for driving a facial action; performing emotion feature extraction processing on the target sound to obtain an emotion feature corresponding to the target sound; based on the emotion features, processing the target sound through a preset three-dimensional facial action model to obtain a three-dimensional facial action corresponding to the target sound; and controlling a preset virtual digital human to execute the three-dimensional face action. Through the method and the device, the emotion features in the sound can be extracted, and the corresponding three-dimensional facial action is generated based on the emotion features in the facial action driving process, so that the facial action can be emotional like a real human facial action, and is more real and natural.

The embodiment of the invention provides a video generation method and system and a video generation model training method and system. The video generation system can obtain a reference image displaying an image of a digital person, generate video picture features based on the reference image, obtain a target voice to be broadcasted by the digital person, generate face driving features at least based on the target voice, and obtain a limb action to be executed by the digital person. And generating limb driving features based on the limb actions. And furthermore, the video generation system inputs the video picture features, the limb driving features and the face driving features to a video generation model so as to generate a target video through the video generation model, and the target video displays a scene in which the digital human executes the limb actions and broadcasts the target voice. Wherein the video generation model is trained as follows: in the video generation process, the face driving features and the limb driving features are taken as conditions to control the limb actions and the face actions of the digital human.