170 results about "Facial movement" patented technology

Systems, methods, and computer program products are disclosed for generating a common record based on differing source inputs. Generating a common record based on differing source inputs may include receiving via at least one sensor first non-audible signals indicative of verbalization of an individual; interpreting first words of the individual at least in part using the first non-audible signals received from the at least one sensor; receiving second signals generated by a source other than the individual; performing speech recognition on the second signals to interpret second words from the source; and using the first words and the second words to generate a record.

The invention discloses an auxiliary decision-making system for cognitive competence assessment of old people, which relates to the technical field of cognitive auxiliary decision-making and comprises an environmental noise acquisition and feature extraction module, a time sequence fluctuation analysis module, a dynamic threshold reconstruction and judgment module, a cross-domain consistency calibration module, a multi-modal synchronous verification module and a dynamic threshold regulation and control module. According to the method, the environmental noise is dynamically analyzed, so that the defects of a fixed noise suppression threshold and a static feature extraction model in the traditional technology are overcome. A sound field energy distribution model is constructed in real time and time sequence fluctuation analysis is combined so that a noise fluctuation interval can be identified, a voice starting point detection threshold value is dynamically adjusted and evaluation precision is enhanced. Through multi-mode synchronous verification combining a reaction time curve and a facial movement track, a time offset error is corrected, a judgment standard is adjusted in real time according to noise changes, misjudgment and virtual high risk early warning are avoided, and therefore the accuracy and reliability of cognitive ability assessment of the old are improved.

The invention discloses a Parkinson's disease early diagnosis system based on eyeball movement characteristic analysis, and relates to the technical field of Parkinson's disease diagnosis. In order to solve the problems that although eye and face movement parameters possibly change in Parkinson's disease, the change of the parameters also possibly occurs in other neurodegenerative diseases or normal aging processes, and high-precision eye and face movement capture data is complex, so that early diagnosis characteristics of Parkinson's disease cannot be accurately captured, and the Parkinson's disease cannot be accurately diagnosed. And the accuracy and efficiency of diagnosis are influenced. By accurately capturing and analyzing eyeball movement data and facial expression video images of a patient, the Parkinson's disease can be effectively distinguished from other diseases possibly causing similar symptoms, misdiagnosis and missed diagnosis are reduced, the standardization and comparability of data are ensured by combining pupil distance information and diopter information, the consistency and reliability of diagnosis are improved, and the diagnosis accuracy is improved. Therefore, the patient can be diagnosed at the early stage of the disease and is treated as soon as possible, and the treatment effect and life quality are improved.

The invention provides a stroke face auxiliary evaluation method and system based on face motion unit detection, and the method comprises the steps: collecting a face motion video of a patient, and recording a start frame and an end frame of each motion in the video; the actions comprise an eyeball horizontal rotation action, a tooth indication action, an eyebrow lifting action and an eye closing action; according to the start frame and the end frame of the horizontal rotation eyeball action, extracting a horizontal rotation eyeball action video, namely an eye rotation action video, and calculating to obtain a severity score of the staring condition of the patient by using a staring scoring algorithm; and respectively extracting videos of the three actions according to starting frames and ending frames of the tooth indicating action, the eyebrow lifting action and the eye closing action, and calculating to obtain a severity score of the facial paralysis condition of the patient by using a facial paralysis scoring algorithm. According to the method, the abnormal conditions of the face of the patient are captured through accurate detection and quantification of various face actions of the patient, the problems that a conventional evaluation mode is low in accuracy, incomplete in evaluation and the like are solved, and an objective, accurate and comprehensive stroke face evaluation effect is achieved.

The invention discloses a voice-driven facial animation simulation method based on facial muscle linkage, and belongs to the field of artificial intelligence, and the method comprises the steps: S1, constructing a PPMF encoder; s2, constructing a decoder based on FDCP to decode FP features provided by the PPMF so as to obtain a facial animation; s3, training a 3D face animation framework DCPTalk driven by voice; s4, optimizing the model; and S5, quantitatively evaluating the model. According to the invention, a DCPTalk framework is provided, and the Mouth2Face is provided based on the linkage characteristic of the facial muscle group. Motion of the mouth has strong correlation with voice signals, and can be synthesized with sound track dynamics easily. In order to further enhance the details of the facial movement, the Refine Decoder is adopted to simulate the skin deformation of the surface layer to refine the facial animation. Inherent body characteristics and body characteristics related to facial muscle group movement are embedded into Mouth2Face, a personalized facial muscle control system is constructed, and meanwhile, an external driving signal is modulated by using a speaking style. Qualitative and quantitative experiments and user research show that the DCPTalk is superior to the most advanced method in the prior art.

This invention belongs to the field of facial expression recognition and provides a micro-expression recognition method based on convolutional neural networks and optical flow features, which is used to improve the adaptability and accuracy of micro-expression recognition. The method first constructs a face detection module to perform face detection on each image frame in a video segment, outputs facial key point information, and extracts facial regions of interest (ROIs) based on the facial key point information: nose, mouth, left eye and eyebrow, right eye and eyebrow. Then, optical flow information and optical strain information are introduced to characterize the spatiotemporal information of facial movement and the intensity information of facial deformation. The optical flow eigenvalue of each image frame is calculated, and a binary search algorithm is used to search for the image frame with the maximum optical flow eigenvalue, which is used as the vertex frame for micro-expression recognition. Finally, a micro-expression recognition model based on a convolutional neural network is constructed, which uses the horizontal and vertical components of the optical flow and optical strain of the vertex frame as input. The micro-expression recognition model outputs a micro-expression category prediction result.

The invention relates to a multi-posture portrait animation generation method. The method comprises the following steps: acquiring a portrait image and converting the portrait image into a three-dimensional deformation model; acquiring audio data, inputting the audio data into an audio-driven facial motion parameter extraction network, and calculating facial motion parameters according to the facial expression parameters; calculating a head posture motion parameter and a hand posture motion parameter according to the head posture parameter, the hand posture parameter and the audio data by using a posture generation variational auto-encoder; extracting appearance features of the portrait image, and performing standard key point estimation on the portrait image to generate standard key points; mapping the head posture parameter, the hand posture parameter, the facial expression parameter, the facial motion parameter, the head posture motion parameter and the hand posture motion parameter into parameter representation under a 2D view; and finally, inputting the mapped parameter representation, the standard key points and the appearance features into an image generator, and generating a multi-pose portrait animation containing hand postures and dynamic expressions.

Implementations described herein relate to methods, systems, and computer-readable media to generate animations for a 3D avatar from input video and audio captured at a client device. A camera may capture video of a face while a trained face detection model and a trained regression model output a set of video FACS weights, head poses, and facial landmarks to be translated into the animations of the 3D avatar. Additionally, a microphone may capture audio uttered by a user while a trained facial movement detection model and a trained regression model output a set of audio FACS weights. Additionally, a blending term is provided for identification of lapses in audio. A modularity mixing component fuses the video FACS weights and the audio FACS weights based on the blending term to create final FACS weights for animating the user's avatar, a character rig, or another animation-capable construct.

The invention discloses a cognitive impairment screening method and system based on face analysis. The cognitive impairment screening method comprises the following steps: selecting a video clip with a preset duration for a user to watch, wherein the video clip comprises a plurality of smile points; setting a corresponding time window according to the appearing time of each laughing point; based on spontaneous facial expressions after the user watches the video, facial motion features of the user in each time window are collected, and the facial motion features comprise values of a plurality of facial motion units of the user; inputting the facial motion features of the user in each time window into a preset model to output a cognitive impairment screening result; wherein the preset model carries out model training based on historical big data, so as to form a mapping relation from the facial movement features to the cognitive impairment screening result. Therefore, by utilizing the method, the cognitive impairment can be screened in a diversified scene, the dependence on professionals is reduced, a user does not need to be wounded, and the method is particularly suitable for early screening of the cognitive impairment.

This technology provides systems and methods for tracking facial movements and reconstructing facial expressions by learning skin deformation patterns and facial features. Frontal view images of a user making a variety of facial expressions are acquired to create a data training set for use in a machine-learning process. Head-mounted or neck-mounted wearable devices are equipped with one or more camera(s) or acoustic device(s) in communication with a data processing system. The cameras capture images of contours of the users face from either the cheekbone or the chin profile of the user. The acoustic devices transmit and receive signals to calculate a representation of the skin deformation. A data processing system uses the images, the profile of the contours, or skin deformation to track facial movement or to reconstruct facial expressions of the user based on the data training set.

The invention relates to an interface display method and device, equipment and a storage medium, and relates to the technical field of multimedia. The method comprises the steps of collecting music in a target space; acquiring mirror moving parameters based on at least one of rhythm features of the music and facial information, wherein the facial information comprises at least one of facial expressions and facial actions of an object in the target space; and displaying a live broadcast interface of the target space based on the mirror operation parameter, the live broadcast interface including a mirror operation effect corresponding to the mirror operation parameter. The mirror moving parameters are not artificially determined based on experience, but are obtained through real-time reasoning based on the music rhythm and the face information of the object during live broadcasting, so that the accuracy of the mirror moving parameters is obviously high, and the mirror moving special effect displayed by the live broadcasting interface is more visual and accurate. Moreover, according to the method, manual participation is not needed, a machine can automatically determine the accurate mirror operation parameters and display the live broadcast interface based on the determined mirror operation parameters, and the man-machine interaction efficiency is further improved.

The invention relates to the technical field of three-dimensional face images, in particular to a digital face image generation method and device, a video generation method and device and a storage medium. According to the method, source three-dimensional mesh grid points and target three-dimensional mesh grid points are generated through cross-identity three-dimensional mesh grid point generation processing, the source three-dimensional mesh grid points are constructed by parameters of a source image, the target three-dimensional mesh grid points fuse identity features and facial details of the source image, and dense three-dimensional facial geometric information is obtained by combining expression and posture information of a driving image. In order to further improve the effect of speaker video generation, two three-dimensional mesh point sensing mechanisms are adopted, the first mechanism is a key point detection mechanism guided by three-dimensional mesh points, and geometric representation of the three-dimensional mesh points and appearance features of an input image are combined to predict more accurate face key points. And the second method is based on a multi-layer cross-modal attention mechanism, the completeness of a face structure can be kept, and fine face movement is captured, so that the fine granularity of face detail generation is enhanced.

The invention relates to the field of digital people, and discloses a method for driving a digital population pattern by voice, which comprises the following steps: receiving a voice signal and extracting a time-frequency feature, generating a personalized voiceprint feature based on the voice signal, generating a digital population pattern synchronous with the voice signal by using nonlinear dynamic modeling, and generating a digital population pattern synchronous with the voice signal. And precise synchronization of the mouth shape and the voice signal is ensured through a time sequence modeling method. Furthermore, facial expression generation is physically constrained through a Hamiltonian system, and it is ensured that the generated facial action conforms to the biomechanical law. Finally, a digital population type video is generated and output according to a synchronization result, and personalized, high-precision synchronous and natural facial expressions are achieved. The method is suitable for the fields of medical science popularization, virtual assistants and the like, and can improve the video generation efficiency and enhance the user experience.

Systems and methods are disclosed for driver assistance. In one example a method for an advanced driver alert system for a vehicle comprises monitoring, via a user-facing camera, facial movement of a user over time. The method includes determining a plurality of facial movement metrics based on the facial movement over an adjustable time window, transforming the plurality of facial movement metrics into a machine readable representation of the adjustable time window, and determining, via a code reading model, one or more user requests based on the machine readable representation of the facial movement metrics. The method includes executing one or more user assistance operations based on the determined user requests.

The invention provides a dynamic fine-grained expression detection method and system based on computer vision. The method comprises the following steps: acquiring a facial action video; taking the face action video as input, and executing a continuous feasible frame extraction algorithm to obtain all continuous feasible frame sequences in the video; and for each continuous feasible frame sequence, executing a dynamic action unit detection algorithm to obtain dynamic action unit strength, a static action maximum frame and a static action minimum frame of the left and right side parts corresponding to the current sequence, and completing detection of expression actions of the left and right side parts. By adopting a continuous feasible frame extraction algorithm and a dynamic action unit detection algorithm, the problems that a conventional expression detection method is difficult to accurately reflect facial movement only through a single-frame facial image and a conventional discrete expression detection result is difficult to accurately describe specific facial movement are solved; effects of efficient detection and accurate quantification of facial expressions are realized.

The present invention provides an emotion recognition method and device for road and bridge engineering scenarios, relating to the field of emotion recognition technology. The present invention pre-processes the acquired user's facial video and physiological information, then performs feature extraction to obtain the user's facial motion units and physiological features; then selects and fuses the optimal feature subset of the user's facial motion units and physiological features; and finally, based on the multi-channel feature inverse reasoning of a dynamic Bayesian network and an interpretable emotion recognition model, obtains the mapping relationship between the fused optimal feature subset and the emotion component, thereby obtaining the user's emotional state recognition result. Compared with the existing technology, the emotion recognition system of the present invention is more reliable and robust, and can achieve multi-level, all-round, and stable emotion recognition of users in the complex environment of road and bridge engineering.

The invention relates to the technical field of 3D facial motion generation, and discloses a facial motion generation method, device and equipment and a storage medium, and the method comprises the steps: responding to multimedia data and emotion data inputted by a user, and determining a voice sequence according to the multimedia data; acquiring facial motion data, and adding noise according to the facial motion data to obtain facial noise motion; and inputting the voice sequence, the facial noise motion and the emotion data into a facial motion generation model to obtain a facial expression video, the facial expression video including a single-emotion facial expression video or a facial expression video with continuous emotion transformation. According to the method and the device, various and vivid facial expressions can be generated, and flexible and continuous emotion control is realized.

Systems, methods, and non-transitory computer-readable media including instructions for detecting and utilizing facial skin micromovements are disclosed. In some non-limiting embodiments, the detection of the facial skin micromovements occurs using a speech detection system that may include a wearable housing, a light source (either a coherent light source or a non-coherent light source), a light detector, and at least one processor. One or more processors may be configured to analyze light reflections received from a facial region to determine the facial skin micromovements, and extract meaning from the determined facial skin micromovements. Examples of meaning that may be extracted from the determined facial skin micromovements may include words spoken by the individual (either silently spoken or vocally spoken), an identification of the individual, an emotional state of the individual, a heart rate of the individual, a respiration rate of the individual, or any other biometric, emotion, or speech-related indicator.

The invention provides a multi-source single-graph digital human generation method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a to-be-processed image, a driving audio and a driving video; performing face detection processing on the to-be-processed image to obtain a first face image; determining an audio driving result based on the first face image and the driving audio; performing face detection processing on the driving video, and determining a face video containing a second face image; and performing fusion processing based on the face video and the audio driving result to generate a face image video of the digital human. According to the embodiment of the invention, the defects that the face action and the speaking audio of the generated digital human are not matched and the accuracy is relatively low in the prior art are overcome, the single-image digital human with high matching degree between the audio and the face action can be generated by combining the to-be-processed image, the driving audio and the driving video, and the single-image digital human has diversity and flexibility.

The invention relates to a face forgery identification method, a face forgery identification device, equipment and a medium. The method comprises the following steps: determining a first frame image from a to-be-identified video; wherein the first frame image is a frame image of which the face action amplitude change exceeds a threshold value; acquiring first optical flow motion information of the first frame image and second optical flow motion information of the second frame image; wherein the sampling time of the second frame image is before the sampling time of the first frame image; performing gradient calculation on the first frame image to obtain first gradient information corresponding to the first frame image; and based on the first optical flow motion information, the second optical flow motion information and the first gradient information, obtaining a face forgery identification result of the to-be-identified video.

The invention provides an intelligent expression generation method and system based on a facial motion unit, and belongs to the technical field of intelligent generation, and the method comprises the steps: determining a unit data set of each facial motion unit of a corresponding individual based on a multi-modal facial sample; inputting the phoneme codes into an expression predictor to obtain a prediction data set of each facial motion unit; establishing a mapping relationship based on the unit data set of each facial movement unit according to different individuals in various expression states, and establishing a neutral relationship between each facial movement unit and the corresponding expression state in combination with a deviation set determined based on the prediction data set and the unit data set; establishing a facial motion unit expression model; and searching required motion features and parameter settings from the established facial motion unit expression model, and performing dynamic adjustment based on physical constraints of facial muscle motion and natural transition constraints of expression changes to generate expression animations for interaction. And a high-precision, high-reality and controllable expression generation effect is realized.

The invention discloses a digital human video rendering method and device and a medium, and relates to the crossing field of computer graphics and generative adversarial networks, and the method comprises the steps: constructing a generative adversarial model based on a generative adversarial network of a single fusion architecture; performing multi-modal preprocessing on the original audio and video data based on the digital human reference image; extracting dual-granularity speech features through a speech feature extraction module of a generative adversarial model, and fusing the dual-granularity speech features; determining a local deformation field of the digital human reference image in a UV parameterized space based on a reference key point corresponding to the digital human reference image and the fused speech features; sampling the identity texture of the digital human reference image to generate a target digital human face image; and verifying the digital human face image based on multiple scales through a discriminator of a generative adversarial model. Through dual-granularity speech feature fusion and generator multi-resolution injection, deep alignment of speech semantics and facial actions is realized.

The invention provides a bionic robot facial action debugging method and related equipment, and relates to the technical field of intelligent robots. According to the method, the to-be-debugged biomimetic robot can be automatically adjusted by taking the manually finely adjusted reference biomimetic robot as a reference object, so that the reference biomimetic robot and the adjusted to-be-debugged biomimetic robot can directly show the same or similar facial expression effect when executing the same facial action; therefore, in the robot batch production process, the manual adjustment workload and the robot facial action expression deviation are effectively reduced, the robot adjustment efficiency and the robot facial action expression consistency are improved, and the aging requirement and the facial action expression consistency requirement in the robot batch production process are met.

The invention discloses a voice-driven multi-expression 3D facial animation generation method and system, and relates to the field of computer animation and virtual reality. The method comprises the following steps: acquiring an input audio, and encoding the audio by using a fine-tuned encoder to obtain features of the input audio; performing feature dimension reduction by using a full-connection neural network to obtain potential codes of audio features; inputting the potential audio code, the emotion code and the diffusion step number into a face denoising model to generate a potential face motion code; performing discrete quantization processing on the facial motion codes by using a codebook to obtain approximate potential facial motion codes in the codebook; and decoding the discretized potential facial motion code to obtain facial motion, and adding the facial motion to a static facial template to obtain a final complete facial motion sequence. According to the method, the 3D facial animation which is high in quality and closely synchronous with the input audio and the emotion label is generated, and a new effective means is provided for creating a virtual character with rich emotion.

Utilization of windowing to set a region of interest (ROI) of a camera used for tracking facial expressions. In one embodiment, a system includes an inward-facing head-mounted camera that captures images of a region on a user's head utilizing a sensor that supports changing of its ROI. The system also includes a computer that detects, in a first subset of the images, a first sub-region in which changes due to a first facial movement reach a first threshold and reads from the camera a first ROI that covers at least a portion of the first sub-region. The computer detects, in a second subset of the images, a second sub-region in which changes due to a second facial movement reach a second threshold, and then reads from the camera a second ROI that covers at least a portion of the second sub-region, with the first and second ROIs being different.

The invention relates to the technical field of three-dimensional face animation generation, in particular to a voice-driven dynamic three-dimensional face and dynamic texture generation method and system, and the method comprises the following steps: S1, training two independent auto-encoders to respectively learn change primitives of face geometry and wrinkles, the facial motion map and the wrinkle map are compressed to low-dimensional implicit features; s2, based on implicit features zw and zf generated by an encoder, training a collaborative generation network based on a diffusion model, and generating clean implicit features from noise input under the guidance of audio features; and S3, constructing an implicit facial motion feature by using the speaking style axis feature pf, i of the style i, and constructing an implicit texture change feature by using the wrinkle style axis feature pw, j of the style j. The method can effectively solve the problems that an existing voice-driven face animation method cannot generate dynamic texture maps which are consistent and coordinated with face movement at the same time, the sense of reality of the face animation is reduced due to the lack of important face details, even the terrorist valley effect is caused, and the animation effect is affected.

The embodiment of the invention provides a video generation method and system and a video generation model training method and system. The video generation system can obtain a reference image displaying an image of a digital person, generate video picture features based on the reference image, obtain a target voice to be broadcasted by the digital person, generate face driving features at least based on the target voice, and obtain a limb action to be executed by the digital person. And generating limb driving features based on the limb actions. And furthermore, the video generation system inputs the video picture features, the limb driving features and the face driving features to a video generation model so as to generate a target video through the video generation model, and the target video displays a scene in which the digital human executes the limb actions and broadcasts the target voice. Wherein the video generation model is trained as follows: in the video generation process, the face driving features and the limb driving features are taken as conditions to control the limb actions and the face actions of the digital human.

The present disclosure relates to a device and method for generating avatar lip-sync animation based on multimodal biosignals, The device comprises a multimodal data collection unit configured to collect data including biosignal data including brain waves when a user imagines speaking and image data; a preprocessing unit configured to preprocess the multimodal data; a feature extraction unit configured to extract feature vectors including the user's biosignal feature and facial feature from the preprocessed multimodal data; an avatar generation unit configured to generate an avatar; a lip-sync reconstruction unit configured to predict the mouth shape and facial movement when the user imagines speaking by inputting the extracted feature vectors to a pre-prepared lip-sync reconstruction model; and a lip-sync animation implementation unit for implementing an avatar lip-sync animation by applying the mouth shape and facial movement predicted by the lip-sync reconstruction unit to the avatar generated by the avatar generation unit.

The invention discloses a method for matching facial expressions and actions of digital cat puppet images through human voice, which comprises the following steps of: S1, acquiring voice data of a user, and preprocessing the voice data; s2, converting the preprocessed voice data into text data, performing sentiment analysis, and extracting sentiment information; s3, constructing an action library, pre-defining various facial actions of the digital cat puppet, and distributing an emotion label for each action; according to a sentiment analysis result, selecting proper facial expressions and actions for mapping; and S4, generating a corresponding facial action according to a mapping result, and rendering the generated facial action to a model of the digital cat puppet in real time. According to the method, dynamic interaction between a virtual character and user voice input is realized through voice emotion driven digital cat puppet face action matching and real-time rendering, based on voice preprocessing and emotion analysis technologies, emotion information in voice can be accurately extracted, and the naturalness and coherence of digital cat puppet face actions are ensured.

This application provides a method, apparatus, and electronic device for detecting attacks on identity authentication, relating to the field of identity authentication technology. The method for detecting attacks on identity authentication includes: performing specified segmentation processing on target audio and target video to obtain multiple data segments, wherein the multiple data segments include various audio segments and various video segments, and the target audio and target video are content recorded during user authentication based on reading specified verification text; extracting speech features from each audio segment to obtain feature vectors for each audio segment, and extracting facial motion features from each video segment to obtain feature vectors for each video segment; determining the deviation result corresponding to each target data segment of a specified media type based on the obtained feature vectors; and determining the detection result based on the deviation result corresponding to each target data segment. Therefore, this solution can improve the accuracy of attack detection.