120 results about "Facial movement" patented technology

The invention discloses an auxiliary decision-making system for cognitive competence assessment of old people, which relates to the technical field of cognitive auxiliary decision-making and comprises an environmental noise acquisition and feature extraction module, a time sequence fluctuation analysis module, a dynamic threshold reconstruction and judgment module, a cross-domain consistency calibration module, a multi-modal synchronous verification module and a dynamic threshold regulation and control module. According to the method, the environmental noise is dynamically analyzed, so that the defects of a fixed noise suppression threshold and a static feature extraction model in the traditional technology are overcome. A sound field energy distribution model is constructed in real time and time sequence fluctuation analysis is combined so that a noise fluctuation interval can be identified, a voice starting point detection threshold value is dynamically adjusted and evaluation precision is enhanced. Through multi-mode synchronous verification combining a reaction time curve and a facial movement track, a time offset error is corrected, a judgment standard is adjusted in real time according to noise changes, misjudgment and virtual high risk early warning are avoided, and therefore the accuracy and reliability of cognitive ability assessment of the old are improved.

This invention belongs to the field of facial expression recognition and provides a micro-expression recognition method based on convolutional neural networks and optical flow features, which is used to improve the adaptability and accuracy of micro-expression recognition. The method first constructs a face detection module to perform face detection on each image frame in a video segment, outputs facial key point information, and extracts facial regions of interest (ROIs) based on the facial key point information: nose, mouth, left eye and eyebrow, right eye and eyebrow. Then, optical flow information and optical strain information are introduced to characterize the spatiotemporal information of facial movement and the intensity information of facial deformation. The optical flow eigenvalue of each image frame is calculated, and a binary search algorithm is used to search for the image frame with the maximum optical flow eigenvalue, which is used as the vertex frame for micro-expression recognition. Finally, a micro-expression recognition model based on a convolutional neural network is constructed, which uses the horizontal and vertical components of the optical flow and optical strain of the vertex frame as input. The micro-expression recognition model outputs a micro-expression category prediction result.

This technology provides systems and methods for tracking facial movements and reconstructing facial expressions by learning skin deformation patterns and facial features. Frontal view images of a user making a variety of facial expressions are acquired to create a data training set for use in a machine-learning process. Head-mounted or neck-mounted wearable devices are equipped with one or more camera(s) or acoustic device(s) in communication with a data processing system. The cameras capture images of contours of the users face from either the cheekbone or the chin profile of the user. The acoustic devices transmit and receive signals to calculate a representation of the skin deformation. A data processing system uses the images, the profile of the contours, or skin deformation to track facial movement or to reconstruct facial expressions of the user based on the data training set.

Systems and methods are disclosed for driver assistance. In one example a method for an advanced driver alert system for a vehicle comprises monitoring, via a user-facing camera, facial movement of a user over time. The method includes determining a plurality of facial movement metrics based on the facial movement over an adjustable time window, transforming the plurality of facial movement metrics into a machine readable representation of the adjustable time window, and determining, via a code reading model, one or more user requests based on the machine readable representation of the facial movement metrics. The method includes executing one or more user assistance operations based on the determined user requests.

The present invention provides an emotion recognition method and device for road and bridge engineering scenarios, relating to the field of emotion recognition technology. The present invention pre-processes the acquired user's facial video and physiological information, then performs feature extraction to obtain the user's facial motion units and physiological features; then selects and fuses the optimal feature subset of the user's facial motion units and physiological features; and finally, based on the multi-channel feature inverse reasoning of a dynamic Bayesian network and an interpretable emotion recognition model, obtains the mapping relationship between the fused optimal feature subset and the emotion component, thereby obtaining the user's emotional state recognition result. Compared with the existing technology, the emotion recognition system of the present invention is more reliable and robust, and can achieve multi-level, all-round, and stable emotion recognition of users in the complex environment of road and bridge engineering.

The invention provides a multi-source single-graph digital human generation method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a to-be-processed image, a driving audio and a driving video; performing face detection processing on the to-be-processed image to obtain a first face image; determining an audio driving result based on the first face image and the driving audio; performing face detection processing on the driving video, and determining a face video containing a second face image; and performing fusion processing based on the face video and the audio driving result to generate a face image video of the digital human. According to the embodiment of the invention, the defects that the face action and the speaking audio of the generated digital human are not matched and the accuracy is relatively low in the prior art are overcome, the single-image digital human with high matching degree between the audio and the face action can be generated by combining the to-be-processed image, the driving audio and the driving video, and the single-image digital human has diversity and flexibility.

The invention relates to a face forgery identification method, a face forgery identification device, equipment and a medium. The method comprises the following steps: determining a first frame image from a to-be-identified video; wherein the first frame image is a frame image of which the face action amplitude change exceeds a threshold value; acquiring first optical flow motion information of the first frame image and second optical flow motion information of the second frame image; wherein the sampling time of the second frame image is before the sampling time of the first frame image; performing gradient calculation on the first frame image to obtain first gradient information corresponding to the first frame image; and based on the first optical flow motion information, the second optical flow motion information and the first gradient information, obtaining a face forgery identification result of the to-be-identified video.

The invention provides an intelligent expression generation method and system based on a facial motion unit, and belongs to the technical field of intelligent generation, and the method comprises the steps: determining a unit data set of each facial motion unit of a corresponding individual based on a multi-modal facial sample; inputting the phoneme codes into an expression predictor to obtain a prediction data set of each facial motion unit; establishing a mapping relationship based on the unit data set of each facial movement unit according to different individuals in various expression states, and establishing a neutral relationship between each facial movement unit and the corresponding expression state in combination with a deviation set determined based on the prediction data set and the unit data set; establishing a facial motion unit expression model; and searching required motion features and parameter settings from the established facial motion unit expression model, and performing dynamic adjustment based on physical constraints of facial muscle motion and natural transition constraints of expression changes to generate expression animations for interaction. And a high-precision, high-reality and controllable expression generation effect is realized.

The invention discloses a digital human video rendering method and device and a medium, and relates to the crossing field of computer graphics and generative adversarial networks, and the method comprises the steps: constructing a generative adversarial model based on a generative adversarial network of a single fusion architecture; performing multi-modal preprocessing on the original audio and video data based on the digital human reference image; extracting dual-granularity speech features through a speech feature extraction module of a generative adversarial model, and fusing the dual-granularity speech features; determining a local deformation field of the digital human reference image in a UV parameterized space based on a reference key point corresponding to the digital human reference image and the fused speech features; sampling the identity texture of the digital human reference image to generate a target digital human face image; and verifying the digital human face image based on multiple scales through a discriminator of a generative adversarial model. Through dual-granularity speech feature fusion and generator multi-resolution injection, deep alignment of speech semantics and facial actions is realized.

Utilization of windowing to set a region of interest (ROI) of a camera used for tracking facial expressions. In one embodiment, a system includes an inward-facing head-mounted camera that captures images of a region on a user's head utilizing a sensor that supports changing of its ROI. The system also includes a computer that detects, in a first subset of the images, a first sub-region in which changes due to a first facial movement reach a first threshold and reads from the camera a first ROI that covers at least a portion of the first sub-region. The computer detects, in a second subset of the images, a second sub-region in which changes due to a second facial movement reach a second threshold, and then reads from the camera a second ROI that covers at least a portion of the second sub-region, with the first and second ROIs being different.

The present disclosure relates to a device and method for generating avatar lip-sync animation based on multimodal biosignals, The device comprises a multimodal data collection unit configured to collect data including biosignal data including brain waves when a user imagines speaking and image data; a preprocessing unit configured to preprocess the multimodal data; a feature extraction unit configured to extract feature vectors including the user's biosignal feature and facial feature from the preprocessed multimodal data; an avatar generation unit configured to generate an avatar; a lip-sync reconstruction unit configured to predict the mouth shape and facial movement when the user imagines speaking by inputting the extracted feature vectors to a pre-prepared lip-sync reconstruction model; and a lip-sync animation implementation unit for implementing an avatar lip-sync animation by applying the mouth shape and facial movement predicted by the lip-sync reconstruction unit to the avatar generated by the avatar generation unit.

The invention discloses a method for matching facial expressions and actions of digital cat puppet images through human voice, which comprises the following steps of: S1, acquiring voice data of a user, and preprocessing the voice data; s2, converting the preprocessed voice data into text data, performing sentiment analysis, and extracting sentiment information; s3, constructing an action library, pre-defining various facial actions of the digital cat puppet, and distributing an emotion label for each action; according to a sentiment analysis result, selecting proper facial expressions and actions for mapping; and S4, generating a corresponding facial action according to a mapping result, and rendering the generated facial action to a model of the digital cat puppet in real time. According to the method, dynamic interaction between a virtual character and user voice input is realized through voice emotion driven digital cat puppet face action matching and real-time rendering, based on voice preprocessing and emotion analysis technologies, emotion information in voice can be accurately extracted, and the naturalness and coherence of digital cat puppet face actions are ensured.

This application provides a method, apparatus, and electronic device for detecting attacks on identity authentication, relating to the field of identity authentication technology. The method for detecting attacks on identity authentication includes: performing specified segmentation processing on target audio and target video to obtain multiple data segments, wherein the multiple data segments include various audio segments and various video segments, and the target audio and target video are content recorded during user authentication based on reading specified verification text; extracting speech features from each audio segment to obtain feature vectors for each audio segment, and extracting facial motion features from each video segment to obtain feature vectors for each video segment; determining the deviation result corresponding to each target data segment of a specified media type based on the obtained feature vectors; and determining the detection result based on the deviation result corresponding to each target data segment. Therefore, this solution can improve the accuracy of attack detection.

The application discloses a speech-driven lip shape generation method, device and equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: extracting facial motion parameters and face identification features based on target image frames of a video sequence; encoding a driving audio sequence to obtain a time sequence audio feature sequence; performing prediction based on the facial motion parameters and the time sequence audio feature sequence to obtain target implicit key point expression coefficients; and generating target lip shape video frames based on the target implicit key point expression coefficients and the time sequence audio feature sequence. The application further generates target lip shape video frames by calculating target implicit key point expression coefficients, simplifies the overall process of lip shape generation, and improves the efficiency of speech-driven lip shape generation.

The application discloses a face video generation method and system based on a diffusion model and capable of continuously controlling emotions, and the method comprises the following steps: obtaining a speaker face video, performing 3D face reconstruction on video frames, and extracting expression vectors; a corresponding mapping relationship between the extracted expression vectors and emotion labels is established, an emotion editing condition generator is constructed and trained, and corresponding expression vectors are generated according to the emotion labels; a diffusion model is constructed and trained, noise frames, identity frames and motion frames are spliced in the channel dimension to serve as the input of the diffusion model, and time embedding, audio embedding and expression embedding are fused for regulation and control; a reference feature network is constructed, and features output by each layer of the reference feature network are integrated into the diffusion model based on a cross attention and a spatial attention mechanism; and the face video frames output by the diffusion model are subjected to super-resolution processing to obtain a speaker face video. The application can explicitly and continuously edit the emotions of a speaker face video and realize fine-grained control.

The invention relates to the technical field of face recognition, and discloses a fatigue driving detection method and device, computer equipment and a storage medium, and the method comprises the steps: collecting video stream data in a driving process, and employing a preset target detection model to position an initial face region based on the video stream data; fitting face feature points based on the initial face region, constructing a face feature triangle based on the face feature points, constructing a face feature vector based on the face feature triangle, and constructing a driver state analysis data set according to a time sequence; based on the state analysis data sets and a preset sliding window, obtaining facial feature vectors in all the state analysis data sets, and projecting all the facial feature vectors to a preset face projection reference plane to obtain a facial motion feature point set; and calculating a facial motion information entropy based on the facial motion feature point set to detect the driving state of the driver. The problems that the driving state of a driver cannot be detected in real time in a complex scene and the detection accuracy is low are solved.

The invention discloses a facial stimulation waveform generation method and device based on visual feedback, and the method comprises the steps: collecting a neuroelectric signal of the motion of a healthy side surface of a patient in real time through an implanted neural electrode, decoding the neuroelectric signal, and obtaining an electrode facial motion parameter; synchronizing, acquiring a facial movement image of the healthy side of the patient in real time by using an image acquisition device, and identifying the facial movement image to obtain image facial movement parameters; iteratively optimizing the decoding process of the electroneurographic signal based on the feedback of the image facial motion parameters, and outputting the facial motion mode of the unhealthy side; based on the face movement mode of the unhealthy side, a matched face stimulation waveform is generated and used for stimulating the affected side. According to the method, the decoding process of the electroneurographic signals is iteratively optimized based on the feedback of the image facial movement parameters, so that the finally obtained facial movement mode tends to be real, the generated facial stimulation waveform matching is better, the stimulation on a patient is more effective, and the curative effect is better.

Implementations described herein relate to methods, systems, and computer-readable media to generate animations for a 3D avatar from input video and audio captured at a client device. A camera may capture video of a face while a trained face detection model and a trained regression model output a set of video FACS weights, head poses, and facial landmarks to be translated into the animations of the 3D avatar. Additionally, a microphone may capture audio uttered by a user while a trained facial movement detection model and a trained regression model output a set of audio FACS weights. Additionally, a blending term is provided for identification of lapses in audio. A modularity mixing component fuses the video FACS weights and the audio FACS weights based on the blending term to create final FACS weights for animating the user's avatar, a character rig, or another animation-capable construct.

An electronic device is provided. The electronic device includes a camera, a display, and a processor configured to obtain a first image including one or more external objects by using the camera, display to output a three-dimensional (3D) object generated based on attributes related to a face among the one or more external objects using the display, receive a selection of at least one graphic attribute from a plurality of graphic attributes which can be applied to the 3D object, generate a 3D avatar for the face based on the at least one graphic attribute, and generate a second image including at least one object reflecting a predetermined facial expression or motion using the 3D avatar.

The present application provides a method and related equipment for debugging facial movements of a bionic robot, and relates to the field of intelligent robot technology. The present application can automatically adjust the motor movement amplitude of a bionic robot to be debugged by using a manually fine-tuned reference bionic robot as a reference object, ensuring that the reference bionic robot and the adjusted bionic robot to be debugged can directly display the same or similar facial expression effects when performing the same facial movements, thereby effectively reducing the manual adjustment workload and the deviation in the robot's facial movement performance during the robot mass production process, and improving the robot's adjustment efficiency and the consistency of the robot's facial movement performance, thereby meeting the timeliness requirements and facial movement performance consistency requirements during robot mass production.

The invention relates to the field of image reconstruction, and provides a face image cross-modal modeling method and device based on voice features. The method comprises the following steps: for a to-be-reconstructed object, collecting real-time voice data of the to-be-reconstructed object; identity information of the to-be-reconstructed object is confirmed based on the real-time voice data, and real-time voice features of the to-be-reconstructed object are extracted from the real-time voice data; selecting a reference face model matched with the identity information from a reference face model library; predicting a real-time face motion state of the object to be reconstructed in combination with the real-time voice features, and mapping the real-time face motion state to a face motion space of the reference face model to form a motion state sequence of the reference face model; and adaptively generating a face image of the to-be-reconstructed object based on the motion state sequence. The identity feature stability of cross-modal modeling, the synchronization rate between face images and voice changes and the quality of generated results can be improved in diversified application scenes and large-scale user environments.

The invention discloses a face-driven model training method and device, equipment, a medium and a product. The method comprises the following steps: extracting a target face motion coefficient from three-dimensional model training data, wherein the face motion coefficient is used for representing face motion information corresponding to the three-dimensional model training data; performing motion synthesis on audio training data corresponding to the three-dimensional model training data through a first generator of a cyclic generative adversarial network to obtain a generated face motion coefficient, the first generator at least comprising a U-Net network; the generated face motion coefficient and the target face motion coefficient are discriminated through a first discriminator of a cyclic generative adversarial network, a first discrimination result is obtained, and the first discriminator at least comprises a pyramid network; and iteratively updating model parameters of the initial face driving model based on the first judgment result to obtain a target face driving model. According to the scheme provided by the invention, the training efficiency of the face driving model is improved while the training precision of the face driving model is improved.

The application provides a real-time digital human video generation method and device, electronic equipment and storage medium, and relates to the technical field of digital human video generation. The method can realize real-time and low-computing-cost digital human video generation by using a predefined face motion template and a material database. The computationally intensive tasks such as design and optimization of the face motion template, customization of the digital human template, and generation of the face motion material are completed in the pre-production stage, which can greatly reduce the computing burden of the real-time digital human video generation stage, simplify the process of the real-time generation stage, and accordingly reduce the performance requirements of the hardware device. The method does not need to rely on expensive computing resources such as high-end GPUs, thereby significantly reducing the overall computing cost and hardware investment of the real-time digital human video generation. Meanwhile, the face motion material obtained in the pre-production stage can be reused, which further improves the resource utilization efficiency and reduces the long-term operating cost.

The application discloses an intelligent vocal music learning and assisting method and system based on artificial intelligence, and relates to the technical field of vocal music teaching.The method comprises the following steps: constructing a teacher pronunciation expression database comprising original pronunciation audio of multiple teachers, frame-level aligned real face videos and international phonetic transcription sequences, and generating teacher 3D face animation data for associated storage; acquiring learner audio data and extracting audio feature vectors; selecting the most similar target teacher based on audio feature similarity; inputting the learner audio into an audio-driven 3D face animation generation model to generate learner 3D face animation data; extracting the key frame sequences of the learner and the teacher for pronunciation actions; performing three-dimensional model difference calculation on the corresponding key frames to obtain frame-level difference quantization data, and generating visualized difference images.The application visualizes abstract vocal music pronunciation skills into intuitive 3D facial movements, realizes precise personalized guidance highly consistent with the individual characteristics of learners, and improves teaching efficiency.

Provided in the present application are a video generation method and a related apparatus. The embodiments of the present application can be applied to various scenarios such as artificial intelligence and computer vision. An embodiment of the present application comprises: first acquiring a face image and a video frame sequence comprising facial motion information of a talking face; then, extracting facial feature information from the face image, and extracting from the video frame sequence head pose features and facial expression features of the talking face; then, performing rendering on the basis of the facial feature information, head pose feature information and facial expression feature information, so as to obtain target frame images; and finally, on the basis of the target frame images, generating a target synthetic video, facial features of the target synthetic video being the same as facial features of the face image. The method provided by the embodiments of the present application introduces a video frame sequence comprising facial motion information of a talking face, so as to add to a static face image dynamic information missing during talking, thereby improving the quality of a target synthetic video.

The embodiment of the invention discloses a video generation method and device, computer equipment, a medium and a program product, and the method comprises the steps: obtaining a face corresponding relation and an original video, obtaining face movement tracks corresponding to all historical faces included in previous i-1 original video frames, and according to a plurality of face movement tracks, obtaining a plurality of original video frames; predicting a prediction area corresponding to each historical face in the i-th original video frame, detecting the i-th original video frame to obtain a detection area of the target face in the i-th original video frame, determining a target area of the target face in the i-th original video frame according to the detection area and the plurality of prediction areas, and fusing the reference face to the target area of the ith original video frame based on the indication of the face corresponding relation, and generating a target video. Therefore, the face changing accuracy is improved, and the face changing stability is guaranteed.

The invention discloses a non-contact real-time heart rate detection method and system based on rPPG and wavelet enhancement, and belongs to the technical field of medical health. Face video signals are collected through the camera, a sensor or an electrode does not need to be worn, real non-contact physiological signal detection is achieved, and the comfort level and use convenience of a user are improved; discrete wavelet transform denoising and enhancement processing is introduced on the basis of pulse signals extracted by a POS algorithm, and high-frequency noise caused by illumination variation, camera quantization errors and facial movement is removed through multi-scale decomposition and threshold filtering, so that the signal-to-noise ratio of the pulse signals is remarkably increased; the fast Fourier transform is combined with a sliding window mechanism, so that real-time updating and continuous calculation of the heart rate can be realized; the signal after wavelet enhancement is clearer in periodicity, so that the heart rate calculation error is remarkably reduced, a dynamic visual interface of a heart rate change curve and a pulse waveform is provided, a user can observe physiological state changes in real time, and good interactive experience is achieved.

Systems and methods generating an animation ng corresponding to a pose of a subject include accessing image data corresponding to the pose of the subject. The image data can include the face of the subject. The systems and methods process the image data by successively analyzing subregions of the image according to a solver order. The solver order can be biologically or anatomically ordered to proceed from subregions that cause larger scale movements to subregions that cause smaller scale movements. In each subregion, the systems and methods can perform an optimization technique to fit parameters of the animation rig to the input image data. After all subregions have been processed, the animation rig can be used to animate an avatar to appear to be performing the pose of the subject.

The invention relates to the technical field of artificial intelligence and human-computer interaction, and discloses a rehabilitation training adjustment method and system based on multi-modal feature fusion and a storage medium, and the method comprises the steps: obtaining voice, face and physiological signals, constructing a multi-modal data stream, extracting features, and outputting a speech error classification identifier and a physiological signal quality index; splitting the facial action net displacement into healthy and affected sides, correcting affected side features based on healthy side features, and constructing visual representation; fusing each feature to output a comprehensive state vector; updating a dynamic baseline in a task gap, mapping a state vector, and screening to obtain an effective action set; calculating a composite reward and storing the composite reward in an experience playback pool to update the strategy network; and based on the action set, re-weighting the large model output probability, and generating a target interaction corpus. According to the method, the state sensing precision is improved by correcting the deviation of the affected side through the uninjured side, and a closed loop of self-adaptive adjustment of the rehabilitation difficulty and safe interaction text generation is realized by combining the dynamic baseline and penalty mechanism optimization reinforcement learning.

A face mask apparatus configured to more closely adhere to the face of the user such that it is fitted to the bridge of the nose to minimize any vision obstruction by a wearer. The face mask is equipped with a nose bridge segment, a mouth segment, and a chin segment, each disposed in serial communication at obtuse angles. A nose bridge support is disposed within a material of the nose bridge segment and is configured to ensure that the mask remains in position during movement of the face. Additionally, a secondary support is present within the mouth segment to ensure the mask is fitted to the face of the user for comfort and to ensure efficacy. The material of the mask is configured to filter bacteria, viruses, pollen, mold, dust, and other debris, and is preferably removably affixed to the face of the user via retention straps.