268 results about "Facial motion" patented technology

Systems, methods, and computer program products are disclosed for generating a common record based on differing source inputs. Generating a common record based on differing source inputs may include receiving via at least one sensor first non-audible signals indicative of verbalization of an individual; interpreting first words of the individual at least in part using the first non-audible signals received from the at least one sensor; receiving second signals generated by a source other than the individual; performing speech recognition on the second signals to interpret second words from the source; and using the first words and the second words to generate a record.

The invention discloses an emotion detection system based on facial recognition, and relates to the technical field of computer vision and emotion calculation. A video stream time sequence analysis module is used for extracting a facial micro-expression image sequence of continuous frames, a time sequence feature vector containing a micro-expression intensity gradient, an illumination robustness coefficient and a facial action unit cooperation feature is generated, and a multi-mode dynamic sensing module is combined to carry out real-time analysis on an emotion classification probability, voice emotion parameters and physiological signals. And the fusion decision module performs dynamic weighted fusion on the multi-modal data based on the scene adaptive weight, and finally generates a comprehensive emotion score. Through multi-modal time sequence modeling and a dynamic weight optimization mechanism, the accuracy and environmental adaptability of emotion recognition are remarkably improved, and real-time perception and accurate decision making of customer emotion are realized in a target scene.

Provided in the embodiments of the present disclosure are a video generation method and a parameter generation model training method. The video generation method comprises: acquiring speech to be processed; inputting an emotion feature of a target object and said speech into a parameter generation model, so as to obtain an expression parameter, wherein the expression parameter is used for describing facial motion information of the target object under the influence of the emotion feature, the parameter generation model is obtained by means of performing training on the basis of a sample emotion feature, sample speech and an expression parameter label that corresponds to the sample speech, and the sample emotion feature and the sample speech are obtained on the basis of a sample video; and inputting an object image of the target object and the expression parameter into a video generation model, so as to obtain a target video of the target object. By means of generating an expression parameter on the basis of an emotion feature and speech to be processed, and further generating a target video on the basis of the expression parameter, diversified emotion information is fused into the target video while ensuring the synchronization between speech and expression in the target video, thereby improving the accuracy and vividness of the target video.

The invention discloses an interpretable multi-modal information system for postoperative chronic pain assessment, which is characterized by comprising a facial expression algorithm framework for acquiring dynamic change of a facial expression facial action unit (AU) and a physiological signal algorithm module for receiving physiological signals, and the multi-modal fusion framework is used for fusing the facial expression algorithm framework and the physiological signal algorithm module. The method has the effects of accuracy, individuation and interpretability.

The invention discloses an audio and video depth forgery detection method based on quality perception and multi-scale alignment, and the method comprises the following steps: coding a synchronous audio and video sequence, and obtaining a frame-level visual feature, a facial action unit and a phoneme-level voice representation; a visual quality evaluation module is introduced to generate a spatial reliability mask, and quality weighting is carried out on the visual features; designing a global-local multi-scale cross-modal alignment mechanism, performing bidirectional cross-attention modeling on voice and face dynamic synchronization globally, and performing physiological coupling alignment on phonemes and face action units locally; and an uncertainty perception reasoning and calibration scheme is provided, adaptive temperature scaling is carried out according to quality and consistency, and uncertainty calibration is carried out by self-supervision loss. According to the method, the problems of insufficient robustness and excessive self-confidence misjudgment of an existing method in a low-quality video and high-synchronization counterfeit scene are solved, and the cross-dataset generalization capability and the actual deployment reliability are remarkably improved.

The invention discloses a personalized emotion automatic detection method based on a patient before and after an operation, and the method specifically comprises the following steps: 1, carrying out the preprocessing of a patient expression video; 2, discovering change of a human face action unit related to the emotion AU intensity difference; 3, humanized threshold value selection based on the emotion and human face action unit relation; after non-positive face frames of the video in the step (1) are removed, OpenFace mature toolkit software is used for intensity prediction and existence detection of a face action unit; and 4, judging a personalized emotion evaluation application scene through a threshold value of the AU. According to the method, data support such as an accurate emotion change trend chart is provided for clinical emotion management and operation recovery monitoring, meanwhile, the emotion state of a patient is scientifically and quantitatively displayed for a doctor, the doctor is assisted in performing subsequent decision diagnosis, and the treatment effect is assisted to be improved.

The invention provides a student classroom attention analysis system based on facial multi-modal emotion fusion, aiming at solving the problem that an effective micro-expression recognition method for judging the attention analysis of students on teaching content is lacked in the current intelligent education classroom environment. The invention mainly solves the technical problem of providing a facial micro-expression recognition method based on multi-modal fusion aiming at the deficiency of facial micro-expression recognition analysis in an intelligent teaching classroom at present, and the facial micro-expression recognition method comprises the following steps of: dynamically enhancing local detail features of a facial image by utilizing coding features of a facial action unit; according to the method, the image difference subtle influence caused by low intensity of micro-expression motion is reduced, whether the students hide the real emotional states by using micro-expressions or not is analyzed, the attention condition of the students in class is analyzed according to the recognition condition of the micro-expressions and information such as the usual performance of the students and the teacher-student communication frequency, and then the teaching content of teachers is dynamically adjusted.

The invention relates to the technical field of data processing, in particular to a method for driving emotion interaction of an intelligent device based on multi-modal understanding, and aims to eliminate illumination and noise interference and output a standardized face video stream, an effective voice segment and a touch thermodynamic diagram through an environment adaptive acquisition module. The feature extraction module extracts facial action optical flow features, voice Mel-frequency cepstral coefficient vectors and tactile pressure gradient parameters. The cross-modal correlation model adopts a tensor decomposition algorithm to calculate a space-time correlation matrix of visual and voice features, and the tactile feature weight is dynamically adjusted in combination with environmental parameters. According to the response strategy, an intervention scheme is retrieved based on a graph database, emotion confirmation statements, guide statements and behavior suggestions are fused to generate multi-mode response, and PID adjustment of the temperature control device and tactile pulse output of the vibration device are synchronously driven. And the feedback evaluation module verifies the emotion recognition consistency through a Pearson's correlation coefficient, triggers conflict sample separation storage and model increment training, and realizes closed-loop optimization.

The invention discloses a facial action unit identification method based on CLIP and multi-mode mask prompt learning. The method comprises the following steps: 1, carrying out facial action unit identification on the basis of the CLIP and the multi-mode mask prompt learning; the method comprises the steps that 1, a multi-modal shared AU prompt (AU-prompt) and an attention mask are designed based on an AU prior position, the AU-prompt comprises a plurality of learnable tokens, the attention mask is designed according to prior positions of different AU facial muscle changes, and a specific AU attention mask is obtained; 2, in a multi-mode prompt learning process, introducing an attention mask of a specific AU into different layers of a CLIP model text and an image encoder for prompt learning, and learning local detail features of the AU and global association features among different AUs to obtain local features and global features; and step 3, fusing the local features and the global features, calculating loss with the text features, and finally realizing AU identification. According to the method, the facial action unit can be effectively recognized under the condition that the labeled data is limited.

The invention discloses a facial action unit recognition method and system based on multi-modal fusion and text enhancement. The method comprises the following steps: extracting a facial image attribute set by a visual language model, constructing an adaptive text cue word based on the facial image attribute set, and processing the adaptive text cue word by adopting CLIP to obtain a text semantic feature; extracting facial key point features by the facial key point detection model; the facial expression recognition model extracts facial expression features; the image semantic feature fusion module processes the facial key point features and the facial expression features to obtain multi-modal image semantic features; inputting the multi-modal image semantic features and the text semantic features into a multi-modal feature fusion module to obtain multi-modal fusion features after text information enhancement; the recognition of the facial action unit of the facial image is realized. According to the method, the complementarity of cross-modal features and the robustness of joint representation can be enhanced, and the recognition precision of facial actions in a complex scene is effectively improved.

The invention provides a stroke face auxiliary evaluation method and system based on face motion unit detection, and the method comprises the steps: collecting a face motion video of a patient, and recording a start frame and an end frame of each motion in the video; the actions comprise an eyeball horizontal rotation action, a tooth indication action, an eyebrow lifting action and an eye closing action; according to the start frame and the end frame of the horizontal rotation eyeball action, extracting a horizontal rotation eyeball action video, namely an eye rotation action video, and calculating to obtain a severity score of the staring condition of the patient by using a staring scoring algorithm; and respectively extracting videos of the three actions according to starting frames and ending frames of the tooth indicating action, the eyebrow lifting action and the eye closing action, and calculating to obtain a severity score of the facial paralysis condition of the patient by using a facial paralysis scoring algorithm. According to the method, the abnormal conditions of the face of the patient are captured through accurate detection and quantification of various face actions of the patient, the problems that a conventional evaluation mode is low in accuracy, incomplete in evaluation and the like are solved, and an objective, accurate and comprehensive stroke face evaluation effect is achieved.

The invention relates to the technical field of artificial intelligence and medical health, and discloses a pressure detection model training method and device, electronic equipment and a storage medium, and the method comprises the steps: constructing a pressure detection model which comprises a feature extraction layer, a feature fusion layer and a result prediction layer which are connected in sequence; wherein the feature extraction layer comprises an action unit feature recognition network, an emotion feature recognition network, an expression feature recognition network and a recessive feature recognition network; a training set is constructed, each data sample in the training set is a face image of the recognition object, and the data samples comprise marks of face action unit features, face emotion features, face expression features and face image coding features; and inputting the training set into a pressure detection model for training to obtain the pressure detection model. The method greatly reduces the pressure detection cost, is high in detection result efficiency, improves the operability, and is convenient for a user to use. And the practicability is high.

The invention belongs to the technical field of man-machine interaction and robot control, and particularly relates to a man-machine physical twin system based on multi-modal video analysis and adaptive mapping and a control method, and the system comprises a multi-modal collection unit which is used for collecting original video streams of human body actions, expressions and voices, and compensating and repairing dynamic shielding; the semantic analysis unit comprises an action analysis module used for extracting a human skeleton key point set object interaction track from the original video stream, and an expression / voice analysis module used for outputting a facial action unit coefficient and a voice text with an emotion label; the self-adaptive mapping unit is used for establishing human body-robot joint kinematics mapping, distributing control bandwidth in real time according to task types and degrading non-key modes based on network states to guarantee action continuity, and non-contact action capture is achieved through a pure vision scheme by eliminating dependence on a wearable sensor; and the problems of failure and the like of a traditional visual scheme under shielding and illumination variation are solved.

The invention relates to a multi-posture portrait animation generation method. The method comprises the following steps: acquiring a portrait image and converting the portrait image into a three-dimensional deformation model; acquiring audio data, inputting the audio data into an audio-driven facial motion parameter extraction network, and calculating facial motion parameters according to the facial expression parameters; calculating a head posture motion parameter and a hand posture motion parameter according to the head posture parameter, the hand posture parameter and the audio data by using a posture generation variational auto-encoder; extracting appearance features of the portrait image, and performing standard key point estimation on the portrait image to generate standard key points; mapping the head posture parameter, the hand posture parameter, the facial expression parameter, the facial motion parameter, the head posture motion parameter and the hand posture motion parameter into parameter representation under a 2D view; and finally, inputting the mapped parameter representation, the standard key points and the appearance features into an image generator, and generating a multi-pose portrait animation containing hand postures and dynamic expressions.

The invention discloses a cognitive impairment screening method and system based on face analysis. The cognitive impairment screening method comprises the following steps: selecting a video clip with a preset duration for a user to watch, wherein the video clip comprises a plurality of smile points; setting a corresponding time window according to the appearing time of each laughing point; based on spontaneous facial expressions after the user watches the video, facial motion features of the user in each time window are collected, and the facial motion features comprise values of a plurality of facial motion units of the user; inputting the facial motion features of the user in each time window into a preset model to output a cognitive impairment screening result; wherein the preset model carries out model training based on historical big data, so as to form a mapping relation from the facial movement features to the cognitive impairment screening result. Therefore, by utilizing the method, the cognitive impairment can be screened in a diversified scene, the dependence on professionals is reduced, a user does not need to be wounded, and the method is particularly suitable for early screening of the cognitive impairment.

The invention provides a method for tracking a speaker based on a panoramic camera and a close-up camera. The method comprises the steps that the panoramic camera shoots all people into a panoramic picture; based on a sound source of a speaker and / or analyzing facial actions of all people in the panoramic picture, positioning face position information of the speaker in the panoramic picture; based on the position relation between the panoramic camera and the close-up camera and the face position information of the speaker, the angle and the focal length of the close-up camera are determined, and close-up shooting is carried out on the speaker; the face position of the speaker in the close-up picture is analyzed, and the close-up camera is finely adjusted, so that the speaker is centered in the close-up picture. The position of the speaker is positioned by combining face detection and voice recognition technologies, and the close-up camera picture is triggered to track the close-up picture of the speaker, so that the conference control operation is avoided, and the intelligence of the conference terminal is remarkably improved.

The invention relates to an interface display method and device, equipment and a storage medium, and relates to the technical field of multimedia. The method comprises the steps of collecting music in a target space; acquiring mirror moving parameters based on at least one of rhythm features of the music and facial information, wherein the facial information comprises at least one of facial expressions and facial actions of an object in the target space; and displaying a live broadcast interface of the target space based on the mirror operation parameter, the live broadcast interface including a mirror operation effect corresponding to the mirror operation parameter. The mirror moving parameters are not artificially determined based on experience, but are obtained through real-time reasoning based on the music rhythm and the face information of the object during live broadcasting, so that the accuracy of the mirror moving parameters is obviously high, and the mirror moving special effect displayed by the live broadcasting interface is more visual and accurate. Moreover, according to the method, manual participation is not needed, a machine can automatically determine the accurate mirror operation parameters and display the live broadcast interface based on the determined mirror operation parameters, and the man-machine interaction efficiency is further improved.

The invention relates to the field of digital people, and discloses a method for driving a digital population pattern by voice, which comprises the following steps: receiving a voice signal and extracting a time-frequency feature, generating a personalized voiceprint feature based on the voice signal, generating a digital population pattern synchronous with the voice signal by using nonlinear dynamic modeling, and generating a digital population pattern synchronous with the voice signal. And precise synchronization of the mouth shape and the voice signal is ensured through a time sequence modeling method. Furthermore, facial expression generation is physically constrained through a Hamiltonian system, and it is ensured that the generated facial action conforms to the biomechanical law. Finally, a digital population type video is generated and output according to a synchronization result, and personalized, high-precision synchronous and natural facial expressions are achieved. The method is suitable for the fields of medical science popularization, virtual assistants and the like, and can improve the video generation efficiency and enhance the user experience.

The invention provides a facial action unit recognition and model training method and device and electronic equipment. The method comprises the steps of obtaining a training sample set and inputting the training sample set into an initial facial action unit recognition model; wherein the training sample set comprises multiple frames and face action unit (AU) real labels corresponding to each frame; performing multi-level feature extraction on each frame by the facial action unit recognition model, generating a visual token according to the extracted features, performing semantic reasoning based on the visual token and a prompt text of a facial action unit recognition task, and generating an AU prediction probability corresponding to each frame; determining a loss value of the loss function according to the AU real label and the AU prediction probability, and reversely adjusting model parameters of the facial action unit recognition model based on the loss value; when a preset training condition is met, training is stopped, and a trained facial action unit recognition model is obtained. According to the invention, semantic reasoning can be carried out by using visual features to complete AU identification, and the accuracy of a facial action unit detection result can be improved.

The invention provides a dynamic fine-grained expression detection method and system based on computer vision. The method comprises the following steps: acquiring a facial action video; taking the face action video as input, and executing a continuous feasible frame extraction algorithm to obtain all continuous feasible frame sequences in the video; and for each continuous feasible frame sequence, executing a dynamic action unit detection algorithm to obtain dynamic action unit strength, a static action maximum frame and a static action minimum frame of the left and right side parts corresponding to the current sequence, and completing detection of expression actions of the left and right side parts. By adopting a continuous feasible frame extraction algorithm and a dynamic action unit detection algorithm, the problems that a conventional expression detection method is difficult to accurately reflect facial movement only through a single-frame facial image and a conventional discrete expression detection result is difficult to accurately describe specific facial movement are solved; effects of efficient detection and accurate quantification of facial expressions are realized.

The present invention provides an emotion recognition method and device for road and bridge engineering scenarios, relating to the field of emotion recognition technology. The present invention pre-processes the acquired user's facial video and physiological information, then performs feature extraction to obtain the user's facial motion units and physiological features; then selects and fuses the optimal feature subset of the user's facial motion units and physiological features; and finally, based on the multi-channel feature inverse reasoning of a dynamic Bayesian network and an interpretable emotion recognition model, obtains the mapping relationship between the fused optimal feature subset and the emotion component, thereby obtaining the user's emotional state recognition result. Compared with the existing technology, the emotion recognition system of the present invention is more reliable and robust, and can achieve multi-level, all-round, and stable emotion recognition of users in the complex environment of road and bridge engineering.

The invention relates to the technical field of 3D facial motion generation, and discloses a facial motion generation method, device and equipment and a storage medium, and the method comprises the steps: responding to multimedia data and emotion data inputted by a user, and determining a voice sequence according to the multimedia data; acquiring facial motion data, and adding noise according to the facial motion data to obtain facial noise motion; and inputting the voice sequence, the facial noise motion and the emotion data into a facial motion generation model to obtain a facial expression video, the facial expression video including a single-emotion facial expression video or a facial expression video with continuous emotion transformation. According to the method and the device, various and vivid facial expressions can be generated, and flexible and continuous emotion control is realized.

The invention relates to a human body state detection method, system and device based on facial actions and a medium, and the method comprises the steps: obtaining an image sequence of a human face, carrying out the human face detection and mark point positioning of the image sequence, and obtaining the coordinates of a face mark point; performing micro-expression action extraction on the image sequence based on the facial mark point coordinates to obtain facial micro-expression action features; performing time sequence analysis on the facial action features to obtain time sequence facial action features; performing multi-scale illumination adjustment on the time sequence facial action features to obtain illumination facial action features; performing multi-modal analysis on the illumination facial action features through a pre-trained long-short term memory network to obtain comprehensive human body state features; and performing real-time state analysis based on the comprehensive human body state characteristics to obtain a human body state detection result. According to the method, different types of facial action features can be comprehensively processed, and the reliability of a detection result is enhanced.

The invention discloses a digital human face action generation method and device and electronic equipment, and relates to the technical field of computers. The method is applied to a server and comprises the following steps: receiving driving information and prompt action information sent by a client; inputting the driving information and the prompt action information into a facial action generation model, and outputting a facial action coefficient sequence; taking the facial action coefficient sequence as a facial action curve parameter, and generating a digital human facial action video frame; and sending the digital human face action video frame to a client. According to the embodiment of the invention, the driving information can be converted into the face action sequence based on the prompt action information sent by the client, so that the digital human has the ability of simulating different styles of figures to speak, the user can realize a customized and personalized digital human face driving effect through the prompt action information, and the use requirements of different users are met.

The invention relates to the technical field of movie and television animation production, in particular to a movie and television animation character expression generation method and system combined with motion capture, and the method comprises the steps: firstly collecting a real-time motion data set of a target actor through motion capture equipment, including face and body motion data subsets; the facial action data is composed of facial key point displacement data with multiple continuous timestamps, then calling a preset expression mapping model to decouple the facial action data to obtain a basic and personalized expression feature set, then adjusting the basic expression feature set according to preset emotion parameters of an animation role, generating an emotion adaptive expression feature set, and generating a personalized expression feature set according to the emotion adaptive expression feature set; the method comprises the steps of obtaining a personalized expression feature set, fusing with the personalized expression feature set to obtain a target expression feature set, finally driving a three-dimensional face model to generate an expression animation by using the target expression feature set, and binding body action data to a three-dimensional body model to generate a complete character animation, thereby improving the authenticity and expressive force of animation character expressions.

The invention discloses an emotion recognition method and system based on facial expression semi-supervised learning, and the method comprises the steps: converting video data into a video frame sequence, and carrying out the preprocessing of the video frame sequence; performing facial key point labeling on the video frame sequence by using OpenPose, and constructing an expression data set with key point labeling; selecting a part of key frames from the expression data set with key point labels, performing manual labeling and auditing of the action units in combination with the facial action units and the relationship between the facial key points and the action units, and establishing an expression and emotion data set; constructing a semi-supervised learning expression recognition network based on manual labeling and auditing of a part of facial action unit expert group on the basis of an expression data set with key point labeling and an expression and emotion data set; and inputting the newly collected video data into the semi-supervised learning expression recognition network to obtain an emotion recognition result. According to the method, the labeling difficulty is reduced, and the precision of infant facial emotion recognition is improved.

A method for generating a talking head video includes: obtaining a text and an image containing a face of a user; determining a phoneme sequence that corresponds to the text and includes one or more phonemes; determining acoustic features corresponding to the text according to the phoneme sequence, and obtaining synthesized speech corresponding to the text according to the acoustic features; determining a first mouth movement sequence corresponding to the text according to the phoneme sequence, and determining a second mouth movement sequence corresponding to the text according to the acoustic features; creating a facial action video corresponding to the user according to the first mouth movement sequence, the second mouth movement sequence and the image; and processing the synthesized speech and the facial action video synchronously to obtain a talking head video corresponding to the user.

The invention relates to the technical field of finance, medical health and artificial intelligence, and provides a TTS voice and 3D mouth shape synchronous generation method, device, equipment and medium, a TTS voice and 3D mouth shape synchronous generation model is obtained by using a small sample adaptive training engine and based on transfer learning framework training, and the generalization ability and training efficiency of the model are improved; multi-modal joint coding and a dynamic weight distribution module are used for joint coding, so that the precision of a generated result is improved; a 3D mouth shape dynamic sequence is obtained based on a diffusion-autoregression hybrid generation module, the detail generation capability of a diffusion model and the time sequence coherence of an autoregression model are fused, and the generation precision and real-time performance are improved; and the real-time synchronization module based on cross-modal attention guidance obtains the synchronously adjusted 3D mouth shape parameters and the facial action sequence matched with the emotion and generates a target video, so that multi-modal dynamic fusion is realized, the lip movement-voice synchronization precision is improved, and the authenticity of emotion expression is enhanced.

The invention provides a GAN-based vehicle-mounted expression data construction method, system and device and a vehicle, and relates to the technical field of intelligent driving. A key point detection technology is utilized to analyze and extract a facial action unit and expression intensity and duration thereof to generate a structured action description. And then, calling a generative adversarial network model to generate a synthetic video corresponding to the target expression type, and carrying out dynamic cutting and frame extraction according to a preset interval on the videos to obtain a multi-frame synthetic image. And finally, performing dual-model filtering on the synthesized image to ensure data quality, and executing privacy anonymization processing, thereby generating a high-quality vehicle-mounted expression data set. According to the method, the data acquisition cost is reduced, the diversity and accuracy of a data set are improved, and the performance and reliability of an expression recognition algorithm in an actual vehicle-mounted environment are effectively enhanced.

The invention relates to a face forgery identification method, a face forgery identification device, equipment and a medium. The method comprises the following steps: determining a first frame image from a to-be-identified video; wherein the first frame image is a frame image of which the face action amplitude change exceeds a threshold value; acquiring first optical flow motion information of the first frame image and second optical flow motion information of the second frame image; wherein the sampling time of the second frame image is before the sampling time of the first frame image; performing gradient calculation on the first frame image to obtain first gradient information corresponding to the first frame image; and based on the first optical flow motion information, the second optical flow motion information and the first gradient information, obtaining a face forgery identification result of the to-be-identified video.