222 results about "Facial expression" patented technology

The invention relates to a multi-modal virtual human real-time generation method based on an expression control embedding space, and belongs to the field of artificial intelligence. According to the method, an expression control embedding space is constructed and used for fusing voice semantics, a rhythm structure and multi-dimensional emotion information, and continuous and controllable multi-modal driving vectors are generated. The whole system has an end-to-end linkage mechanism from audio input to expression and action output. Semantic features, rhythm structures and emotional states jointly act on generation paths of lip and upper body postures and expression modalities, and all modal features are fused and expressed in a unified control space through a collaborative coding and time sequence alignment mechanism. And finally, a high-consistency and high-fidelity virtual human video is generated in real time through an output scheduling mechanism. The method has remarkable advantages in the aspects of modal fusion consistency, generation expression naturalness and emotion control flexibility, and can be widely applied to key scenes such as virtual human broadcasting, voice interaction agency and meta-universe digital identity construction.

The invention relates to the technical field of video action detection, and provides a macro-micro expression interval positioning method based on meta-learning, which comprises the following steps: performing face alignment and image size unified processing on an input face expression video frame sequence; performing down-sampling on the processed facial expression video frame sequence based on a pre-trained video understanding model, and extracting time sequence features; dividing and sampling meta-learning tasks according to preset attribute dimensions on the basis of the time sequence characteristics, and constructing a time sequence positioning model at the same time; performing meta-learning training on the time sequence positioning model according to the meta-learning task to obtain a meta-basic model with a generalized spatial-temporal feature representation capability; and carrying out adaptive fine tuning on the element basic model, and using the fine-tuned model to position macro expression and micro expression intervals of the target domain facial expression video. The method can be more easily deployed in various real and complex new scenes, so that the progress of the expression analysis related technology from the laboratory to the practical application is accelerated.

The invention discloses a multi-modal sentiment classification method based on multi-view interaction representation, and relates to the technical field of multi-modal information processing, and the method comprises the steps: collecting and preprocessing the voice, facial expression and text data of a user, and generating a multi-modal data packet; based on the multi-modal data packet, performing feature extraction by adopting a hierarchical attention mechanism to obtain an aligned voice expression text feature sequence, and generating a multi-modal feature flow; performing sentiment classification on the sentiment benchmark result by adopting a time sequence gating network to obtain a service state vector and a conflict view vector; according to the multi-view fusion emotion feature sequence, a time sequence gating network and a multi-mode recognition model are adopted for recognition, and a matched emotion classification result and a natural language reply are generated in combination with the service state. According to the method, sentiment classification is carried out on the sentiment benchmark result by adopting the time sequence gating network, the service state vector and the conflict view vector are obtained, and synchronous description of the sentiment state and the context is realized.

The invention discloses an interactive digital human generation method based on picture and audio synthesis, and belongs to the crossing field of artificial intelligence and computer graphics. According to the method, a full-link process of'feature extraction-model construction-emotion driving-real-time interaction-video output 'can be automatically completed only by uploading a character picture and a section of audio by a user: key point and semantic feature extraction is performed on the picture to obtain face / posture information; performing voice recognition, semantic analysis and emotion recognition on the audio to obtain a semantic tag and an emotion parameter; generating a personalized three-dimensional digital human based on the information, and driving the personalized three-dimensional digital human to generate facial expressions and limb actions which are synchronous with emotions; the user intention is analyzed in real time through natural language understanding and computer vision, and multi-modal interaction is achieved. According to the method, digital human creation can be completed without professional modeling and motion capture equipment, the generation cost is reduced, and the method can be widely applied to virtual anchors, online education, intelligent customer service and movie and television entertainment scenes.

The invention discloses a real-time multi-modal man-machine interaction method and system based on a large model and a storage medium. Collected user interaction information is converted into interaction data in a preset format, the interaction data are input into a full-modal large model, and an output pre-generation result is obtained; secondly, semantic anchor points are marked for reply text information, absolute prediction timestamps of all the semantic anchor points are calculated, semantic time sequence windows are divided, and a global time sequence reference skeleton is constructed; combining and packaging the expression degree-of-freedom sequence and the action degree-of-freedom sequence into a unified expression frame; and locking the rigid segments to the corresponding semantic anchor point timestamps according to the anchor point tags, performing nonlinear filling on the elastic segments between the rigid segments, outputting alignment information, and finally analyzing the alignment information into control instructions to respectively drive a loudspeaker, a facial expression component and a limb motor to execute reply operation. The limitation of dependence of a single mode is broken through, the interaction stability in a complex environment is improved, and meanwhile, the real-time performance of interaction response is improved.

The invention discloses a personnel state analysis method and system based on limb movement recognition, relates to the technical field of limb movement recognition, and determines the movement state of a corresponding limb part based on recognition of each movement-expression combination. And the driving response state of the personnel is determined based on the action state of each limb part, the corresponding part priority and the driving state of the truck, so that the accuracy of the driving response state of the personnel is improved. Therefore, the change event of the facial expressions is determined according to the plurality of facial expressions at different time, and the driving fatigue state of the personnel is determined according to the change event of the facial expressions and the lane changing frequency of the truck; according to the method, the voice interaction event of the person in the driving process is collected, the multiple state key contents are determined according to recognition of the voice interaction event, the state analysis system of the person is determined according to the multiple state key contents, the driving response state of the person and the driving fatigue state, and the accuracy of the state analysis system of the person is improved.

The present disclosure belongs to the technical field of image processing. Provided are a video generation method and apparatus, and an electronic device and a computer program product. The video generation method comprises: extracting a facial identity feature of a target virtual human by using a facial recognition model; extracting an audio feature vector and an emotion prompt word from target audio by using an audio classifier; and using the audio feature vector and the emotion prompt word as control features, merging the control features with a diffusion model, using the facial identity feature to control the output of the diffusion model, and using the diffusion model to generate speech video data of the target virtual human, which speech video data matches the target audio. The technical solution of the present disclosure can generate a speech video stream of a virtual human with natural facial expressions.

The invention discloses a bionic robot facial expression synchronization method and device based on visual perception, equipment and a medium, and relates to the technical field of man-machine interaction, and the method comprises the steps: controlling a target bionic robot to execute a target expression control instruction used for achieving a preset expected robot facial expression, and capturing a robot visual facial expression of the robot; based on the robot visual facial expression and a preset expected robot facial expression, carrying out calibration on the target bionic robot so as to construct a hardware compensation parameter vector; executing a facial expression synchronization operation based on the hardware compensation parameter vector and a pre-trained target expression mapping model by asynchronously running a preset sensing thread and a preset driving thread; the preset sensing thread is used for collecting a target face image in real time and outputting a target steering engine control vector through a target expression mapping model; and the preset driving thread is used for outputting a steering engine control instruction at a preset output frequency based on a preset frame compensation algorithm, the hardware compensation parameter vector and the target steering engine control vector.

The invention relates to the technical field of digital human interaction, in particular to a digital human intelligent question and answer interaction method for a multi-modal visual platform, which comprises the following steps of: receiving multi-modal input of a user, performing privacy protection processing, and performing sensitive field identification, desensitization and hierarchical storage; then semantic analysis and interaction intention recognition are conducted on the processed input, a retrieval request is generated, relevant information is searched in a professional knowledge base based on the retrieval request, meanwhile, source identifiers of entries are reserved, a retrieval result and a user intention are input into a question and answer generation model, and an interaction answer is generated; and adding a corresponding source identifier to the answer to realize content traceability, and finally performing multi-modal output, including voice broadcast, expression action and visual display, through a digital human image, so as to realize safe, credible and intuitive interactive experience.

The invention discloses a speech emotion recognition method based on multi-modal multi-view pseudo-tag fusion, and relates to the technical field of emotion recognition, and the method comprises the steps: firstly, constructing a visual-view pseudo-tag generation module, extracting high-quality facial expression features from a video through employing an optimized OfficientNet-V2 network and a RetinaFace face detector, and carrying out the recognition of the high-quality facial expression features; generating a visual pseudo label through time sequence aggregation; secondly, adopting a semi-supervised learning framework FixMatch to generate a reliable voice pseudo tag for the non-tag voice; secondly, fusing emotional knowledge from multiple visual angles and modes of vision and voice based on a pseudo-tag fusion function, and generating a more accurate and robust fused pseudo-tag; and finally, training a voice emotion recognition model based on a Conformer encoder to perform voice emotion recognition by combining the fused pseudo label data with the labeled voice data. According to the method, the problem of error accumulation in traditional semi-supervised learning is effectively solved, cross-modal emotion complementarity is utilized, and the accuracy and generalization ability of speech emotion recognition are remarkably improved.

Provide a system. 【Solution means】 Means for collecting audio data, Means for collecting video data, Means for preprocessing the collected audio data and video data, Means for analyzing audio, facial expression, and gesture data to extract emotions and intentions, Means for integrating the analysis results and evaluating the negotiation situation, Means for generating advice based on the evaluation results, Means for presenting the generated advice to the user, Means for analyzing non-verbal elements in a residents' briefing or town hall meeting and providing immediate feedback, A system including means for displaying the immediate feedback on the display of a smart device.

1. Name of the product in this design: Voice and Expression Interactive Robot. 2. Purpose of this design: This design is used to control the robot's facial expressions via voice commands. 3. The key design feature of this product is its shape. 4. The image or photograph that best illustrates the design's key points: a 3D model.

A digital human generation method based on a video generation large model includes five steps: S1: receiving an original video signal from a video data source, performing time sequence frame decomposition and key action extraction operations on the original video signal, and forming a preprocessed signal containing human posture features and facial expression features; S2: inputting the preprocessed signal into a pre-trained video generation large model, and generating an initial digital human motion trajectory signal and an appearance rendering signal through the space-time attention mechanism of the model; S3: fusing the initial digital human motion trajectory signal and a preset voice driving signal, and generating a lip shape synchronous control signal by using a cross-modal alignment module; S4: generating a high-real-sense digital human video stream signal by using an adaptive lighting rendering engine according to the appearance rendering signal and the lip shape synchronous control signal; and S5: outputting the high-real-sense digital human video stream signal to a display terminal, simultaneously generating a dynamic detail enhancement feedback signal, and iteratively optimizing the parameter weight of the video generation large model. The digital human generation method based on the video generation large model can solve the problems that it is difficult to eliminate the experience dependence of manual adjustment and optimization, and the die casting parameters cannot be accurately self-optimized.

The invention discloses a mechanical driving system and method for facial expressions, and belongs to the technical field of bionic robots. The system comprises a stress response unit, a mechanical chaos engine and an expression execution mechanism. The stress response unit adopts a design structure that an arc-shaped elastic sheet is in bridge connection with an elliptical ring, converts external mechanical force into nonlinear deformation and realizes automatic resetting; the mechanical chaos engine takes mechanical tolerance as an internal random source, realizes multi-physics field coupling and chaos amplification through a three-stage driving structure, and drives an expression execution mechanism composed of a gradient hardness material, an asymmetric groove array and an intercommunication ring. According to the method, by sensing external force input, a chaos engine is excited to generate unpredictable, natural and rich facial random expressions. Controllable expression of emotional tendency is achieved through a pure mechanical structure, and static and dynamic expressions reflecting the emotional change process can be output. The system is simple and compact in structure, the method randomness is reliable, and the method is suitable for the fields of robots, intelligent toys, emotion counseling, man-machine interaction and the like.

The invention discloses a human-simulated robot facial expression mechanism of a multi-degree-of-freedom micro-driver, which comprises a bottom plate, a supporting disc is fixedly arranged at the top of the bottom plate, a bracket is fixedly arranged on the supporting disc, an adjusting assembly is arranged on the bracket, a supporting frame is fixedly arranged at the top of the bracket, a shell is connected to the supporting frame, and a driving device is arranged on the shell. The shell is covered with an electronic skin layer, a limiting block is fixedly arranged in the electronic skin layer, a limiting cylinder corresponding to the limiting block is fixedly arranged on the shell, the limiting block is arranged in the limiting cylinder in a penetrating mode, a frame is fixedly arranged in the supporting frame, and an air pressure mechanism is arranged in the frame. A controller is fixedly arranged on the bottom plate; according to the humanoid robot facial expression mechanism of the multi-degree-of-freedom micro-driver, the air pump can be controlled to work, auxiliary limiting of the electronic skin layer is achieved by means of the limiting block, and therefore the installation stability of the electronic skin layer is effectively improved.

Evaluating content based on video or audio data acquired during communication. [Solution] The present invention relates to an information processing device 20 for evaluating content displayed on a user's terminal device in online communication, wherein the audio data recorded in the communication includes the user's utterances in response to the explanation of the content, and the video data recorded in the communication includes the video of the user receiving the explanation of the content, and comprises an acquisition unit 23 for acquiring at least one of the audio data or the video data, and an evaluation unit 27 for analyzing at least one of the user's utterances included in the audio data or the user's facial expressions included in the video data acquired by the acquisition unit to determine an evaluation score for the content.

The invention relates to the technical field of computers, and particularly discloses a digital human intelligent hospital guide interaction system and method based on a large model, and the method comprises the steps: judging whether a patient has hesitation in a hospital guide key link or not through the residence time, verifying the demand urgency through the combination of facial expressions, and finally locking the specific department inquiry direction through disease voice. The conversion from passive response to active identification of the potential demand is realized, the identification precision of the potential department inquiry demand of the patient is improved, the potential department inquiry demand of the patient is accurately identified, and fuzzy identification of the digital human hospital guide demand is prevented; medical entity data are extracted and accurately matched with the dynamic department information base, so that the patient does not need to autonomously screen departments, the hospital guide communication cost of the patient is reduced, and low hospital guide efficiency caused by the fact that the patient does not know the departments due to diseases is avoided; and in combination with the real-time state of the department, the generated optimal moving path can adapt to the real-time position of the department and the regional people flow state, so that the dynamic adaptability and practicability of the hospital guide service are improved.

To provide a system capable of effectively training a presentation in an environment close to a real world.SOLUTION: The specification processing unit 290 of the data processing device 12 in the system receives the voice, the facial expression, the gesture, and the slide material input by the user, converts the received voice data into text, evaluates the speaking speed, the volume, and the pause, analyzes the facial expression, the line of sight, and the gesture of the user from the received video data, evaluates the degree of calmness and the degree of confidence of the presentation, analyzes the slide material, evaluates the consistency of the content and the visual effect, generates feedback to the user based on these evaluation results, and transmits the feedback to the terminal of the user.SELECTED DRAWING: Figure 2

The invention discloses a bionic human face robot which comprises a head device, the head device comprises an eye module, the eye module comprises a mounting base, an eyeball component, a first driving assembly and a connecting piece, the first driving assembly is mounted on the mounting base, the driving end of the first driving assembly is in transmission connection with the eyeball component, and the connecting piece is connected with the eyeball component. The mounting seat is arranged on the eyeball part to drive the eyeball part to turn up and down, a center hole is formed in the middle of the back side of the eyeball part, one end of the connecting piece is connected with the mounting seat, the other end of the connecting piece is spherically hinged to the center hole, and the connecting piece is always horizontally arranged to limit the turning stroke of the eyeball part. By means of horizontal arrangement of spherical hinge matching of the connecting pieces, the overturning stroke of the eyeballs can be limited, excessive overturning or separation from the original position is avoided, accurate eye movement is guaranteed, and the simulation degree of facial expressions of the bionic human face robot is effectively improved.

The embodiment of the application provides a kind of based on multi-dimension recognition's vehicle emotion interaction method and device, through innovatively constructing emotion fusion identification model, by integrating facial expression, speech emotion, driving behavior and physiological state characteristics, the accurate judgment of driver emotion is realized.Design scene-based adaptive interaction strategy, combined with external environment data and danger level, establish interaction trigger threshold for intelligent matching.Introduce interaction effect evaluation mechanism, through online learning module, continuously optimize interaction strategy model, realize the dynamic adjustment of personalized interaction content.The method effectively solves the deficiency of traditional technology in emotion recognition, interaction strategy and effect evaluation, significantly improves the intelligent level and user experience of vehicle emotion interaction.

The present disclosure relates to systems, methods, and non-transitory computer- readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations. 20 24 20 05 05 28 J an 2 02 4 A B S T R A C T O F T H E D I S C L O S U R E 2 0 2 4 2 0 0 5 0 5 2 8 J a n 2 0 2 4 o r a n i m a t i o n s .

This application discloses a method, apparatus, electronic device, and storage medium for recognizing facial expression images, relating to the field of image recognition technology. It includes: determining the original vector of the expression image based on its pixel values; performing principal component analysis on the original vector to obtain the target vector of the expression image; inputting the target vector into a feature extraction network to obtain feature information of the expression image; encoding the feature information using pulse frequency based on a Poisson distribution to obtain a pulse sequence of the expression image; and inputting the pulse sequence into an expression recognition model to obtain the recognition result of the expression image. The expression recognition model is a multilayer spiking neural network, where the spiking neuron model is an improvement based on the leaky integral discharge neuron (LIF) model. This application's technical solution offers better interpretability and flexibility, significantly reducing network energy consumption while improving computational power, making it suitable for more intelligent small robots such as customer service robots.

The invention relates to the technical field of data processing, and discloses an occupational quality and tendency evaluation system and method based on artificial intelligence, and the system comprises a data collection module which judges whether to adjust a data collection device or not based on a signal-to-noise ratio, and determines the state data of a person to be evaluated; the evaluation analysis module performs classification processing on energy distribution in the state data based on a clustering algorithm, determines an expression state of the to-be-evaluated person based on facial features in the state data, and determines an emotion type of the to-be-evaluated person based on a convolutional neural network, voice fluctuation in the state data and the expression state; the evaluation first processing module determines whether the to-be-evaluated person meets occupational requirements or not based on the brain wave-response reaction curve; and the second evaluation processing module determines the occupational evaluation grade of the person to be evaluated based on the slope of the brain wave-response reaction curve and the emotion type. According to the invention, the comprehensiveness and credibility of the evaluation result are ensured.

This invention proposes an anesthesia recovery prediction system and method based on facial features. The anesthesia recovery prediction system includes a processing device, a 3D camera, a visible light camera, and a millimeter-wave radar module. The processing device is communicatively connected to the 3D camera, the visible light camera, and the millimeter-wave radar module. Sliding tracks are set on both sides of the upper half of the hospital bed. Two 3D cameras are mounted on the sliding tracks on the left and right sides of the bed via first movable seats. The millimeter-wave radar module and the visible light camera are mounted on the sliding track on one side of the bed via second movable seats. The first and second movable seats are sequentially set on the sliding tracks. In this invention, the recovery status is analyzed layer by layer based on changes in the eyes, facial expressions, and mouth features, which reduces the workload of medical staff while meeting monitoring requirements.

The invention relates to the technical field of network live broadcast, in particular to an agricultural product personalized recommendation digital human live broadcast interaction system based on a large model, and the system comprises a configuration module which is used for configuring the image and voice of a digital human anchor and the initial behavior logic related to agricultural product recommendation according to a live broadcast marketing target; the interaction acquisition module is used for acquiring multi-modal user interaction content of a live broadcast room in real time, the multi-modal analysis module is used for analyzing the multi-modal user interaction content through a large model, and the personalized content generation module is used for generating personalized recommended voice texts and the like according to comprehensive interaction instructions. And the synchronous stream pushing module is used for carrying out time sequence alignment and synchronous rendering on the generated broadcast voice, facial expression and limb action sequence, and pushing a final audio and video stream to a live broadcast platform. According to the system, through deep customization and system integration in the field of agricultural products, the professionality and credibility of interaction are improved, the user experience is remarkably improved, and the transaction conversion rate is increased.

This invention belongs to the field of teaching evaluation technology, specifically a data-driven intelligent classroom evaluation and feedback device for teaching behavior. It includes a control device, a mobile blackboard structure, a wearable structure, a monitoring device, and a recording device. The control device comprises a controller body integrating a classroom data acquisition module, a classroom scene recognition module, a student attention acquisition module, a student interaction quality acquisition module, and a classroom content analysis report generation module. The monitoring device connects to the mobile blackboard structure, and the recording device is worn on the teacher's chest via the wearable structure. This allows the monitoring device to capture real-time images of the entire classroom environment from the blackboard, showing student performance and interaction. The recording device worn on the teacher's chest collects detailed data such as students' body movements and facial expressions at close range. This multi-dimensional data is summarized and analyzed by the control device to understand the strengths and weaknesses of the teacher's teaching.

The invention discloses a digital human generation method and device, a program product and electronic equipment, and relates to the field of artificial intelligence or other related fields, and the generation method comprises the steps: receiving a reference image and a target video, splitting the target video into a driving video and an audio, carrying out the feature extraction of the reference image and the driving video, and carrying out the feature extraction of the audio; obtaining an original feature vector of the reference image and a driving feature vector of the driving video, carrying out feature extraction on the audio to obtain an audio feature vector of the audio, determining a feature change vector by adopting a preset action driving network based on the original feature vector and the driving feature vector, and fusing the audio feature vector and the driving feature vector to obtain an audio feature vector of the audio. And inputting the original feature vector, the feature change vector and the fused feature vector into a preset image decoder to obtain a target digital human. According to the invention, the technical problem that the matching degree of the generated digital human in terms of expressions, actions and audios is poor in the prior art is solved.

SYSTEM AND METHOD FOR EVALUATING DYNAMIC WRINKLES ON KERATINIC MATERIAL FROM A USER TO BE TESTED. This disclosure relates to a system and method for evaluating dynamic wrinkles on the keratinic material of a user to be tested. The system comprises: a positioning device configured to position the user; an image acquisition device comprising a first computer circuit configured to obtain images related to the user's facial expressions from simultaneous directions; and a computer device configured to be coupled to the image acquisition device. The computer device includes a memory and a processor, in which the memory stores said images and the processor includes a second computer circuit configured to process said images in order to evaluate dynamic wrinkles.The disclosure can provide an all-in-one solution for simultaneous image capture, repositioning, and storage; it can perform a promising dynamic wrinkle assessment and obtain a clinical classification of dynamic wrinkles. Figure for the abstract: Figure 1.

This invention provides a smart glasses for myopia prevention and control in teenagers, relating to the field of smart glasses technology. It includes a nose bridge and a frame. The nose bridge houses a main control module, and an auxiliary camera module is mounted on the top of the nose bridge to capture the wearer's facial expressions. Ambient light sensors are mounted on the top of the front sides of both sides of the frame, and temple mounting heads are mounted on both sides of the frame. Ultraviolet sensors are mounted on the outer sides of the temple mounting heads. Composite lenses are snapped into the inside of the frame. A main control system is embedded within the main control module. This invention utilizes the auxiliary camera module, ambient light sensor, ultraviolet sensor, biosensor, and non-contact eye posture sensor to comprehensively collect multi-dimensional data such as facial expressions, ambient light, ultraviolet intensity, physiological indicators, and eye posture, providing rich and accurate information for subsequent analysis and decision-making.

This invention provides a method, apparatus, and electronic device for analyzing the emotion of an object. The method includes: extracting static facial features and dynamic features from multimedia data associated with the target object; the dynamic features include one or more of facial expression change features, vocal features, and linguistic content features; inputting the static facial features and dynamic features into a pre-trained object emotion analysis model; and performing feature fusion processing on the static facial features and dynamic features through the object emotion analysis model to output the emotion analysis result. This method performs feature fusion processing on static facial features and dynamic features. Since dynamic features also contain feature information representing emotion, combining static facial features with dynamic features for emotion analysis can, to some extent, reduce the influence of interfering features in static facial features on the emotion analysis result, strengthen the role of feature information representing emotion, and thus improve the accuracy of the emotion analysis result.