360 results about "Facial expression" patented technology

The invention discloses a scene interactive AI rehabilitation evaluation training and health monitoring system, and relates to the technical field of rehabilitation medical treatment and artificial intelligence, a semantic perception module is used for collecting and recognizing voice input, facial expressions, action tracks and eye movement paths of a user in a training process, and extracting context parameters; the knowledge-driven training generation module is used for calling a rehabilitation knowledge graph constructed by a graph neural network based on context parameters and individual training history, and generating a multi-path training scheme; training a feedback regulation engine, collecting posture offset, physiological stress and emotion feedback, and dynamically adjusting task difficulty, rhythm and prompt mode based on a dual-channel reinforcement learning model; the prediction module fuses training and monitoring data, and predicts a network identification function degradation risk through degradation driving; the cloud edge fusion platform is used for realizing task quick response and graph strategy iterative updating; according to the invention, the individuation, self-adaption and intelligent prediction capabilities of rehabilitation training are improved, and the rehabilitation effect and the system practicability are obviously optimized.

The embodiment of the invention provides a vehicle-mounted emotion interaction method and device based on multi-dimensional recognition, and the method and device achieve the precise judgment of the emotion of a driver through innovatively constructing an emotion fusion recognition model and integrating the facial expression, voice emotion, driving behavior and physiological state features. And designing a scene-based self-adaptive interaction strategy, and establishing an interaction triggering threshold value for intelligent matching in combination with external environment data and a danger level. An interaction effect evaluation mechanism is introduced, an interaction strategy model is continuously optimized through an online learning module, and dynamic adjustment of personalized interaction content is achieved. According to the method, the defects of the traditional technology in the aspects of emotion recognition, interaction strategies, effect evaluation and the like are effectively overcome, and the intelligent level and the user experience of vehicle-mounted emotion interaction are remarkably improved.

The embodiment of the invention provides a digital human interaction system and method based on multi-modal emotion recognition, and belongs to the technical field of digital human interaction. The system comprises a multi-modal sensing module used for collecting multi-modal data and preprocessing the multi-modal data to generate a standardized data stream; the cross-modal fusion and emotion recognition module is used for carrying out interactive modeling on the multi-modal features and outputting a current emotion label and emotion intensity; the reaction planning module is used for generating a composite reaction strategy; and the digital human rendering module is used for mapping the composite reaction strategy into control signals corresponding to the voice, the facial expression and the action respectively, and driving a digital human to execute corresponding voice output, facial expression change and limb action through the control signals so as to realize interaction. According to the method, multi-modal data are deeply fused through the cross-modal graph neural network and comparative learning, the weight is dynamically adjusted in combination with the modal confidence, and the emotion recognition accuracy and robustness are improved.

The embodiment of the invention provides an elderly emotion recognition method and device based on multi-modal perception, and the method and device achieve the optimization and enhancement of the signal quality through innovatively constructing a multi-modal data preprocessing mechanism and integrating the facial expression, voice and posture features. And designing a personalized feature mapping model based on historical emotion expression data, and establishing an adaptive feature fusion strategy for intelligent matching in combination with a cross-modal attention network. A hierarchical time sequence classification mechanism is introduced, dynamic modeling of the emotional development trend is realized through a long-short term memory network, and accurate prediction of the emotional state is supported. According to the method, the defects of the traditional technology in the aspects of multi-modal processing, personalized modeling, time sequence analysis and the like are effectively overcome, and the accuracy and reliability of sentiment recognition of the old people are remarkably improved.

The invention provides a pain assessment system and method based on multi-modal physiological signals. The system comprises a multi-modal signal acquisition module, a signal preprocessing module, a multi-modal feature extraction module, a deep fusion analysis module, an individualized calibration module and a result output and early warning module. By synchronously collecting and analyzing multi-dimensional data such as facial expressions, sound features, physiological signs and behavior responses and combining deep learning and multi-modal information fusion technologies, objective quantitative evaluation and real-time monitoring of the pain degree are achieved, and accurate decision support is provided for clinical pain management.

The invention relates to a multi-modal virtual human real-time generation method based on an expression control embedding space, and belongs to the field of artificial intelligence. According to the method, an expression control embedding space is constructed and used for fusing voice semantics, a rhythm structure and multi-dimensional emotion information, and continuous and controllable multi-modal driving vectors are generated. The whole system has an end-to-end linkage mechanism from audio input to expression and action output. Semantic features, rhythm structures and emotional states jointly act on generation paths of lip and upper body postures and expression modalities, and all modal features are fused and expressed in a unified control space through a collaborative coding and time sequence alignment mechanism. And finally, a high-consistency and high-fidelity virtual human video is generated in real time through an output scheduling mechanism. The method has remarkable advantages in the aspects of modal fusion consistency, generation expression naturalness and emotion control flexibility, and can be widely applied to key scenes such as virtual human broadcasting, voice interaction agency and meta-universe digital identity construction.

The invention belongs to the technical field of computer vision and image recognition, and particularly relates to a fine-grained image block embedded multi-task emotion recognition method, which comprises the following steps of: constructing a golden snub monkey multi-modal emotion data set for wild primate animals, and covering emotion, individual and gender multi-dimensional labels; the method comprises the following steps: preprocessing an input wild primate image, dividing the input wild primate image into non-overlapping local image blocks with fixed sizes through blocking and feature extraction, and mapping the non-overlapping local image blocks to a high-dimensional feature space through linear projection to form a series of image block embedding vectors; performing local feature modeling on the image block embedded vector based on a fine-grained local scanning module to enhance fine-grained perception of local features such as facial expression and hair texture of the golden snub monkey, and performing global feature modeling on the image block feature vector by a global scanning module to enhance global semantic representation; according to the invention, the performance and generalization ability of multi-task identification of golden snub monkeys are improved.

The invention discloses an intelligent accompanying robot system, and the system comprises the following modules: a multi-mode interaction module which is composed of a voice recognition unit, a voice synthesis unit, an emotion recognition unit, and a visual recognition unit, and is used for collecting the voice, facial expression, motion, and environment information of a user; the localized AI decision module is used for deploying a lightweight DeepSeekR1 large language model based on an ESP32-S3 edge computing chip, performing real-time processing on the multi-modal data and generating a social guidance strategy and an emotion intervention instruction; the data management module comprises a user behavior database and a privacy protection unit and is used for desensitizing the data and realizing sensitive data isolation through local storage; according to the method, the single-person intervention cost is reduced, and the time consumption of manual scene simulation is reduced.

The invention provides an intelligent interaction system and method based on a multi-stage cognitive mode, and the system comprises a multi-modal data collection module which is used for collecting user interaction data through a multi-modal sensor, and the data comprise language input, non-language behaviors, interface operation data, expressions, eye movement tracks and the like; and the cognitive feature analysis module is used for calling a deep learning model to perform feature extraction on the interaction data. According to the method, language, behavior, interaction, physiology and other data are fused through the multi-modal sensor, the cognitive driving vector is generated by using the deep learning model, the real-time cognitive state of the user is effectively captured, then the probability distribution of the cognitive stage is constructed in combination with Bayesian reasoning, the problems that in the prior art, only the cognitive level can be statically judged, and real-time updating is difficult are solved, and the user experience is improved. The accuracy and timeliness of user state perception are remarkably improved, and dynamic accurate recognition and continuous modeling in the cognitive stage are achieved.

The invention relates to an automatic emergency braking threshold value adjusting method and system based on in-cabin multi-mode information, and relates to the technical field of auxiliary driving, the method comprises the steps that visual information and audio information of a driver and at least one passenger in a cabin are obtained, the visual information comprises facial expressions and head postures, and the audio information comprises audio information of the driver and the at least one passenger; the audio information comprises voice features and non-voice sounds; respectively calculating a driver state coefficient and a passenger state coefficient based on the visual information and the audio information; fusing the driver state coefficient and the passenger state coefficient to obtain an in-cabin safety situation coefficient; and according to the in-cabin safety situation coefficient, a triggering threshold value of the automatic emergency braking system is adjusted in a self-adaptive mode. According to the method, multi-modal information such as vision and audio in the cabin is fused, and state analysis of all passengers is combined, so that intelligent self-adaptive adjustment of the AEB braking threshold value can be realized, and the sensing accuracy and decision reliability of an emergency braking system in a complex scene are improved.

The invention provides an image processing method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring image data of a facial expression; extracting a first expression feature and a first identity feature from the image data; performing feature splicing on the first expression feature and the first identity feature to obtain a first joint feature, and extracting joint information between the first expression feature and the first identity feature from the first joint feature; estimating mutual information between the first expression feature and the first identity feature, and adjusting the first expression feature into a second expression feature by taking minimization of the mutual information as a target; predicting a classification result of the facial expression based on the second expression feature and the joint information; therefore, expression and identity collaborative modeling is realized.

The invention relates to the medical field, and discloses a gyromagnetic pulse action path optimization method and system, and the method comprises the steps: S1, carrying out the pulse data collection of a patient through pulse analysis equipment, and generating original pulse data; performing denoising processing on the original pulse condition data to obtain clear pulse condition data; s2, fusing the clear pulse condition data with other diagnosis information of the patient on the basis of a four-diagnosis combined parameter concept to generate comprehensive health assessment data; performing simulated fingering acquisition and analysis according to the comprehensive health assessment data, simulating the influence of different fingering on the pulse condition, and obtaining a simulated fingering analysis result; according to the method, through multi-part synchronous pulse condition collection, intelligent denoising and a deep learning filtering model, the definition and authenticity of pulse condition data are effectively improved, and the influence of environmental noise is reduced; the pulse condition data, the tongue condition, the facial expression, the voiceprint feature, metabonomics detection and other information are subjected to cross-modal fusion, and comprehensive assessment of the health state of the patient is achieved in combination with the traditional Chinese medicine four-diagnosis combined reference concept.

The invention relates to the technical field of video action detection, and provides a macro-micro expression interval positioning method based on meta-learning, which comprises the following steps: performing face alignment and image size unified processing on an input face expression video frame sequence; performing down-sampling on the processed facial expression video frame sequence based on a pre-trained video understanding model, and extracting time sequence features; dividing and sampling meta-learning tasks according to preset attribute dimensions on the basis of the time sequence characteristics, and constructing a time sequence positioning model at the same time; performing meta-learning training on the time sequence positioning model according to the meta-learning task to obtain a meta-basic model with a generalized spatial-temporal feature representation capability; and carrying out adaptive fine tuning on the element basic model, and using the fine-tuned model to position macro expression and micro expression intervals of the target domain facial expression video. The method can be more easily deployed in various real and complex new scenes, so that the progress of the expression analysis related technology from the laboratory to the practical application is accelerated.

The invention discloses a multi-modal sentiment classification method based on multi-view interaction representation, and relates to the technical field of multi-modal information processing, and the method comprises the steps: collecting and preprocessing the voice, facial expression and text data of a user, and generating a multi-modal data packet; based on the multi-modal data packet, performing feature extraction by adopting a hierarchical attention mechanism to obtain an aligned voice expression text feature sequence, and generating a multi-modal feature flow; performing sentiment classification on the sentiment benchmark result by adopting a time sequence gating network to obtain a service state vector and a conflict view vector; according to the multi-view fusion emotion feature sequence, a time sequence gating network and a multi-mode recognition model are adopted for recognition, and a matched emotion classification result and a natural language reply are generated in combination with the service state. According to the method, sentiment classification is carried out on the sentiment benchmark result by adopting the time sequence gating network, the service state vector and the conflict view vector are obtained, and synchronous description of the sentiment state and the context is realized.

The embodiment of the invention relates to the field of multi-modal emotion analysis, and discloses a multi-modal emotion fusion method based on inverse reinforcement learning. Dynamic weighted fusion is carried out through modal features such as voice, facial expressions and texts, an optimal modal fusion strategy is derived from known emotion label data by adopting inverse reinforcement learning, and the contribution weight of each modal in emotion analysis is automatically adjusted, so that efficient, accurate and stable emotion recognition is realized. Through the technology of the invention, the sentiment analysis system can adaptively optimize modal fusion in various sentiment situations, the generalization ability of the sentiment analysis system in different scenes is significantly improved, the dependence on a large amount of annotated data is reduced, and the sentiment recognition precision is improved at the same time. The method has a wide application prospect and can be widely applied to the fields of intelligent customer service, emotion calculation and the like. The method and the device can be at least used for solving the problems of fixed modal weight and low emotion recognition accuracy in the prior art.

The invention discloses an interactive digital human generation method based on picture and audio synthesis, and belongs to the crossing field of artificial intelligence and computer graphics. According to the method, a full-link process of'feature extraction-model construction-emotion driving-real-time interaction-video output 'can be automatically completed only by uploading a character picture and a section of audio by a user: key point and semantic feature extraction is performed on the picture to obtain face / posture information; performing voice recognition, semantic analysis and emotion recognition on the audio to obtain a semantic tag and an emotion parameter; generating a personalized three-dimensional digital human based on the information, and driving the personalized three-dimensional digital human to generate facial expressions and limb actions which are synchronous with emotions; the user intention is analyzed in real time through natural language understanding and computer vision, and multi-modal interaction is achieved. According to the method, digital human creation can be completed without professional modeling and motion capture equipment, the generation cost is reduced, and the method can be widely applied to virtual anchors, online education, intelligent customer service and movie and television entertainment scenes.

The invention discloses a real-time multi-modal man-machine interaction method and system based on a large model and a storage medium. Collected user interaction information is converted into interaction data in a preset format, the interaction data are input into a full-modal large model, and an output pre-generation result is obtained; secondly, semantic anchor points are marked for reply text information, absolute prediction timestamps of all the semantic anchor points are calculated, semantic time sequence windows are divided, and a global time sequence reference skeleton is constructed; combining and packaging the expression degree-of-freedom sequence and the action degree-of-freedom sequence into a unified expression frame; and locking the rigid segments to the corresponding semantic anchor point timestamps according to the anchor point tags, performing nonlinear filling on the elastic segments between the rigid segments, outputting alignment information, and finally analyzing the alignment information into control instructions to respectively drive a loudspeaker, a facial expression component and a limb motor to execute reply operation. The limitation of dependence of a single mode is broken through, the interaction stability in a complex environment is improved, and meanwhile, the real-time performance of interaction response is improved.

The invention discloses a personnel state analysis method and system based on limb movement recognition, relates to the technical field of limb movement recognition, and determines the movement state of a corresponding limb part based on recognition of each movement-expression combination. And the driving response state of the personnel is determined based on the action state of each limb part, the corresponding part priority and the driving state of the truck, so that the accuracy of the driving response state of the personnel is improved. Therefore, the change event of the facial expressions is determined according to the plurality of facial expressions at different time, and the driving fatigue state of the personnel is determined according to the change event of the facial expressions and the lane changing frequency of the truck; according to the method, the voice interaction event of the person in the driving process is collected, the multiple state key contents are determined according to recognition of the voice interaction event, the state analysis system of the person is determined according to the multiple state key contents, the driving response state of the person and the driving fatigue state, and the accuracy of the state analysis system of the person is improved.

The present disclosure belongs to the technical field of image processing. Provided are a video generation method and apparatus, and an electronic device and a computer program product. The video generation method comprises: extracting a facial identity feature of a target virtual human by using a facial recognition model; extracting an audio feature vector and an emotion prompt word from target audio by using an audio classifier; and using the audio feature vector and the emotion prompt word as control features, merging the control features with a diffusion model, using the facial identity feature to control the output of the diffusion model, and using the diffusion model to generate speech video data of the target virtual human, which speech video data matches the target audio. The technical solution of the present disclosure can generate a speech video stream of a virtual human with natural facial expressions.

The invention discloses a bionic robot facial expression synchronization method and device based on visual perception, equipment and a medium, and relates to the technical field of man-machine interaction, and the method comprises the steps: controlling a target bionic robot to execute a target expression control instruction used for achieving a preset expected robot facial expression, and capturing a robot visual facial expression of the robot; based on the robot visual facial expression and a preset expected robot facial expression, carrying out calibration on the target bionic robot so as to construct a hardware compensation parameter vector; executing a facial expression synchronization operation based on the hardware compensation parameter vector and a pre-trained target expression mapping model by asynchronously running a preset sensing thread and a preset driving thread; the preset sensing thread is used for collecting a target face image in real time and outputting a target steering engine control vector through a target expression mapping model; and the preset driving thread is used for outputting a steering engine control instruction at a preset output frequency based on a preset frame compensation algorithm, the hardware compensation parameter vector and the target steering engine control vector.

The invention relates to the technical field of digital human interaction, in particular to a digital human intelligent question and answer interaction method for a multi-modal visual platform, which comprises the following steps of: receiving multi-modal input of a user, performing privacy protection processing, and performing sensitive field identification, desensitization and hierarchical storage; then semantic analysis and interaction intention recognition are conducted on the processed input, a retrieval request is generated, relevant information is searched in a professional knowledge base based on the retrieval request, meanwhile, source identifiers of entries are reserved, a retrieval result and a user intention are input into a question and answer generation model, and an interaction answer is generated; and adding a corresponding source identifier to the answer to realize content traceability, and finally performing multi-modal output, including voice broadcast, expression action and visual display, through a digital human image, so as to realize safe, credible and intuitive interactive experience.

The invention discloses a speech emotion recognition method based on multi-modal multi-view pseudo-tag fusion, and relates to the technical field of emotion recognition, and the method comprises the steps: firstly, constructing a visual-view pseudo-tag generation module, extracting high-quality facial expression features from a video through employing an optimized OfficientNet-V2 network and a RetinaFace face detector, and carrying out the recognition of the high-quality facial expression features; generating a visual pseudo label through time sequence aggregation; secondly, adopting a semi-supervised learning framework FixMatch to generate a reliable voice pseudo tag for the non-tag voice; secondly, fusing emotional knowledge from multiple visual angles and modes of vision and voice based on a pseudo-tag fusion function, and generating a more accurate and robust fused pseudo-tag; and finally, training a voice emotion recognition model based on a Conformer encoder to perform voice emotion recognition by combining the fused pseudo label data with the labeled voice data. According to the method, the problem of error accumulation in traditional semi-supervised learning is effectively solved, cross-modal emotion complementarity is utilized, and the accuracy and generalization ability of speech emotion recognition are remarkably improved.

Provide a system. 【Solution means】 Means for collecting audio data, Means for collecting video data, Means for preprocessing the collected audio data and video data, Means for analyzing audio, facial expression, and gesture data to extract emotions and intentions, Means for integrating the analysis results and evaluating the negotiation situation, Means for generating advice based on the evaluation results, Means for presenting the generated advice to the user, Means for analyzing non-verbal elements in a residents' briefing or town hall meeting and providing immediate feedback, A system including means for displaying the immediate feedback on the display of a smart device.

1. Name of the product in this design: Voice and Expression Interactive Robot. 2. Purpose of this design: This design is used to control the robot's facial expressions via voice commands. 3. The key design feature of this product is its shape. 4. The image or photograph that best illustrates the design's key points: a 3D model.

A digital human generation method based on a video generation large model includes five steps: S1: receiving an original video signal from a video data source, performing time sequence frame decomposition and key action extraction operations on the original video signal, and forming a preprocessed signal containing human posture features and facial expression features; S2: inputting the preprocessed signal into a pre-trained video generation large model, and generating an initial digital human motion trajectory signal and an appearance rendering signal through the space-time attention mechanism of the model; S3: fusing the initial digital human motion trajectory signal and a preset voice driving signal, and generating a lip shape synchronous control signal by using a cross-modal alignment module; S4: generating a high-real-sense digital human video stream signal by using an adaptive lighting rendering engine according to the appearance rendering signal and the lip shape synchronous control signal; and S5: outputting the high-real-sense digital human video stream signal to a display terminal, simultaneously generating a dynamic detail enhancement feedback signal, and iteratively optimizing the parameter weight of the video generation large model. The digital human generation method based on the video generation large model can solve the problems that it is difficult to eliminate the experience dependence of manual adjustment and optimization, and the die casting parameters cannot be accurately self-optimized.

The present invention presents a driver with a route that provides the brain with stimulation for improving cognitive function and promoting brain development.　A route search device 1 comprises: a route search unit 112 that searches for a plurality of routes from the current location of a vehicle 10 to the destination thereof; a driving difficulty calculation unit 113 that calculates a driving difficulty on the basis of the travel environment of a route; an estimation unit 116 that estimates the driver's level of concentration on the basis of the recognition results of a first recognition unit 114, which recognizes the driver's facial expressions, and a second recognition unit 115, which recognizes the driver's biological information; a provisional total driving load value calculation unit 117 that calculates a provisional total driving load value, in a case where the destination is reached, on the basis of the driving difficulty and an arbitrarily determined degree of concentration; an actual driving load value calculation unit 118 that calculates an actual driving load value on the basis of the driver's level of concentration while traveling the route and the driving difficulty; and a control unit 119 that re-searches for a route on the basis of an estimated total driving load value at a destination arrival time calculated on the basis of the actual driving load value, and the estimated total driving load value.

The invention discloses a mechanical driving system and method for facial expressions, and belongs to the technical field of bionic robots. The system comprises a stress response unit, a mechanical chaos engine and an expression execution mechanism. The stress response unit adopts a design structure that an arc-shaped elastic sheet is in bridge connection with an elliptical ring, converts external mechanical force into nonlinear deformation and realizes automatic resetting; the mechanical chaos engine takes mechanical tolerance as an internal random source, realizes multi-physics field coupling and chaos amplification through a three-stage driving structure, and drives an expression execution mechanism composed of a gradient hardness material, an asymmetric groove array and an intercommunication ring. According to the method, by sensing external force input, a chaos engine is excited to generate unpredictable, natural and rich facial random expressions. Controllable expression of emotional tendency is achieved through a pure mechanical structure, and static and dynamic expressions reflecting the emotional change process can be output. The system is simple and compact in structure, the method randomness is reliable, and the method is suitable for the fields of robots, intelligent toys, emotion counseling, man-machine interaction and the like.

The present application relates to the technical field of real-time snapshot of highlight pictures, and particularly relates to a real-time snapshot method of highlight pictures based on artificial intelligence. The method comprises the following steps: collecting physiological data and sports data of athletes; obtaining explosive power score, concentration score and attention score of athletes; setting a threshold range, defining the moment when the explosive power score, concentration score and attention score exceed the threshold range as a highlight moment; and taking a snapshot of the highlight picture. The present application can more accurately evaluate the explosive power performance of athletes by collecting their physiological data and sports data. The concentration score of athletes can be accurately evaluated by collecting their facial expression data in real time. The attention of the audience can be analyzed by pattern recognition algorithm through real-time collection of the sound and body data of the audience, thereby adding a new dimension to the identification of highlight moments. The intelligences camera technologies such as phase detection auto focus and pan-tilt automatic adjustment are integrated to ensure the clarity and continuity of the highlight pictures.

The invention discloses a human-simulated robot facial expression mechanism of a multi-degree-of-freedom micro-driver, which comprises a bottom plate, a supporting disc is fixedly arranged at the top of the bottom plate, a bracket is fixedly arranged on the supporting disc, an adjusting assembly is arranged on the bracket, a supporting frame is fixedly arranged at the top of the bracket, a shell is connected to the supporting frame, and a driving device is arranged on the shell. The shell is covered with an electronic skin layer, a limiting block is fixedly arranged in the electronic skin layer, a limiting cylinder corresponding to the limiting block is fixedly arranged on the shell, the limiting block is arranged in the limiting cylinder in a penetrating mode, a frame is fixedly arranged in the supporting frame, and an air pressure mechanism is arranged in the frame. A controller is fixedly arranged on the bottom plate; according to the humanoid robot facial expression mechanism of the multi-degree-of-freedom micro-driver, the air pump can be controlled to work, auxiliary limiting of the electronic skin layer is achieved by means of the limiting block, and therefore the installation stability of the electronic skin layer is effectively improved.

Evaluating content based on video or audio data acquired during communication. [Solution] The present invention relates to an information processing device 20 for evaluating content displayed on a user's terminal device in online communication, wherein the audio data recorded in the communication includes the user's utterances in response to the explanation of the content, and the video data recorded in the communication includes the video of the user receiving the explanation of the content, and comprises an acquisition unit 23 for acquiring at least one of the audio data or the video data, and an evaluation unit 27 for analyzing at least one of the user's utterances included in the audio data or the user's facial expressions included in the video data acquired by the acquisition unit to determine an evaluation score for the content.