21 results about "Expression - action" patented technology

The invention relates to the field of human-computer interaction, and discloses a human-computer interaction robot facial expression implementation control method, which comprises the following steps: dynamically capturing the face, sound and posture of a user in a human-computer interaction scene, and generating an initial emotion perception sequence; performing hierarchical cross mapping on the initial emotion perception sequence, and performing combinatorial analysis on facial expressions, voices and intonations and body movement features to form an emotion matching vector; based on the emotion matching vector, utilizing a priority regulation model to predict the emotion trend of the user at the next moment, and identifying an expression enhancement point and an emotion attenuation area; mapping the generated facial micro-expression adjusting instruction to a robot facial driving unit, and performing real-time correction and conflict resolution on an expression action sequence; and reversely fusing a user fixation point, facial muscle micro-motion and voice emotion which are acquired in real time into an emotion matching vector, and dynamically updating a perception weight and expression regulation and control parameters. The method has the advantage of improving the emotion matching degree of the robot expressions.

The invention relates to a human body state detection method, system and device based on facial actions and a medium, and the method comprises the steps: obtaining an image sequence of a human face, carrying out the human face detection and mark point positioning of the image sequence, and obtaining the coordinates of a face mark point; performing micro-expression action extraction on the image sequence based on the facial mark point coordinates to obtain facial micro-expression action features; performing time sequence analysis on the facial action features to obtain time sequence facial action features; performing multi-scale illumination adjustment on the time sequence facial action features to obtain illumination facial action features; performing multi-modal analysis on the illumination facial action features through a pre-trained long-short term memory network to obtain comprehensive human body state features; and performing real-time state analysis based on the comprehensive human body state characteristics to obtain a human body state detection result. According to the method, different types of facial action features can be comprehensively processed, and the reliability of a detection result is enhanced.

The invention discloses an intelligent interaction robot control system which comprises a sensing input module, a core processing module, an output execution module and a system management and maintenance module. The perception input module collects multi-modal data through a visual, auditory, environment and touch unit; the core processing module carries out fusion analysis on the data and generates a decision instruction through a multi-modal information fusion unit, a situation cognition and decision unit, an emotion calculation unit and a motion planning unit; the output execution module executes an interaction task through a voice unit, an expression action unit and a motion unit; and the system management and maintenance module provides knowledge base, cloud collaboration and energy monitoring support. Through multi-modal information deep fusion and situational cognition, accurate understanding of user intentions and environments is realized, emotional interaction and personalized service capabilities are provided, the problems that an existing system is single in interaction mode and lacks context understanding are solved, and the intelligent level and user experience of the robot are remarkably improved.

The application discloses a multi-part coordinated voice AI programming expression robot control system and method, the system comprises a master control unit, a voice processing module, an AI acceleration unit, a secure storage area, a debugging and program injection interface, an actuator driving circuit and a multi-stage safety hardware; the system extracts a structured control intention vector through voice recognition, combines an emotional action mapping library and physical constraints, and automatically generates or modifies control code by an AI programming agent; after digital twin simulation verification and automatic debugging optimization, the control code is safely burned into hardware and executed, realizing automatic generation and reliable execution of multi-part coordinated expression actions, significantly reducing programming complexity, and improving interaction naturalness and system safety.

The application discloses a kind of virtual image model construction method and system based on image clone, it is related to computer graphics field, including: from real-time audio and video stream of real person, the multimodal data of fusion vision, voice and action are obtained;Respectively extract facial expression, voice and action emotional data;Adopt dynamic time warping algorithm to calculate the similarity between each modal emotional data, generate the modal correlation mapping matrix of quantitative modal synchronization relationship;Adopt machine learning model to construct emotional label to limb posture expression action mapping model;According to the deviation of expression and action, the feature is adjusted back;Finally, based on the feature after adjustment, mapping matrix and mapping model, generate the virtual image that expression, voice and action are accurately aligned on time axis.The application establishes the quantitative correlation and feedback adjustment mechanism between modal, significantly improves the real sense and coordination of virtual image when cloning real person in multidimensional emotional expression.

The application provides a kind of expression action control method and device for three-dimensional cartoon production, comprising: step S1, obtaining the description vocabulary of the current to-be-produced character of the to-be-produced three-dimensional cartoon;Step S2, obtain the historical wind evaluation related to the description vocabulary consistent with the cartoon character and the facial expression of the cartoon character, filter out the top N cartoon characters of historical wind evaluation as positive reference characters, N is a positive integer;Step S3, obtain the basic face model of the current to-be-produced character, respectively migrate the facial expression action of all positive reference characters to the basic face model of the current to-be-produced character, obtain and control the facial expression action of the current to-be-produced character is generated.The facial expression action of the application is good by migrating the facial expression action of the historical wind evaluation of the cartoon character to the basic face model of the current to-be-produced character, thereby improving the facial expression action of the three-dimensional cartoon production.

The invention relates to the technical field of artificial intelligence, and provides an old-age care robot content output method and device based on mental health safety modeling, and the method comprises the steps: carrying out the mapping of expression motion features, voice features and physiological denoising information, and obtaining a mental state feature vector; obtaining a content candidate set corresponding to the voice information, extracting an output content candidate feature matrix corresponding to the content candidate set according to a natural language processing model and historical preference data, screening output contents in the content candidate set, and obtaining a final psychological security risk matrix and optimal output content feature parameters, and obtaining the target output content and the dynamic adjustment strategy through psychological safety verification for dynamic output. The method can be applied to a content interaction system in the business fields of medical health, old-age care and the like, psychological safety assessment modeling is carried out on the output content, it is guaranteed that content output is matched with the psychological state, and interaction adaptability and content output reliability are improved.

The invention relates to a multi-mode synchronous control method for driving a robot to narrate, which comprises the following steps: receiving to-be-described text data, and standardizing the to-be-described text data to obtain preprocessed text data; performing hierarchical decoding on the preprocessed text data based on a large language model to obtain structured feature annotation data; generating a multi-modal expression sequence according to the structured feature annotation data, and performing timestamp alignment synchronization on the multi-modal expression sequence to obtain a collaborative multi-modal instruction; and carrying out bandwidth allocation and execution scheduling on the collaborative multi-modal instruction by adopting a dynamic priority strategy, and driving robot hardware to execute a corresponding multi-modal expression action. The diversity of robot emotion expression content is improved, and interaction delay is reduced.

The invention discloses a vehicle-mounted digital human voice interaction system and method based on an unreal engine. The system comprises a voice acquisition module which adopts a microphone array and supports noise reduction processing; the natural language processing module is used for integrating the fine-tuned large language model, judging the emotion of the user through the text subjected to voice recognition, and outputting an emotion label and an answer text; the streaming speech synthesis module is used for performing speech synthesis according to the emotion label by adopting a neural network so as to realize low-delay output; the digital human rendering module is constructed based on an unreal engine MetaHuman technology, realizes synchronization of voice data and animation driving through a WebSocket protocol, and supports expression action driving of mouth shape matching and emotion labels; and the vehicle-mounted display module is used for displaying the digital human image in the screen of the vehicle-mounted terminal and synchronously playing the audio. According to the invention, the problem that an existing vehicle-mounted voice assistant lacks naturalness and emotion feedback is solved, and low-delay and high-immersion vehicle-mounted digital human interaction experience is realized.

The application discloses a kind of 2D digital person generation method and device for multi-element driving source, including obtaining text, voice and video various elements driving source and virtual image to be driven, and using text-to-speech, audio feature extraction and video preprocessing algorithm in driving source analysis module to obtain lip shape driving source and action driving source;The posture action and expression action in action driving source are migrated to the virtual image to be driven, to obtain the driving result synchronized with the action of action driving source;And according to the lip shape driving source, the mouth shape in action driving result is replaced with new mouth shape, to obtain the result synchronized with the lip shape driving source;The fusion audio corresponding to text and voice signal is synthesized with the double driving result of action and lip shape, to obtain the 2D digital person generation result of sound picture matching.The application supports the more controllable 2D digital person generation of multi-element driving source.

The invention discloses holographic interaction equipment based on expression driving. The holographic interaction equipment comprises a holographic display module, a face capture module, an expression recognition module, an interaction decision module, a system control module and a feedback module. Through high-precision three-dimensional face capture, dynamic expression recognition and interactive decision-making mechanisms, facial expression changes of a user can be analyzed in real time, natural expression actions are mapped into virtual interface operation instructions, and then a holographic display interface is dynamically controlled. According to the method, the naturalness, fluency and immersion of interaction are improved, the perception effect of a user on operation is greatly improved through a multi-mode feedback mechanism, such as combination of tactile feedback and visual feedback, and the non-contact and humanized interaction requirements under different application scenes are met. The method is suitable for various application fields such as smart home, rehabilitation assistance, virtual display, immersive entertainment and the like, and has good practical value and wide popularization prospect.

The application relates to a data allocation permission adjustment method and device based on video expression actions, an electronic device and a computer readable medium. The method comprises the following steps: obtaining initial data allocation permission and access information of a target; determining video text content through the access information; establishing a real-time video connection with the target, and displaying the video text content according to the video connection to generate video data; identifying the expression actions of a user in the video data to determine a corresponding permission adjustment coefficient; and adjusting the data allocation permission of the user according to the initial permission and the permission adjustment coefficient. The data allocation permission adjustment method and device based on video expression actions, the electronic device and the computer readable medium can identify micro expressions and micro actions of a user, check the security level of the user from multiple angles, quickly and accurately adjust the access permission of the user, and ensure system safety, data safety and transaction safety.

The application relates to the technical field of artificial intelligence, and relates to a child social accompanying robot interaction system based on emotion recognition, which comprises a multi-source perception and feature extraction module, a cross-modal correlation analysis module, a multi-dimensional emotion fusion module, an emotion strategy mapping and generation module, a double-channel instruction compiling module and a collaborative execution and interaction response module. Multi-scale feature extraction is performed on a face image and a voice signal to obtain a face feature vector and a voice emotion vector. A correlation weight matrix of a target child is constructed. The face feature vector and the voice feature vector are coupled in multiple dimensions to obtain a composite emotion feature. The emotion state is mapped in a strategy mode to obtain an emotion interaction strategy. The composite emotion feature is compiled in an instruction mode to obtain a voice content instruction and an expression action instruction. The voice content instruction and the expression action instruction are input into an execution terminal to obtain an emotion accompanying interaction response of the robot. The application can improve the efficiency of child emotion interaction response.

The application provides a face camouflage method and system based on expression action migration. The method comprises the following steps: extracting a driving video made by a third person according to a random instruction from a face recognition system, taking the difference between the key points of each frame image in the driving video and the key points of a target face, generating an action video of the target face, migrating the action of the third person in the driving video to the target face, retaining the appearance of the target face, adding the motion characteristics of the face of the third person in the driving video, obtaining an action video of the target face, and completing face recognition.

The invention provides a virtual image emotion optimization method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the processing based on a to-be-output text, and determining a main emotion tag, emotion intensity and duration corresponding to the to-be-output text; determining a target action from a pre-constructed emotional expression action database by utilizing the emotional intensity and the main emotional label; determining a target action amplitude of the limb under the target action, and an action frequency and a facial expression parameter corresponding to the target action; adjusting an initial mouth shape action corresponding to the to-be-output text based on the emotion intensity and the main emotion label to obtain a target mouth shape action; optimizing the target action, the target action amplitude and the action frequency to obtain adaptive parameters; performing dynamic rendering on the virtual image based on the adaptation parameters, the duration time and the target mouth shape action so as to facilitate execution of the virtual image; therefore, the emotion expression accuracy of the virtual image is improved.

The application discloses a micro-expression feature extraction method based on a motion unit prototype template, and comprises the following steps: in the preprocessing, in view of the face difference problems among different subjects and the head translation problems of the same subject in the video, an organ-based face position correction and cutting method is used to ensure the accuracy of the optical flow feature extraction; in order to accurately analyze the motion unit in the micro-expression, a motion unit-oriented prototype template is proposed. The representative face motion unit dynamic information is recorded in each prototype template, and the AU recognition of the face action with certain difference has strong robustness; in order to accurately capture the micro-expression action, the optical flow map sequence of the micro-expression video is matched with the AU motion template, the complex micro-expression action in the video is deeply analyzed, and the accuracy and the explainability of the micro-expression feature are improved. The application has the advantages that the subtle face action when the micro-expression occurs can be effectively captured, and the application can be used for micro-expression detection, recognition and generation.

1. The name of the design product: processing expression and action graphical user interface for electronic device. 2. The use of the design product: an electronic device. 3. The design points of the design product: the content displayed on the user interface. 4. The picture or photo that best indicates the design points: design 1 interface change state figure 1. 5. The electronic device is a common design, and the rear view, top view, bottom view, left view, and right view of each design are omitted. 6. Design 1 is designated as the basic design. 7. The use of the graphical user interface: the graphical user interface is used to send expressions or actions. In the front view of design 1, click the human-shaped button at the bottom left to enter design 1 interface change state figure 1. In the expression action bar that pops up below design 1 interface change state figure 1, click the expression or action in the rectangular box to enter design 1 interface change state figure 2 and display the corresponding content. In the front view of design 2, click the human-shaped button at the bottom left to enter design 2 interface change state figure 1. In the expression action bar that pops up below design 2 interface change state figure 1, click the edit button in the upper left corner of the feather shape to enter design 2 interface change state figure 2 and edit the expression or action at the top. 8. Other circumstances that need to be explained: X in each view represents text or characters, and the gray block coating area represents a variable content screen.

The invention discloses a 24-point game innovation ability intelligent evaluation method and device based on multi-modal data, and the method comprises the steps: 1, recording four segments of short videos for explaining 24-point intelligence development rapid calculation rules in advance; the system records the total learning time of the tested micro-course in real time; step 2, constructing a 24-point intelligence development rapid calculation test question bank containing five difficulty levels; step 3, recording the operation process of the subject by adopting a screen recording technology, synchronously acquiring expression action videos of the subject by adopting a video recording technology, and recording audio speech data of the subject by adopting an audio acquisition technology; step 4, processing the expression action video acquired in the step 3 by adopting a pre-trained visual network MANET, and extracting facial expression features; and step 5, constructing a time sequence fusion architecture based on a bidirectional long and short time memory network, and performing dynamic modeling on the features of the modes respectively. According to the invention, online micro-class learning is realized by using online short videos, learning agility is evaluated, and problem solving capability and anti-contusion capability are evaluated by using customs clearance tests.

The application provides a multi-modal data emotion recognition deep learning network based on an rPPG principle. Through the introduction of multi-modal data and auxiliary tasks, end-to-end emotion recognition is realized from a physiological meaning. The network attempts to obtain emotion and physiological labels from a physiological point of view based on video and heart rate signals by observing local blood flow actions of a face instead of macro muscle expression actions such as winking and so on. Blood flow action features containing physiological information in the video are captured through a deep learning network, time attention is provided based on the heart rate signals, and the network is guided to learn key feature representations containing physiological meanings in the face video. Meanwhile, heart rate self-similarity feature extraction is introduced as an auxiliary task to improve the network performance.

The invention discloses a video editing processing method based on artificial intelligence, and relates to the technical field of video processing, and the method comprises the following steps: building a light flash feature description rule for an illumination rapid jump scene, converting each brightness sudden change in a video frame into a light flash event, forming the continuous flash events into a flash event sequence to construct a flash track; and screening a time slice with the maximum brightness change amplitude on the light flash track, extracting a slice with the character expression action kept stable from the time slices, and taking the screened slice as an emotion reference slice to form an emotion baseline. According to the method, through compression of a light flash track, an emotion baseline and an emotion curve, emotion change expression is closer to a real state of a person, and emotion interference caused by illumination jump is weakened; and meanwhile, self-adjusting editing rhythm control is added into the smooth emotion band, so that the rhythm can be adjusted along with the emotion change and the illumination disturbance is avoided, and the rhythm continuity and the emotion expression accuracy of a piece are improved.

The invention relates to the technical field of language processing, in particular to a multi-modal sentiment analysis method for natural language processing, which comprises the following steps: acquiring multi-modal data input by a user, including a text sequence, a voice waveform and a facial expression image; performing context-aware word segmentation processing on the text sequence to generate structured text features including part-of-speech tags and syntactic dependency relationships; inputting the voice waveform and the facial expression image into an asymmetric noise suppression module, and respectively outputting a pure voice spectrum without environmental noise and a standardized expression action sequence without illumination interference; and inputting the structured text features, the pure speech spectrum and the standardized expression action sequence into a dynamic association fusion module, generating a time-aligned joint emotion representation vector through a cross-modal attention mechanism, and outputting an emotion classification result by adopting a hierarchical decision network. According to the method, multi-level emotion understanding and classification are realized, and the recognition capability of an emotion analysis system on complex and fine emotion states is improved.