627 results about "Facial expression" patented technology

The invention discloses a scene interactive AI rehabilitation evaluation training and health monitoring system, and relates to the technical field of rehabilitation medical treatment and artificial intelligence, a semantic perception module is used for collecting and recognizing voice input, facial expressions, action tracks and eye movement paths of a user in a training process, and extracting context parameters; the knowledge-driven training generation module is used for calling a rehabilitation knowledge graph constructed by a graph neural network based on context parameters and individual training history, and generating a multi-path training scheme; training a feedback regulation engine, collecting posture offset, physiological stress and emotion feedback, and dynamically adjusting task difficulty, rhythm and prompt mode based on a dual-channel reinforcement learning model; the prediction module fuses training and monitoring data, and predicts a network identification function degradation risk through degradation driving; the cloud edge fusion platform is used for realizing task quick response and graph strategy iterative updating; according to the invention, the individuation, self-adaption and intelligent prediction capabilities of rehabilitation training are improved, and the rehabilitation effect and the system practicability are obviously optimized.

The invention discloses a multi-modal cognitive impairment evaluation system based on a multi-dimensional cognitive function hypergraph. The multi-modal cognitive impairment evaluation system comprises a multi-modal signal acquisition system, a cognitive function evaluation normal form design, a universal feature extraction framework of each modal and online hypergraph learning based on a multi-dimensional cognitive function. The system constructs a dimensionality evaluation normal form containing the cognitive function based on multi-modal physiological and behavior signals, so that comprehensive evaluation of the cognitive function of the elderly is realized. Firstly, physiological and behavior responses of a user under different tasks and situations are recorded and analyzed in real time by fusing multi-mode signals such as electroencephalogram, electrocardio, voice and facial expressions. And then, mining fusion representation among the multi-modal signals, generating unified feature representation, introducing an online hypergraph of clinical knowledge, capturing a complex multi-modal data relationship, and providing an analysis result containing cognitive function evaluation. Finally, in combination with the technical means, the system can realize early detection of cognitive impairment.

The invention provides an intelligent learning behavior monitoring and abnormity early warning method and device, and relates to the technical field of education analysis, and the device architecture comprises a multi-modal data collection module, a behavior analysis module, a dynamic abnormity grading early warning module and a processing measure module. The multi-modal data acquisition unit integrates learning platform operation records, facial expression data and physiological data as well as environment and situation information collected by the situation awareness data acquisition unit, so that omnibearing and multi-dimensional data acquisition is realized; the long-time monitoring and personalized model construction unit uses a plurality of data mining algorithms and LSTM in deep learning to combine with an attention mechanism, accurately mines student behavior rules according to long-term data and constructs a personalized model, and can flexibly and accurately judge abnormal behaviors according to student individual conditions and learning situations. Graded early warning is provided for related personnel in time, deterioration of learning problems is effectively prevented, and timeliness and effectiveness of education intervention are improved.

The invention relates to the technical field of AI interaction, and discloses an AI multi-modal emotion interaction memory terminal, which realizes microsecond-level synchronization of voice, facial expression and text data through a multi-thread acquisition engine, dynamically allocates each modal weight by adopting a multi-head cross attention mechanism, and adaptively adjusts modal importance based on a conversation context hidden state; when the cross-modal confidence difference exceeds a threshold value, a gating LSTM conflict resolution module is activated, and the multi-source data collaboration problem is solved; the emotional memory modeling constructs an emotional state transition topology based on a graph convolutional network, protects user privacy in combination with a differential privacy mechanism, and realizes associated event storage of millisecond backtracking of short-term memory and long-term memory. The technology integrates multi-modal dynamic perception, privacy security calculation and adaptive learning ability, significantly improves the real-time performance and personification degree of emotion interaction, and can be applied to the fields of intelligent customer service, emotion accompanying, health monitoring and the like.

An AI-driven system for personalized advertising in real time, where: ◯ an analytics unit to monitor and analyze user behavior in real time across multiple digital platforms such as websites, mobile applications, social media, and smart devices; the unit collects data on user engagement, browsing patterns, time spent, and content preferences to enable targeted, personalized advertising; ◯ a prediction module that predicts preferences and interests of a user, operatively connected to the user interaction data collection unit, wherein the prediction module uses machine learning models such as deep learning, recurrent neural networks (RNNs) and transformer-based architectures to predict interests of users based on historical interactions and inferred preferences; ◯ an emotion and sentiment analysis unit that assesses the user's mood in real time through computer vision, natural language processing (NLP) and voice analysis, whereby the analysis of facial expressions, voice pitch and linguistic mood is used to determine emotional states and receptivity to advertising content; ◯ an embodiment of a context awareness component in operational communication with the emotion analysis and mood unit, in some cases further augmented by various environmental and situational data such as the device type and its physical location, date and time, and the content processed in the device, processing methods, etc., in establishing adaptability and automatic ad placement to be as relevant as possible to the user and their status as prescribed; ◯ Use reinforcement learning algorithms and generative AI models to drive advertising with personalization engines. Creative elements, messaging, and presentations are dynamically adjusted in real time based on user responses to ensure advertising is personalized and always optimized for best performance; o a privacy-focused AI system with federated learning, differential privacy methods, and on-device AI processing to reduce targeted advertising while complying with international data protection laws such as the General Data Protection Regulation (GDPR) and California consumer privacy laws; o an operationally adapted ad delivery mechanism to engage with real-time bidding (RTB) networks, programmatic advertising exchanges and demand-side platforms (DSPs) and place advertisements through digital advertising networks, which guarantees the delivery of tailored advertising to the most relevant audience in real time; and o a contextual feedback loop in which the machine learning models used in the preference and interest prediction module and in advertising personalization The engine is continuously updated to reflect the latest user engagement data, improving personalization over time and optimizing advertising performance.

The embodiment of the invention provides a vehicle-mounted emotion interaction method and device based on multi-dimensional recognition, and the method and device achieve the precise judgment of the emotion of a driver through innovatively constructing an emotion fusion recognition model and integrating the facial expression, voice emotion, driving behavior and physiological state features. And designing a scene-based self-adaptive interaction strategy, and establishing an interaction triggering threshold value for intelligent matching in combination with external environment data and a danger level. An interaction effect evaluation mechanism is introduced, an interaction strategy model is continuously optimized through an online learning module, and dynamic adjustment of personalized interaction content is achieved. According to the method, the defects of the traditional technology in the aspects of emotion recognition, interaction strategies, effect evaluation and the like are effectively overcome, and the intelligent level and the user experience of vehicle-mounted emotion interaction are remarkably improved.

The invention discloses a personalized digital human generation method based on a single video, and relates to the field of virtual digital human modeling and driving, and the method aims at the video and voice data of a target person, through introducing a multi-modal alignment constrained voice driving synchronization mechanism and combining semantic understanding and an emotion label expression generation model, a personalized digital human model is generated. High synchronization, nature and vividness of digital human facial expressions and voice contents are realized. According to the method, a self-supervised style consistency constraint is added in model training to ensure that the style of a generated character image is stable, and a 3D semantic mask is added in an image fusion stage to improve the fusion precision and realistic effect of a synthetic facial expression and a reference face. According to the method, the digital human can be rapidly cloned and driven to synthesize the expression through the voice only through a single video sample, the generation process is efficient, and the obtained digital human video has excellent sense of reality and interactivity.

The invention discloses an aircraft cabin environment personalized adjustment method based on sentiment analysis, and the method comprises the steps: collecting the facial expression, voice waveform and environment parameters of a passenger in real time through a cabin multi-source sensor, and generating a standardized physiological signal matrix and anonymized voice features through noise reduction and feature extraction; inputting the physiological signal matrix and the voice features into a pre-trained deep learning model, outputting an emotion index and a classification label, and updating model parameters through a federal learning framework; dynamically generating temperature, humidity and oxygen concentration adjusting instructions and dynamic weights by adopting a fuzzy reasoning system in combination with the emotion indexes and passenger preset preferences; environment adjustment is executed through a closed-loop control system, and environment parameter errors are fed back; synchronously updating a fuzzy inference system rule base and deep learning model parameters by utilizing reinforcement learning in combination with environmental parameter errors and emotion index changes, and completing optimization of a closed loop; according to the invention, real-time dynamic regulation and control and continuous adaptation optimization of the personalized cabin environment can be realized.

The invention discloses a multi-mode driven virtual digital human face animation generation method and system, and relates to the field of computer graphics, and the method comprises the steps: obtaining voice input and text input, and extracting voice features and text features; dynamically fusing the two features through an attention fusion model to generate facial expression and head posture control parameters, and dynamically adjusting contribution weights of a voice mode and a text mode to the control parameters by adopting a driving strategy of differentiation of facial upper expression and facial lower expression; and performing local deformation on the virtual digital human face image based on the control parameter to generate an initial animation frame, and performing refining processing on the initial animation by using a generative adversarial network to obtain a refined face animation. By means of the technical scheme, the natural and vivid virtual human face animation matched with the voice content and the text semantics can be generated.

The embodiment of the invention provides a digital human interaction system and method based on multi-modal emotion recognition, and belongs to the technical field of digital human interaction. The system comprises a multi-modal sensing module used for collecting multi-modal data and preprocessing the multi-modal data to generate a standardized data stream; the cross-modal fusion and emotion recognition module is used for carrying out interactive modeling on the multi-modal features and outputting a current emotion label and emotion intensity; the reaction planning module is used for generating a composite reaction strategy; and the digital human rendering module is used for mapping the composite reaction strategy into control signals corresponding to the voice, the facial expression and the action respectively, and driving a digital human to execute corresponding voice output, facial expression change and limb action through the control signals so as to realize interaction. According to the method, multi-modal data are deeply fused through the cross-modal graph neural network and comparative learning, the weight is dynamically adjusted in combination with the modal confidence, and the emotion recognition accuracy and robustness are improved.

The invention discloses a video portrait consistency editing method based on optical flow guidance and text driving, and the method comprises the steps: firstly, employing a face key point extraction method and a depth estimation method to extract a face key point and a depth map of each frame of an input video, and guaranteeing the consistency of the facial expression of an edited portrait and a source image; a diffusion control network is used to edit and generate the face depth map, the key points and the user input text description as conditions, and the consistency of multi-view editing is realized through the potential code alignment of an attention mechanism and the guidance of optical flow change; thirdly, constructing a diffusion model with consistent time by utilizing the inter-frame optical flow change and the depth map change; through the above steps, the problems of multi-view and time consistency in video editing are solved, and high-quality video portrait editing is realized.

The invention discloses a psychological state pre-screening system and method based on multi-modal data. The psychological state pre-screening system comprises a data acquisition module, a data preprocessing module, a psychological state analysis module, a psychological state evaluation and screening module and a feedback and early warning module. The system collects multi-modal data such as facial expressions, voices, texts, physiological signals and the like, and performs psychological state analysis by using a deep learning algorithm. The psychological state score is calculated through a multi-modal data fusion algorithm, and feedback or early warning is provided, so that the user or related mechanisms can perform further intervention. According to the method, the accuracy and the real-time performance of psychological state screening can be improved, and the problem of misjudgment of a single data source is avoided. The system can be integrated to a smart phone, a smart bracelet, a computer and other equipment, realizes low-cost and non-perceptual mental health monitoring, is suitable for personal health management, enterprise employee mental monitoring, psychological counseling auxiliary diagnosis and school psychological screening, and is beneficial to early discovery and intervention of mental health problems.

The embodiment of the invention provides an elderly emotion recognition method and device based on multi-modal perception, and the method and device achieve the optimization and enhancement of the signal quality through innovatively constructing a multi-modal data preprocessing mechanism and integrating the facial expression, voice and posture features. And designing a personalized feature mapping model based on historical emotion expression data, and establishing an adaptive feature fusion strategy for intelligent matching in combination with a cross-modal attention network. A hierarchical time sequence classification mechanism is introduced, dynamic modeling of the emotional development trend is realized through a long-short term memory network, and accurate prediction of the emotional state is supported. According to the method, the defects of the traditional technology in the aspects of multi-modal processing, personalized modeling, time sequence analysis and the like are effectively overcome, and the accuracy and reliability of sentiment recognition of the old people are remarkably improved.

The invention provides a multi-modal emotion recognition software development method for voice interaction of an intelligent robot, and the method achieves the synchronous collection and fusion of multi-modal data through the weighted fusion preprocessing of voice signals, facial expression video streams and text contents. A DCNN and LSTM combined model is adopted to deeply extract emotional features, and an attention mechanism is introduced to strengthen key information capture. And a comprehensive emotion representation vector is calculated by using a cross-modal emotion association network CMAN, so that the emotion distinguishing capability is improved. An emotion classifier combining an SVM and a neural network is constructed, and refined output of emotion categories and intensity is achieved. Through real-time feedback and adaptive optimization steps, the model can continuously learn and adapt to user emotion expression habits, and accuracy and user satisfaction are improved. According to the invention, more humanized and accurate multi-modal emotion recognition technical support is provided for voice interaction of the intelligent robot.

A user workload state processing method and apparatus based on multimodal data, and a device. The method comprises: acquiring state information of a first user; determining workload information of the first user on the basis of the acquired state information, wherein the state information comprises at least two of the following information: physiological information, eye movement information, electroencephalogram information, brain function imaging information, motion capture information, spatiotemporal acquisition information, behavior acquisition information, and facial expression and state information; and feeding back the workload information to a first management account, and formulating a training scheme matched with the workload information. According to the embodiments of the present application, workload of users can be identified on the basis of multimodal data of users.

The invention relates to an intelligent video editing method fusing human face features and human voice features, which comprises the steps of input preprocessing, multi-modal analysis, fusion scoring and automatic editing, and adopts a multi-modal mode for analysis, so that the recall rate of key segments is improved, and the efficiency of editing is improved. A personalized editing strategy is supported, facial expression changes and voice emotion peak values are aligned through dynamic time warping, a CLIP-like structure is used for training a bimodal encoder, the human face and the voice are mapped to a unified vector space, the association weight of the human face and the voice is automatically learned, and the intelligent degree of editing is improved.

The invention provides a pain assessment system and method based on multi-modal physiological signals. The system comprises a multi-modal signal acquisition module, a signal preprocessing module, a multi-modal feature extraction module, a deep fusion analysis module, an individualized calibration module and a result output and early warning module. By synchronously collecting and analyzing multi-dimensional data such as facial expressions, sound features, physiological signs and behavior responses and combining deep learning and multi-modal information fusion technologies, objective quantitative evaluation and real-time monitoring of the pain degree are achieved, and accurate decision support is provided for clinical pain management.

The invention discloses a cross-domain expression motion unit detection method based on text bridging, and the method comprises the steps: extracting AU visual features of different scales through employing a visual encoder, introducing a group of learnable prompt vectors, and combining with AU text description to obtain AU text features; the source domain features and the text features are aligned through comparative learning, and the discrimination of the source domain AU visual features is improved; a cross-modal attention mechanism is adopted to realize preliminary interaction of vision and texts, and AU interaction feature representation is obtained; further promoting deep interaction and fusion of the text and the visual information through a graph neural network to obtain a fused AU feature; and on the basis of a negative sample AU description-based comparative learning method, target domain features are aligned with text features, and feature distribution differences between a source domain and a target domain are reduced. According to the method, the field invariance of the text is utilized, the visual features of the source domain and the target domain are unified, the text features are aligned, and the generalization performance of an AU detection model and the accuracy of the AU detection model on the target domain are remarkably improved.

The invention relates to a multi-modal virtual human real-time generation method based on an expression control embedding space, and belongs to the field of artificial intelligence. According to the method, an expression control embedding space is constructed and used for fusing voice semantics, a rhythm structure and multi-dimensional emotion information, and continuous and controllable multi-modal driving vectors are generated. The whole system has an end-to-end linkage mechanism from audio input to expression and action output. Semantic features, rhythm structures and emotional states jointly act on generation paths of lip and upper body postures and expression modalities, and all modal features are fused and expressed in a unified control space through a collaborative coding and time sequence alignment mechanism. And finally, a high-consistency and high-fidelity virtual human video is generated in real time through an output scheduling mechanism. The method has remarkable advantages in the aspects of modal fusion consistency, generation expression naturalness and emotion control flexibility, and can be widely applied to key scenes such as virtual human broadcasting, voice interaction agency and meta-universe digital identity construction.

The invention belongs to the technical field of computer vision and image recognition, and particularly relates to a fine-grained image block embedded multi-task emotion recognition method, which comprises the following steps of: constructing a golden snub monkey multi-modal emotion data set for wild primate animals, and covering emotion, individual and gender multi-dimensional labels; the method comprises the following steps: preprocessing an input wild primate image, dividing the input wild primate image into non-overlapping local image blocks with fixed sizes through blocking and feature extraction, and mapping the non-overlapping local image blocks to a high-dimensional feature space through linear projection to form a series of image block embedding vectors; performing local feature modeling on the image block embedded vector based on a fine-grained local scanning module to enhance fine-grained perception of local features such as facial expression and hair texture of the golden snub monkey, and performing global feature modeling on the image block feature vector by a global scanning module to enhance global semantic representation; according to the invention, the performance and generalization ability of multi-task identification of golden snub monkeys are improved.

The invention discloses an intelligent accompanying robot system, and the system comprises the following modules: a multi-mode interaction module which is composed of a voice recognition unit, a voice synthesis unit, an emotion recognition unit, and a visual recognition unit, and is used for collecting the voice, facial expression, motion, and environment information of a user; the localized AI decision module is used for deploying a lightweight DeepSeekR1 large language model based on an ESP32-S3 edge computing chip, performing real-time processing on the multi-modal data and generating a social guidance strategy and an emotion intervention instruction; the data management module comprises a user behavior database and a privacy protection unit and is used for desensitizing the data and realizing sensitive data isolation through local storage; according to the method, the single-person intervention cost is reduced, and the time consumption of manual scene simulation is reduced.

The invention provides a campus abnormal behavior detection method and device and electronic equipment, and belongs to the technical field of artificial intelligence and biological signal processing, and the method comprises the steps: obtaining the facial expression features, body motion features, voice emotion features and physiological signal data of a target person in a monitoring region in a campus scene; fusing the facial expression features, the voice emotion features, the body action features and the physiological signal data to obtain fused data; performing emotion recognition and behavior prediction on the fused data based on a preset time sequence model to obtain an analysis result; and predicting whether an abnormal behavior occurs based on the analysis result, and performing early warning when the prediction result is that the abnormal behavior occurs. The technical problem that potential safety hazards still exist in actual detection results due to the fact that application of emotion and physiological signals in abnormal behavior detection is not fully considered in the prior art is effectively solved.

The invention discloses a digital human personalized shaping method and device based on user psychology and a medium. The method comprises the following steps: acquiring multi-modal data when a user interacts with a digital human; based on the multi-modal data, according to the style preference of the user, establishing a psychological feature model through an adaptive learning mode; dialogue voice of the digital person is generated according to the psychological feature model, and matched facial expressions and limb actions of the digital person are generated according to the dialogue voice and the psychological features of the user. According to the digital human personalized shaping method and device based on the psychology of the user and the medium, the psychological feature model is established based on the style preference of the user, and the interaction content matched with the conversation voice, the facial expression and the body movement is generated, so that the digital human can show more real and natural personalized behaviors, and the user experience is improved. The problem that it is difficult to carry out personalized behavior shaping on the digital human according to psychological characteristics and interaction preferences of the user can be solved.

The invention discloses an elderly emotion evaluation system based on multi-modal physiological and behavior signals. The elderly emotion evaluation system is composed of a multi-modal signal acquisition module, an emotion induction task design module and a multi-modal feature processing module. By collecting multi-mode signals such as electroencephalogram, electrocardio, voice and facial expressions, the system can synchronously record physiological and behavior changes of a user in a specific situation. The emotion induction task adopts scientifically designed audio and video contents, and six typical emotions such as happiness, tension, boring, anger, calm and sadness are guided. Then, the system extracts emotion related features from the multi-modal signals by using a deep learning method, models a cooperative relationship among the multi-modal signals by using an attention mechanism fusion technology, and generates unified emotion feature representation; by comprehensively analyzing and identifying the emotional state of the elderly user, the system provides important reference for early monitoring of emotional health of the elderly, psychological state evaluation and personalized intervention, and has social and clinical values.

The present invention discloses a safe driving early warning system for intervention based on driver emotions. The system obtains current driver facial expression dynamics through a driver emotion real-time recognition unit, performs intelligent emotional analysis based on the obtained facial expression dynamics, and controls a vehicle running status based on a real-time emotion recognition result. The system monitors driver emotions in real time during driving, and controls a vehicle driving status and / or issues a safety warning to a driver according to a monitoring result, so as to ensure the traffic safety of various vehicles from the perspective of the driver.

The invention provides an intelligent interaction system and method based on a multi-stage cognitive mode, and the system comprises a multi-modal data collection module which is used for collecting user interaction data through a multi-modal sensor, and the data comprise language input, non-language behaviors, interface operation data, expressions, eye movement tracks and the like; and the cognitive feature analysis module is used for calling a deep learning model to perform feature extraction on the interaction data. According to the method, language, behavior, interaction, physiology and other data are fused through the multi-modal sensor, the cognitive driving vector is generated by using the deep learning model, the real-time cognitive state of the user is effectively captured, then the probability distribution of the cognitive stage is constructed in combination with Bayesian reasoning, the problems that in the prior art, only the cognitive level can be statically judged, and real-time updating is difficult are solved, and the user experience is improved. The accuracy and timeliness of user state perception are remarkably improved, and dynamic accurate recognition and continuous modeling in the cognitive stage are achieved.

The invention discloses an elderly falling early warning and intervention method based on emotion recognition and behavior pattern analysis. The elderly falling early warning and intervention method comprises the following steps: S1, collecting a facial expression data set, a voice signal data set and a behavior data set of an elderly; s2, analyzing the collected facial expression data set by using a convolutional neural network model; s3, analyzing the voice signal data set by using a long short-term memory network model; s4, an ElderSense model is set; and S5, when the analysis result shows that the risk score is higher than a preset first threshold value, sending an early warning signal to the old people through the intelligent equipment, and when the analysis result shows that the falling risk is higher than a preset second threshold value, sending an early warning signal to the old people and caregivers thereof through the intelligent equipment. The falling risk of the old people can be comprehensively evaluated from multiple angles, and the accuracy of falling risk judgment in a complex situation is improved.

Systems and methods are provided for generating adaptive content. The system may implement a machine learning model including training data pre-trained, or trained in real-time based on captured content or prestored content associated with gazes of users, pupil dilations, facial expressions, muscle movements, heart rates, or gaze dwell times of users determined previously or in real-time. The system may determine a gaze(s) of an eye of a user or facial features of a face associated with the user viewing, by a device, items of content in an environment. The system may include determining, based on the gaze(s) or facial features, a state(s) or interest(s) of the user. The system may determine, by implementing the machine learning model and based on the state(s) or interest(s) of the user, content to generate a modification of the items of content or to generate new content items associated with the items of content.

The invention discloses a robot interaction system and method based on machine learning and emotion calculation, and the method comprises the steps: a robot collects the voice, facial expression, body language and physiological signal data of a user through a multi-mode perception device, and carries out the noise elimination, signal enhancement and feature extraction of the data; utilizing a deep neural network and an emotion classification algorithm to identify the emotion state of the user, identifying mixed emotion and performing label classification; a personalized emotion model is established through machine learning in combination with historical emotion data of the user, an emotion change rule is reflected, and long-term learning and adaptation are carried out; based on the sentiment analysis result, a response conforming to the user sentiment state is generated; through user feedback and interaction, the system continuously optimizes emotion understanding and reaction ability and updates emotion archives; emotion computing tasks are shared through edge computing or cloud computing resources, and real-time response to complex emotion computing tasks is ensured.

The invention discloses a depression classification method and system based on multi-modal fusion, and relates to the technical field of multi-modal data processing and intelligent identification. Comprising a data acquisition module used for acquiring an electrocardiosignal, an electroencephalogram signal, a facial expression video and a gastrointestinal environment expiration signal; the pre-processing module is used for performing pre-processing operation on the four modal signals; the data fusion module is used for performing high-order feature extraction and dimensionality reduction on the four modal signals by using different deep learning sub-networks, considering the real-time performance and the mutual relation between different modals, and performing feature fusion on the four modal signals after dimensionality reduction based on an attention soft fusion strategy; and the classification detection module is used for performing classification detection on the comprehensive features by using a deep learning classification model. According to the method, real-time signals of four modes are collected, and efficient and accurate recognition and dynamic monitoring of the depression state are achieved in combination with multi-mode signal preprocessing, multi-domain feature extraction, multi-mode fusion and a deep learning classification model.