229 results about "Expression Feature" patented technology

The invention provides an AI-based marketing video auditing method, and relates to the technical field of AI marketing video auditing, and the method comprises the steps: obtaining a multi-modal data original structure set, and extracting image semantic features, voice expression features, text semantic features and scene label information, and obtaining an image semantic feature set, a visual rhythm feature set, a voice expression feature set, a voice and picture synchronous association vector structure, a text semantic feature set and a subtitle semantic and image main body linkage relation graph. By constructing an image semantic feature set, a voice expression feature set, a text semantic feature set and a visual rhythm feature set and fusing the image semantic feature set, the voice expression feature set, the text semantic feature set and the visual rhythm feature set into a multi-modal content fusion feature tensor, unified modeling of an AI marketing video at visual, auditory and semantic levels can be realized; and subsequent microscopic consistency detection, compliance knowledge graph and emotion semantic conflict identification are effectively performed, so that full-link risk perception and accurate auditing of video contents are realized.

The invention relates to the field of intelligent interaction, and discloses an intelligent hardware dynamic interaction system based on voice semantic fusion and multi-modal perception, which comprises the following steps of: constructing a context model of continuous operation by collecting continuous voice instructions, gesture actions and expression information of a user; semantic analysis and feature fusion are carried out on currently collected voice, gesture and expression features, meanwhile, credibility indexes of all modes are calculated through a weighting or deep learning model, weighting correction is carried out on a fusion result, a real-time feedback algorithm is adopted for weight adjustment for continuous optimization, the next operation intention of a user is predicted through deep learning, and the user experience is improved. And in combination with historical interaction data, online feedback and prediction errors, context management, modal weight and intention prediction strategies are adaptively optimized, and the updated strategies are used for next-round context acquisition and multi-modal fusion. The method has the advantage of improving the recognition accuracy in the continuous interaction scene.

The invention provides an interactive classroom student dynamic portrait generation method, a medium and a system, and belongs to the technical field of student dynamic portrays.The method comprises the steps that a student facial image sequence is collected, facial feature points are extracted to construct an expression feature matrix, and sight line focus data are obtained by combining an eye movement tracking technology to calculate an eye light concentration degree index; and establishing an individual association model, identifying specific expressions and analyzing attention fluctuation characteristics. And fusing multi-dimensional features such as an expression similarity vector and an eye light concentration index into a dynamic attention state score. Learning state evaluation is realized based on a multilayer bidirectional converter network, wherein multi-head attention mechanism parameters are determined by expression change frequency, an eye light concentration threshold and an attention fluctuation rate. And finally, generating a student dynamic portrait feature map, and forming a classroom interaction effect evaluation index through clustering analysis, thereby solving the technical problem that the dynamic change of the student classroom attention state is difficult to accurately evaluate based on a single feature.

The invention relates to an interactive multi-round dialogue digital human modeling method, which comprises the following steps of: extracting double-person multi-modal characteristics in an interactive multi-round dialogue scene, including voice characteristics and expression characteristics of a speaker in a current dialogue round and voice characteristics of a listener in a previous dialogue round in the current dialogue round; performing time sequence alignment and reinforcement on the extracted double-person multi-modal features based on a time dimension to obtain a combined feature sequence; according to the joint feature sequence, generating a voice text of the listener in the current dialogue round and synchronous expression parameters based on a codec fused with an attention mechanism; and generating a corresponding 3D facial animation frame sequence according to the expression parameters of the listener in the current dialogue round.

The invention relates to a micro-expression recognition method based on staged adaptive course learning. The method comprises the following steps: A, preprocessing a micro-expression video sequence and a macro-expression video sequence; b, constructing a spatial-temporal feature fusion model, performing deep feature extraction on the micro-expression data set obtained by preprocessing, and pre-training a macro-expression recognition teacher model; c, constructing a deep learning algorithm based on staged adaptive curriculum learning, and introducing a micro-expression recognition-oriented deep learning algorithm based on staged adaptive curriculum learning in the training process of the constructed spatio-temporal feature fusion model to optimize the training process; and D, carrying out classification identification on the macro expression identification teacher model obtained by training on a test set. According to the method, more effective and discriminative micro-expression features are obtained, the generalization ability of the model is improved, and the problems that in the existing micro-expression recognition field, available data sets are lacked, redundant information contained in the data sets is large, and the recognition accuracy is not high are further solved.

The present invention provides a single-image-based three-dimensional face reconstruction and editing method, comprising: extracting features by means of a pre-trained EG3D network, so as to generate a preliminary three-dimensional face; using an inversion module to map a monocular image to a latent code space, and optimizing a latent code to generate a realistic three-dimensional face image; making use of a depth estimation technology and multi-view projection to create a pseudo multi-view image; then, by means of semantic segmentation and expression feature extraction, fusing the features to generate a comprehensive representation; according to editing requirements, optimizing and adjusting the latent code, so as to implement customized editing; and finally, a generator outputting an edited face image. The present invention can enable more efficient, flexible and high-quality three-dimensional face reconstruction and editing, thereby effectively overcoming the defects of high cost, low universality and lack of customized editing in existing face reconstruction technologies.

The invention relates to a device and a method for assisting in detecting depression degree by utilizing emotional picture cognitive deviation, and relates to the technical field of biomedical engineering and medical instruments, and the device comprises an emotional picture library module which is used for storing a pre-selected emotional picture set; the display interaction module is used for displaying pictures to the user and recording classification selection and reaction time of the user; the data acquisition module is used for acquiring user facial micro-expressions and synchronously storing the facial micro-expressions and classification results; the data processing module is used for calculating positive picture error rate related features and normalizing behavior features and micro expression features into feature vectors; the depression index calculation module is used for outputting depression indexes; and the result output module is used for generating a visual report. The depression index is calculated based on behavior data, deviation is avoided, multi-modal features are fused to improve quantization precision, low-cost non-intrusive screening is achieved through a touch screen and a camera, and an embedded processor is integrated to improve the integration level and generalizability.

The invention discloses a student state real-time analysis method and device based on deep learning, and the method comprises the steps: obtaining and preprocessing image data of students in a classroom, and carrying out the multi-scale face detection and facial feature extraction of the preprocessed image data; expression features are extracted based on standardized facial feature data, fusion is carried out in combination with attention indexes and attitude features, a time sequence feature sequence is constructed, a heavy-tailed recurrent neural network is applied to carry out time sequence modeling, an evaluation standard is established, evaluation parameters are adjusted through a self-adaptive threshold value, and finally a student state evaluation result is obtained. Through the heavy-tailed recurrent neural network and a slow transition mechanism to low-dimensional chaos, subtle changes and long-term trends of student states can be accurately captured, and the technical problems that a traditional student state monitoring method is poor in real-time performance, limited in coverage and insufficient in individuation are solved.

The invention relates to the technical field of facial dynamic video image pain assessment, in particular to a facial dynamic video image pain assessment method based on a dynamic fusion module, and the method comprises the steps: collecting data through a multi-modal sensor, and generating a three-dimensional feature mapping map; activating an adaptive weight adjustment module to generate a configuration parameter set; executing micro-expression feature extraction and dynamic fusion operation to output a high-precision feature vector; inputting a grading evaluation model to complete pain degree quantitative grading. According to the method, facial micro-expression changes can be accurately captured, feature distortion can be dynamically recovered, nonlinear feature components can be mined, feature extraction integrity and evaluation accuracy can be improved, meanwhile, the feature capture capability in a complex environment can be enhanced through a controlled gradient optimization dynamic fusion technology, and the calculation efficiency can be optimized.

The invention discloses an image facial feature extraction and fusion method and system and a memory, and the method comprises the steps: carrying out the feature extraction of a source role facial image, and generating a basic feature map and an identity feature vector; performing decoupling processing on the basic feature map by taking the identity feature vector as a control condition, and separating an expression feature channel and an illumination feature channel; inputting the decoupled expression feature channel and illumination feature channel into a scene perception gating network, and generating an adaptive fusion feature map in combination with the target scene image; executing Poisson fusion and generative adversarial refinement on the adaptive fusion feature map, and outputting an optimized fusion image; through a decoupling mechanism guided by an identity feature vector, texture distortion and illumination sensitivity are avoided, a scene perception gating network dynamically adjusts a feature fusion weight, cross-scene adaptability is remarkably improved, poisson-generative adversarial two-stage fusion introduces antagonism refinement on the basis of gradient domain optimization, boundary splicing traces are eliminated, and the fusion precision is improved. And high-fidelity and cross-scene facial feature fusion is realized.

The invention provides an intelligent computing platform abnormal root cause positioning method fused with a large language model and related equipment, and relates to the technical field of anomaly detection. The method comprises the steps of firstly obtaining semantic expression features of interaction between a target student and a large language model, and judging whether the interaction obstacle degree exceeds a preset value or not; if yes, the feedback type of the obstacle is determined firstly, then dialogues in a preset window before the obstacle appears are searched, the starting point of the obstacle is positioned, and key semantics of the content of the obstacle in the period are extracted. And then based on key semantics, searching a target learning group and a matching dialogue of users in a set area, identifying a plurality of error types and determining a basic score, matching the error types with an obstacle feedback type to obtain a matching score, combining the error types with the obstacle feedback type to determine a basic error type, and finally modifying and generating a reply and feeding back the reply to students. By implementing the method, a closed loop from obstacle identification to root cause positioning to generation of targeted reply is realized, and the accuracy and efficiency of abnormal root cause positioning are greatly improved.

The invention provides an image processing method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring image data of a facial expression; extracting a first expression feature and a first identity feature from the image data; performing feature splicing on the first expression feature and the first identity feature to obtain a first joint feature, and extracting joint information between the first expression feature and the first identity feature from the first joint feature; estimating mutual information between the first expression feature and the first identity feature, and adjusting the first expression feature into a second expression feature by taking minimization of the mutual information as a target; predicting a classification result of the facial expression based on the second expression feature and the joint information; therefore, expression and identity collaborative modeling is realized.

The invention relates to the technical field of image recognition and biological feature authentication, and particularly discloses a client identity real-time checking method based on image recognition. The method comprises the following steps: synchronously acquiring a certificate image, a real-time video stream and environmental parameters, and preprocessing to generate an environmental parameter matrix and a biological pulse sequence; certificate static texture features and video dynamic micro-expression features are separated and extracted through a double-flow network, and feature weights are dynamically adjusted and dimensionality reduction is carried out based on environmental parameters; the historical weight and the illumination factor are fused to generate a dynamic fusion coefficient, and feedback optimization feature extraction is carried out; decoding the iris tremor frequency to generate a living body confidence coefficient, combining the dynamic weight fusion features to obtain an initial similarity, and carrying out multiplication fusion on the initial similarity and the living body confidence coefficient; a micro-expression periodic jitter mode is detected, and risk factors are output after fraud punishment is applied; and triggering model re-calibration based on the risk factor. According to the method, the verification precision and the anti-counterfeiting capability under the scenes of complex illumination, head deflection and screen attack can be remarkably improved, and high-robustness identity authentication is realized.

The invention relates to the technical field of psychological state assessment, and particularly discloses a psychological state assessment method and system based on environmental perception, and a readable storage medium, and the method comprises the following steps: collecting an environment scene and an assessed object behavior image through an RGB camera, and extracting an object category, a spatial layout and a dynamic event through a semantic segmentation model; acquiring an environment voiceprint through a microphone, and recognizing a voice category by using a voice recognition model; synchronously acquiring facial micro-expression feature points, voice rhythm features and head, neck and shoulder vibration frequencies of the evaluated object; and generating a semantic vector from the object / scene label through a semantic embedding model. According to the psychological state assessment method based on environmental perception, through three core innovations of quantitative modeling of environmental semantics, causal reasoning and a dynamic attention mechanism and personalized adaptive calibration, the neglect limitation of traditional psychological assessment on environmental factors is broken through, and the accuracy, robustness and practicability of psychological state assessment are remarkably improved.

The invention discloses a full-process closed-loop nursing assistant robot control system based on multi-modal interaction. The system comprises a multi-modal sensing module, a facial expression analysis module, an action evaluation module, a closed-loop control module, an execution driving module and a data interaction module. The multi-modal sensing module integrates multi-source information to realize accurate feature fusion; the facial expression analysis module generates expression feature vectors; the action evaluation module calculates action normative parameters; the closed-loop control module generates a dynamic task queue and distributes instructions; the execution driving module responds to the instruction to complete operation; and the data interaction module realizes data transmission and storage. The system overcomes the defects that in the prior art, multi-modal information fusion is insufficient, and control closed-loop real-time performance and adaptability are poor, nursing accuracy and efficiency are improved, accurate and personalized nursing requirements are met, manual nursing defects are made up, and operation normalization is unified.

The invention provides a digital human auxiliary diagnosis method and device, electronic equipment and a readable storage medium, and is applied to the technical field of medical data processing. The method comprises the following steps: processing question and answer audio information of a patient to generate question semantic information and sound feature information of the patient; processing the real-time facial video information and the physiological status information of the patient to generate facial micro-expression feature information and physiological index feature information; processing the sound feature information, the facial micro-expression feature information and the physiological index feature information based on a multi-modal fusion network to generate real-time emotional state information of the patient; processing the real-time emotional state information of the patient, the question semantic information of the patient and the preference information of the patient to generate target reply style information of the digital person; and processing the real-time operation information of the hospital department of the target area and the historical case record information of the patient based on the question and answer planning network, generating resource allocation result information, and generating digital human reply information.

The invention relates to the technical field of movie and television animation production, in particular to a movie and television animation character expression generation method and system combined with motion capture, and the method comprises the steps: firstly collecting a real-time motion data set of a target actor through motion capture equipment, including face and body motion data subsets; the facial action data is composed of facial key point displacement data with multiple continuous timestamps, then calling a preset expression mapping model to decouple the facial action data to obtain a basic and personalized expression feature set, then adjusting the basic expression feature set according to preset emotion parameters of an animation role, generating an emotion adaptive expression feature set, and generating a personalized expression feature set according to the emotion adaptive expression feature set; the method comprises the steps of obtaining a personalized expression feature set, fusing with the personalized expression feature set to obtain a target expression feature set, finally driving a three-dimensional face model to generate an expression animation by using the target expression feature set, and binding body action data to a three-dimensional body model to generate a complete character animation, thereby improving the authenticity and expressive force of animation character expressions.

The invention relates to the field of image detection, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a learning state monitoring method, device, equipment and medium, the method comprises the following steps: obtaining a learning video of a target student, and carrying out face area identification on the learning video to obtain a target face area; performing expression recognition and eye expression recognition on the target face region to obtain a face expression feature and an eye expression region feature; performing human skeleton recognition on the learning video to obtain target human skeleton key points, and recognizing posture angle features according to the target human skeleton key points; acquiring a learning audio, and performing voice semantic analysis on the learning audio to obtain voice semantic features; determining a learning state concentration degree according to the facial expression feature, the eye expression region feature, the attitude angle feature and the voice semantic feature; and performing learning state monitoring according to the learning state concentration degree to obtain a state monitoring result. According to the invention, the state monitoring efficiency and accuracy can be improved.

The invention relates to a three-dimensional dynamic face reconstruction method based on mixed features and regional expressions, and belongs to the technical field of computer vision, and the method comprises the following steps: S1, collecting monocular dynamic face video data, and carrying out the preprocessing; s2, acquiring a multi-resolution hash feature and a two-dimensional plane hash feature of the three-dimensional space point, and splicing the two features to construct a mixed feature; s3, key facial expression dimensions are screened out through variance analysis, and multi-resolution hash features are fused to construct regional expression features; s4, inputting the mixed features and the regional expression features into a density network, and calculating volume density and geometric features; s5, color feature branches irrelevant to the visual angle are introduced, and colors of the three-dimensional space points are generated in combination with the color feature branches relevant to the visual angle. According to the method, the accuracy and reality of three-dimensional face reconstruction can be effectively improved, and meanwhile, the training speed is increased, so that rapid and high-fidelity three-dimensional face reconstruction is realized.

The invention discloses a facial expression migration method and device based on an adversarial auto-encoder, equipment and a medium, and relates to the technical field of expression migration. The facial expression migration method comprises the following steps: S1, acquiring a target image for providing appearance features and a source image for providing expression features; s2, extracting an expression feature vector according to the source image; and S3, encoding the target image and the source image by using an encoder to obtain depth features. And S4, adjusting the depth feature in combination with the expression feature vector so as to apply the expression of the source image to the target image, and obtaining an expression fusion feature. And S5, performing decoding processing on the expression fusion features, and performing reconstruction to generate a preliminary image. And S6, performing multi-scale optimization on the preliminary image through a depth facial semantic module to obtain an optimized image. And S7, performing super-resolution reconstruction processing on the optimized image to generate an output image. Wherein the output image migrates the expression dynamic state of the source image while keeping the identity characteristics of the target image.

The invention provides a customer service platform based on an AI digital human, a service method, equipment and a medium, and relates to the field of intelligent customer service, and the platform comprises a front-end display layer which is used for generating a chat interface for interaction between a user and the AI digital human; the digital human customer service interaction layer is used for performing sentiment analysis on the query content to obtain a sentiment category corresponding to the query content; matching a target virtual image and a target expression feature corresponding to the emotion category from a configured comparison table of different emotion categories, different virtual images and different expression features; generating a target virtual image of the AI digital human according to the target virtual image and the target expression feature; outputting response content by using the target virtual image voice; and the business processing layer is used for matching response content corresponding to the query content from the configured customer service knowledge base after receiving the query request. According to the application, the AI digital human image can be combined to realize multi-terminal efficient interaction intelligent customer service.

The invention relates to an interview auxiliary method based on multi-modal data identification, and the method comprises the steps: collecting multi-modal data of a job seeker in an interview process, carrying out the alignment time sequence processing of the multi-modal data, and enabling the multi-modal data to comprise a video stream, an audio stream and text data; judging whether the job seeker is in an abnormal state or not based on micro-expression features, voice features and text features extracted from the multi-modal data of the aligned time sequence; when the job seeker is in an abnormal state, dynamically adjusting feature weights corresponding to the micro-expression features, the voice features and the text features based on an abnormal state probability; calculating an abnormal state probability value based on the adjusted feature weight, the micro-expression feature, the voice feature and the text feature; calculating an abnormal state type according to the abnormal state probability value and a type threshold value; and displaying interview prompt information to an interviewer according to the abnormal state type. According to the method, the misjudgment rate of abnormal states is reduced, and the accuracy and effectiveness of interview analysis are improved.

The invention discloses a motion expression video segmentation method and system based on task decoupling, and the method comprises the steps: respectively inputting a video and a description text into a decoupling motion expression video segmentation frame for processing, i.e., receiving the video through a frozen video instance divider, and tracking and segmenting all candidate objects in the video; generating and inputting a frame query and a video query to a motion expression encoder; extracting motion word features and static word features from the description text and inputting the motion word features and the static word features to a motion expression encoder; the motion expression encoder interacts the frame query, the video query, the motion word features and the static word features to generate motion expression features; and meanwhile, initializing a motion query by adopting a video query, inputting the initialized motion query and the motion expression feature into a motion query decoder for decoding to obtain a decoded motion query, and identifying an object specified by the description text based on the decoded motion query. Compared with an existing method, better performance is achieved under lower training cost.

The invention relates to a micro-expression recognition method for cross-domain feature center assisted emotional intensity invariance feature extraction, belongs to the technical field of deep learning and pattern recognition, designs a cross-domain feature center assisted emotional intensity invariance feature extraction network, and fully utilizes an existing macro-expression data set to assist micro-expression recognition. Macro-expression related features are utilized to guide learning of micro-expression features, a neural network is helped to learn more emotion related features, hyper-spherical constraint is carried out on the micro-expression features and the macro-expression features, multiple angle optimization strategies are designed, intensity information of the features is weakened, angle information is focused, and the effect of improving the accuracy of the micro-expression features is achieved. In addition, a macro-expression feature center and a micro-expression feature center are designed to cooperatively guide training of the network, and a model is helped to learn more compact emotional feature distribution.

The invention discloses a doctor-patient speech communication model training method and system based on multi-modal corpus analysis, and belongs to the crossing field of artificial intelligence and medical treatment, and the method comprises the steps: carrying out the extraction according to an OpenPose algorithm to obtain motion features, carrying out the muscle activity intensity detection to obtain expression features, carrying out the speech recognition to obtain text features, and carrying out the recognition of the text features; speech segmentation is carried out based on the time domain energy parameters, and a segmentation result is subjected to modal analysis to obtain acoustic features; performing feature fusion based on an attention mechanism to obtain fusion features, and inputting an obstacle recognition model to obtain an obstacle type; the method comprises the steps of obtaining an obstacle type, obtaining an intervention strategy and an intervention identity according to a solution mapping relation, taking the obstacle type, the intervention strategy and the intervention identity as multi-dimensional labels, carrying out time domain alignment and structured packaging to obtain target data, and training according to the target data to obtain a communication model used for providing dialogue prompt information. Obstacle recognition accuracy can be improved, and the intelligent agent application effect can be improved.

A methodology for auditing the visual diversity of unlabeled human face image datasets uses a set of core human interpretable dimensions derived from human similarity judgments. Given a face image, a model can output dimensional values aligned with the human mental representational space of faces, where values not only express the presence of a feature, but also its extent. Since the model can be learned entirely from human behavior, the learned dimensions are not biased toward features that are easier to verbalize or quantify.

A model processing method includes, obtaining a first source head model and a plurality of pieces of expression feature data; performing deformation matching on the first source head model according to a first model feature of a target head model, to obtain a second source head model; determining a deformation parameter according to a first deformation relationship between the first and second source head models and a second deformation relationship from a first model expression to a second model expression, the first model expression being determined according to neutral feature data of the first source head model, and the second model expression being determined according to expression feature data indicated by an expression movement instruction in the plurality of pieces of expression feature data; and performing expression transfer on the target head model according to the deformation parameter, to obtain a target head model having the second model expression.

The invention discloses a self-adaptive fatigue state detection method based on multi-modal feature fusion, and belongs to the technical field of fatigue detection, and the method comprises the steps: synchronously collecting facial images through an infrared camera and an RGB camera, carrying out the image fusion through a Laplacian pyramid method, extracting facial features, setting a personalized threshold value, and obtaining fatigue features. Performing special scene judgment under limited face recognition, considering micro-expression features, performing fatigue parameter supplementation, judging a fatigue state through a comprehensive fatigue model with multi-fatigue feature fusion, and performing early warning information reminding; according to the adaptive fatigue state detection method based on multi-modal feature fusion, through multi-modal feature fusion, image details are effectively reserved, a low-light environment is adapted, facial feature deficiency is compensated, and the accuracy and stability of fatigue state detection are remarkably improved by combining personalized threshold values and special scene judgment.

The invention relates to the technical field of image processing, and discloses a facial health state assessment method and system based on layered optical modeling. The objective of the invention is to solve the problems of poor face health state evaluation precision and robustness caused by rough optical modeling, mixed reflection components, weak environmental adaptability and insufficient multi-dimensional health index fusion capability in the prior art. The method comprises the following steps: synchronously acquiring a user face video sequence comprising at least two different spectral bands through a multispectral imaging device; constructing a multilayer optical transmission model of the skin tissue, and separating a specular reflection component and a diffuse reflection component from the face video sequence; extracting a physiological time sequence signal related to subcutaneous blood flow activity from the separated diffuse reflection component, and calculating heart rate, heart rate variability and blood oxygen saturation parameters; and fusing the physiological parameters and expression and micro-expression features extracted from the facial video sequence, and outputting quantitative evaluation results of fatigue degree, pressure level and emotional state through a pre-trained deep neural network model. The system comprises a multispectral image acquisition module, a layered optical modeling module, a reflection component separation module, a physiological parameter inversion module and a health state evaluation module. According to the technical scheme, the signal-to-noise ratio and stability of physiological signal extraction in a complex illumination environment can be remarkably improved, the adaptability to individuals with different skin colors and dynamic illumination conditions is enhanced, and more accurate and more robust judgment of the high-order health state is achieved.

A multimedia data processing method and apparatus, and a computer-readable storage medium are disclosed. The method may include: acquiring an audio stream and a video stream of multimedia data; parsing the audio stream to obtain text feature data, and matching the text feature data according to a preset mapping relationship to determine topic feature data; parsing the video stream to obtain expression feature data, and matching the expression feature data according to a preset mapping relationship to determine an emotion index; and rendering the multimedia data based on the text feature data, the emotion index, and the topic feature data.