190 results about "Visual modality" patented technology

The invention relates to a multi-modal sentiment analysis method based on deep decoupling and cross-modal semantic alignment. The method comprises the following steps: carrying out preprocessing and feature extraction on input video data; the three modes are decoupled through a feature decoupling mechanism based on the HSIC criterion; performing cross-modal alignment on the three modal semantics; hierarchically predicting emotion categories; and finally outputting a prediction result. According to the method, by decoupling features of three modes of text, audio and vision, unique attributes of each mode are separated from shared emotional semantics, and the problem of redundancy and conflict information between modes in an existing multi-mode emotional analysis method is solved. On the basis, a cross-modal alignment mechanism based on text guidance is provided, shared features are aligned through a cross-modal attention mechanism, and semantic consistency among different modals is enhanced. Meanwhile, the decoupled unique features and the aligned shared features are effectively fused through a hierarchical fusion strategy, and the accuracy of emotion prediction is improved.

The invention discloses a vision-guided robot active path detection method, and relates to the technical field of robots, and the method comprises the steps: obtaining visual modal data, laser radar modal data and depth modal data of a detection area of a robot in real time through a multi-modal sensor; preprocessing the visual modal data, the laser radar modal data and the depth modal data by using a convolutional neural network; a plurality of terrain type regions are separated from the preprocessed visual modal data, the preprocessed laser radar modal data and the preprocessed depth modal data by applying a color space conversion strategy; performing target detection on the plurality of terrain type regions through a deep learning model, and extracting candidate regions of terrain obstacles; pixel segmentation is carried out on the candidate region of the terrain obstacle, and the position and the region boundary of the terrain obstacle are determined; and adjusting the real-time advancing path of the robot according to the constructed cost map.

The invention belongs to the technical field of three-dimensional scene modeling, and discloses a dual-branch diffusion three-dimensional scene generation method based on a multi-modal semantic graph, which comprises the following steps of: firstly, receiving multi-modal data such as sketches, texts, automatic completion instructions and scene general knowledge, extracting features and fusing the features into a unified multi-modal semantic graph; utilizing a graph neural network and an attention mechanism to enhance semantic graph features, and optimizing physical constraints through a physical engine; complementing the missing visual modality and graph structure relationship; performing quality scoring on the scene based on semantics, spatial relationships and physical constraints; and finally, respectively generating a spatial layout and a geometric shape through a double-branch diffusion model, and ensuring the coordination of the layout and the shape. The method has the advantages of multi-modal information fusion, physical rationality guarantee, high structure complementation capability, high-quality score optimization and efficient generation process, and is suitable for three-dimensional scene modeling requirements in the fields of virtual reality, augmented reality, robots and the like.

The invention belongs to the technical field of image processing, and discloses a multi-image processing method based on multi-modal entity alignment, which comprises the following steps: acquiring a plurality of images; retrieving rich semantic information of the entity from an external knowledge base; selecting the most representative image for each entity in the multi-image scene by using semantic information; encoding the original input of each mode; performing first-layer fusion between a visual mode and a text mode by using cross-divergent attention, then performing second-layer interaction with a structured mode, and finally aligning an entity representation of an image by using contrast loss; and outputting multiple images fusing the text modality and the visual modality. According to the invention, hierarchical interaction fusion is utilized to enhance multi-modal interaction; enhancing entity text representation by integrating the external attribute value and the context information; and the most representative image is selected by using the semantic text, so that the influence of uncorrelated images is minimized.

The invention relates to the technical field of natural language processing, in particular to a multi-modal dialogue emotion recognition method and system based on cross-modal fusion and comparative learning. According to the method, learnable residual scaling and pre-normalization are introduced through a cross-modal encoder, deep interaction of texts, voices and visual modals is stabilized, and gradient explosion is inhibited; a dialogue graph fusing semantic similarity and time proximity is constructed online through a semantic-time sequence graph enhancement module, and a graph attention network is used for explicitly modeling long-distance round dependence and cross-speaker emotion transmission; through an adaptive comparison and alignment module, a dynamic scheduling comparison loss and index moving average updated mode-emotion prototype library is adopted to realize data distribution adaptive cross-mode alignment; through cooperative work of the modules, the problems that in the prior art, cross-modal fusion is unstable, long-distance and cross-speaker dependence modeling is insufficient, and cross-dataset alignment capacity is weak are solved.

The invention provides a knowledge distillation method, system and equipment for a multi-modal large model and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting multi-modal data sent by a user side into a visual modal analysis unit, a language modal analysis unit and a voice modal analysis unit, correspondingly generating a visual feature vector, a language feature vector and a voice feature vector; carrying out feature fusion to generate a fused feature matrix; on the basis of the fusion feature matrix, migrating knowledge of the teacher model to the student model; obtaining a prediction result output by the student model, feeding back the prediction result to the user side, and receiving feedback information sent by the user side; and the feedback information is stored in a dynamic memory bank, and incremental learning is performed on the student model based on the gradient direction vector stored in the dynamic memory bank. The method has the technical effects that dynamic optimization is carried out according to the actual use condition of the user, so that the model performance meets the actual application requirement.

The invention discloses a method and system for evaluating infusion operation of intern medical staff, and belongs to the field of infusion, and the method comprises the steps that the operation of the intern medical staff is collected in multiple modes, and the multiple modes comprise a visual mode, a touch mode and a physiological signal mode; inputting the collected operation of the intern medical staff into the multi-modal infusion evaluation model, performing feature extraction by the multi-modal infusion evaluation model in a modal manner, and performing cross-modal attention fusion on feature vectors of different modals; and finally, comparing the calculated total feature tensor Hfusion with a standard total feature tensor Hfusion ', and outputting a final score of the practice medical personnel according to a preset scoring standard. According to the scheme, multi-dimensional data such as visual, tactile and physiological signals are synchronously collected through a multi-modal fusion method, operation details are comprehensively covered, comprehensive evaluation is carried out, and evaluation objectivity is improved.

The invention provides a textile cloth defect real-time detection method and system based on multi-modal feature fusion, and the method comprises the steps: collecting visual image data, infrared thermal imaging data and ultrasonic acoustic data of textile cloth through a multi-sensor array, and forming multi-modal input; performing time sequence alignment and noise filtering preprocessing on the multi-modal data to eliminate motion artifacts and environmental interference; a parallel feature extraction module is used for extracting texture features from the visual data, extracting temperature distribution features from the thermal imaging data and extracting acoustic impedance features from the acoustic data. By deeply fusing complementary information of three modes of vision, thermal imaging and ultrasonic wave, the detection capability is improved, the vision mode captures surface texture details, the thermal imaging mode reveals thermodynamic anomalies related to friction and materials, the ultrasonic wave mode perceives subcutaneous structure defects, hidden flaws which cannot be recognized by a single mode can be found, and the detection efficiency is improved. Therefore, the omission ratio is greatly reduced, and flaw types are distinguished more accurately.

The embodiment of the invention provides a multi-mode fusion man-machine interaction control method, system and device and a storage medium, and relates to the technical field of intelligent driving, and the method comprises the steps: obtaining vehicle driving data and driver state data; based on the vehicle driving data and the driver state data, calculating fusion weights of a visual mode, an auditory mode and a tactile mode to obtain a multi-mode fusion weight matrix; generating a multi-modal interaction signal according to the multi-modal fusion weight matrix, wherein the multi-modal interaction signal comprises a visual signal, an auditory signal and a tactile signal; and triggering corresponding visual warning, auditory warning and tactile warning according to the multi-mode interaction signal. In this way, multi-mode fusion is conducted on vision, hearing and touch, the output intensity of visual warning, hearing warning and touch warning is adjusted according to the weight of each mode of vision, hearing and touch, a driver is helped to make the most urgent operation at present according to the warning of each mode, and the driving safety of man-machine interaction in the driving process is improved.

The invention provides a cross-modal semantic analysis method, and relates to the technical field of data processing, and the method comprises the steps: calling a visual modal analysis strategy to analyze a source video, and obtaining visual analysis information; calling an auditory modal analysis strategy to analyze the source audio to obtain auditory analysis information; performing stage comparison analysis on the visual analysis information and the auditory analysis information; if the emotion comparison result reaches a predetermined comparison constraint, analyzing to obtain real-time modal weight distribution; and obtaining character analysis information, and fusing the character analysis information, the visual analysis information and the auditory analysis information in combination with real-time modal weight distribution to obtain a semantic analysis result. By means of the cross-modal semantic analysis method and device, the technical problem that in the prior art, the semantic analysis accuracy is low due to the fact that the information structures and expression modes of all the modes are large in difference is solved, and the accuracy and adaptability of cross-modal semantic analysis are improved through the semantic analysis method combining the visual mode, the auditory mode and the text mode.

Various embodiments classify one or more portions of an image based on deriving an “intrinsic” modality. Such intrinsic modality acts as a substitute to a “text” modality in a multi-modal network. A text modality in image processing is typically a natural language text that describes one or more portions of an image. However, explicit natural language text may not be available across one or more domains for training a multi-modal network. Accordingly, various embodiments described herein generate an intrinsic modality, which is also a description of one or more portions of an image, except that such description is not an explicit natural language description, but rather a machine learning model representation. Some embodiments additionally leverage a visual modality obtained from a vision-only model or branch, which may learn domain characteristics that are not present in the multi-modal network. Some embodiments additionally fuse or integrate the intrinsic modality with the visual modality for better generalization.

The present application provides a video segmentation method, a server, a storage medium, and a program product. In the method of the present application, video data to be segmented is segmented into multiple data segments, unimodal features of the data segments, including text features of a text modality and visual features of a visual modality, are respectively extracted by means of a video topics segmentation model, and then the text features and visual features of the data segments are fused, so that the fusion of multimodal information can be performed at the intermediate representation level, the relationship and interaction between different modalities can be better captured, and higher-quality multimodal fusion features of the data segments are obtained. Furthermore, on the basis of the multimodal fusion features of the data segments, whether the data segments are topic boundaries is predicted, so that the topic boundaries of the video data can be accurately predicted, improving the accuracy of topic boundary recognition, thereby improving the accuracy and quality of video topics segmentation results.

A multimodal video generation framework (MMVID) that benefits from text and images provided jointly or separately as input. Quantized representations of videos are utilized with a bidirectional transformer with multiple modalities as inputs to predict a discrete video representation. A new video token trained with self-learning and an improved mask-prediction algorithm for sampling video tokens is used to improve video quality and consistency. Text augmentation is utilized to improve the robustness of the textual representation and diversity of generated videos. The framework incorporates various visual modalities, such as segmentation masks, drawings, and partially occluded images. In addition, the MMVID extracts visual information as suggested by a textual prompt.

The invention relates to a multi-modal aspect sentiment analysis method and system based on prompt double-layer cross-modal distillation learning, electronic equipment and a readable storage medium, and belongs to the technical field of multi-modal sentiment analysis. An existing method has the problem of insufficient adaptability when facing various visual noise interferences in practical application, and most existing sentiment analysis models are specially designed for specific scenes and are difficult to adapt to different scenes such as visual mode deficiency. According to the invention, the method focuses on distillation learning and aspect perception, and guides the model to focus on aspect information through prompts constructed by aspect words. By minimizing the distillation loss between the output of the multi-mode sentiment analysis model and the current optimal multi-mode sentiment analysis model, the multi-mode representation can be effectively and selectively learned, and the multi-mode representation can be effectively and selectively learned even under the condition of lack of visual information or existence of visual noise interference. The method disclosed by the invention shows extremely high accuracy and robustness in a sentiment analysis task containing noise vision.

The invention relates to a graph convolution multi-modal dialogue emotion recognition method based on emotion dimension compensation. The method belongs to the field of multi-modal emotion recognition. Comprising the following steps: respectively extracting original features of texts, audios and visual modalities by utilizing a pre-training model; detecting a mode missing condition, and generating a pseudo feature by using an available mode feature and context information; the multi-modal features are mapped to a VAD three-dimensional space, and prediction and enhancement are carried out; the global level is based on VAD similarity to connect cross-utterance nodes, and emotional consistency is quantified to capture long-distance interaction. In the local level, information spreading and node updating are carried out by calculating the similarity between different modal features in the same utterance; neighbor information is accumulated through multiple layers of image volumes, node weights are dynamically adjusted to suppress noise propagation, and finally utterance-level emotional representation is generated by fusing multi-modal features through mean pooling operation. According to the method, the emotion recognition performance in a modal missing scene is remarkably improved.

A method for evaluating a user's performance of a medical procedure is disclosed. The method comprises receiving a plurality of training measured metrics collected from a sensor worn by the user during training medical procedures, where these metrics include physiological and biomechanical parameters. An algorithm is trained to identify types of medical procedures, procedural segments, and performance benchmarks. During the medical procedure, the sensor collects similar metrics, which are analyzed to identify the procedure type, procedural segments, and deviations from benchmarks. Feedback is provided based on these deviations. Additional steps include using a training model comprising a neural network with machine learning algorithms to analyze metrics, updating the training model with new data, initiating sensor calibration, detecting a predetermined movement pattern to start data collection, and identifying spatial reference points. The method further comprises providing real-time feedback to the user through haptic, auditory, and visual modalities based on performance deviations.

The invention discloses a multi-modal sentiment analysis model based on hierarchical adaptive cross-modal fusion, belongs to the field of natural language processing, and is used for solving the problems that in the prior art, multi-granularity sentiment feature extraction is insufficient, a cross-modal fusion mechanism is rigid, and the distribution difference between different-source modals is large. The method comprises the following steps: firstly, extracting features of texts, audios and visual modalities from original video data, and coding the features into advanced semantic features; secondly, multi-granularity information is fused through a hierarchical feature extractor to generate enhanced single-mode features; then, a self-adaptive cross-modal fusion network with a text as a core is adopted to realize bidirectional interaction and dynamic weighted fusion between modals; further, a dynamic contrast learning mechanism is introduced to align modal distribution in a unified potential space; and finally, inputting the optimized multi-modal features into a classifier and outputting an emotion analysis result. According to the model, through collaborative optimization of multi-granularity feature extraction, adaptive fusion and comparative learning, the accuracy of sentiment analysis and the robustness of the model are remarkably improved.

A system configured to improve audio processing by performing audio-visual target speech enhancement. The system may include a deep neural network (DNN) configured to jointly mitigate additive noise, reverberation, and / or residual echo. The DNN may include an audio encoder / decoder network, which may correspond to a convolutional recurrent network configured to process complex-valued spectrograms corresponding to the isolated audio data generated during echo cancellation. In addition, the DNN may also include a video encoder / decoder network configured to process image data, enabling the DNN to make use of audio and visual modalities to enhance a target speech signal. Thus, the DNN may include (i) audio / video encoders that generate audio / image features, (ii) a fusion network that combines the audio / image features, and (iii) audio / video decoders that generate output data. By processing these multimodal inputs, the DNN may distinguish between desired speech associated with a face represented in the image data and interfering speech.

The invention is suitable for the technical field of computers, and provides a multi-modal fire rescue detection method and system, and the method comprises the steps: obtaining the related information collected by a rescue area, carrying out the multi-modal feature extraction, and selecting the multi-modal feature fusion; performing deconvolution and up-sampling operation on the prior information, and outputting a target position search prompt; multiplying the fire behavior temperature distribution characteristic Fuc by a calibrated temperature range to obtain an actual temperature at the next moment so as to assist rescue; according to the method, voice features, infrared features and visual features are combined, and sufficient feature fusion is performed, so that the accuracy and recall rate of detecting and recognizing the position information of the trapped person are improved; and meanwhile, audio modality is added, so that the limitation that the visual modality cannot be detected due to building shielding objects, smoke and other reasons can be avoided, and the detection effect is more robust.

The invention provides a pet behavior prediction method and apparatus, a device and a storage medium. The method comprises the steps of obtaining a pet type, image data, audio data and text data describing pet behaviors; performing feature extraction and multi-modal feature fusion on the image data, the audio data and the text data to obtain fusion features of the pet; and based on the type of the pet and the fusion feature, predicting the behavior of the pet by adopting a pre-constructed pet behavior prediction large model to obtain a behavior prediction result of the pet. When pet behaviors are predicted, visual modal data are introduced, auditory features of an audio modal and situational semantics described by an owner text are synchronously fused, behavior ambiguity of single visual data is eliminated through a cross-modal dynamic weighting mechanism, emotion clues in sound signals and recessive states described by the text are captured, and the pet behaviors are predicted. Therefore, a multi-dimensional cognitive model of pet behaviors is comprehensively constructed, and the prediction accuracy is remarkably improved.

The present invention relates to the field of computer vision, and more particularly to a multimodal personality perception method and apparatus that utilizes multiple association features and graph relationship attention. The method comprises: obtaining an input video for personality perception; inputting the input video into a data preprocessing module to obtain visual modality input, audio modality input, and text modality input; inputting the visual modality input, audio modality input, and text modality input into a modal feature extraction network module to obtain scene-audio association features, scene-description word association features, audio-description word association features, and text modality features; inputting the scene-audio association features, scene-description word association features, audio-description word association features, and text modality features into a feature fusion module to obtain multimodal fusion features; and inputting the multimodal fusion features into a perception prediction module to obtain personality perception results. The present invention proposes a multimodal attention fusion framework for personality perception.

The invention relates to a method and a device for generating a report under the assistance of a label, and the method comprises the following steps: 1) extracting a structured label set from a text report of a sample based on a large language model, wherein the structured label set comprises a multi-classification group consisting of dichotomous labels and mutual exclusion options; 2) aggregating the labels in batches, after a threshold value is reached, merging and de-weighting, performing specification and mutual exclusion group merging on synonymous, near-synonymous and redundant labels, and converging into a unified label library; 3) based on the text report and the tag library, outputting a tag subset of each sample through a large language model; 4) multi-modal multi-label classification model training: extracting each visual modal feature, performing weighted aggregation and splicing, and outputting each label group logits through a classification head to perform weighted group loss optimization; (5) carrying out joint training on the multi-modal large language model by using samples of'only images-reports' and'images + labels-reports', and (6) carrying out label prediction and screening on the images by using the classification model, and inputting'images + prediction labels' into the multi-modal large language model to obtain a final report.

The invention discloses an ATS-oriented semantic-driven single-image-based 3D multi-target detection method, and the method comprises the steps: processing an input RGB image, and extracting a 3D bounding box of each object in the image; generating all potential 2D projections of the 3D object in the scene; extracting keywords, phrases and semantic information thereof in the description to form feature information Pt representing language description; fusing the 2D image information and the 3D geometric information of the object to obtain a complete object representation fa; correlating the language features extracted by the language description with the detected 3D object, and capturing a semantic correspondence relationship between the text and the visual modality; and filtering the targets according to the generated matching scores to obtain all targets according with natural language description. According to the method, the accuracy and efficiency of retrieval and recognition are improved, higher recognition precision is realized, the calculation complexity is remarkably reduced, the precision and speed of cross-modal retrieval can be improved, and the accurate recognition and positioning capability of traffic events is greatly improved.

The invention provides a Parkinson's disease early recognition system and method based on multi-task learning, and belongs to the technical field of machine learning. The method comprises the following steps: firstly, constructing a Parkinson's disease discrimination data set covering actions, languages and visual modalities, and providing comprehensive symptom data and identities for subsequent analysis; then, in combination with a medical priori rule, constructing a symptom recognition module integrated with a plurality of traditional machine learning models, and performing preliminary judgment on action and language data; a short-term recognition model is constructed to enhance the multi-modal symptom perception ability and improve the discrimination effect of Parkinson's disease short-term recognition; and finally, carrying out mathematical modeling analysis based on the recognition result sequence of the multiple time periods, fusing statistical trend and group difference, and realizing robust comprehensive judgment on the Parkinson's disease. According to the invention, through cooperative training and knowledge sharing among tasks, the comprehensive discrimination capability of the model on the multi-modal pathological signals of the Parkinson's disease is enhanced, so that more accurate and more sensitive early recognition is realized.

The invention discloses an interactive speaker detection method oriented to a first-person view angle. The interactive speaker detection method is used for identifying a speaker interacting with a camera wearer from the first-person view angle. The method comprises the following steps: firstly, preprocessing an input first-person video stream, and extracting a head clipping signal, a lip clipping signal and an audio signal; then, a visual speaker target recognition module is adopted to analyze head direction and lip motion features, a parallel sharing weight audio encoder extracts robust audio features from clean and mixed audios, and meanwhile, a visual mode missing sensing module is adopted to dynamically evaluate the availability of a visual mode and adjust dependence on multi-mode features; and finally, comprehensively analyzing the visual and audio information through a multi-modal feature fusion module, and generating a classification result of whether to talk to the camera wearer. According to the invention, through consideration of a real scene and module design, the method has significant robustness and accuracy in a complex scene, and can effectively solve the problems of visual mode loss and background noise interference.

Disclosed is a system to generate an adversarial pattern. The system receives an input specifying data describing a target object, an objective indicating whether to reduce detectability of the target object or to induce a false detection of a target type, and context parameters. The system generates candidate adversarial patterns for the target object using an AI-based generative algorithm. Each candidate adversarial pattern represents a potential solution for achieving the objective under context parameters. The system simulates a representation of the target object with candidate adversarial pattern in a virtual environment that models a plurality of sensor modalities. The simulation may use machine learning object detection models. The system analyzes the target object and generates, for each target object, a performance metric for ranking the candidate adversarial pattern based on an effectiveness of the simulated representation. The candidate adversarial pattern with highest rank is generated as an optimized adversarial pattern.

The invention discloses a multi-modal sentiment analysis method and device based on mutual information and cross-modal text enhancement, and the method comprises the steps: (1) carrying out the preprocessing of a public data set, and respectively extracting text, visual and acoustic features in the data set; (2) constructing an MICTE multi-modal sentiment analysis model, effectively fusing rich sentiment information in text modal information into a visual modal and an acoustic modal through a cross-modal text attention enhancement module CTEA, enhancing the relevance between modals at an input level through a mutual information maximization module MIM, further extracting key information related to tasks at a fusion level, and obtaining an MICTE multi-modal sentiment analysis model; eliminating redundant information; (3) training and verifying the MICTE model by using the training set and the verification set, and selecting the model with the best performance as a final model; and (4) inputting the test set into the finally trained MICTE model to obtain an emotion analysis result. According to the multi-modal sentiment analysis method, the relevance among different modals is deeply mined, and the quality of the fused modals is remarkably improved, so that the accuracy of multi-modal sentiment analysis is improved.

The invention discloses a method for accelerating reasoning of a multi-modal large language model based on speculative decoding, the input of the multi-modal large language model is composed of a system lexical element, a visual lexical element, an instruction lexical element and an output lexical element, and the method comprises the following steps: S1, extracting a predetermined number of samples from a ShareGPT data set which is finely adjusted by using an LLaMA model instruction, pre-training an Eagle small model, and obtaining a pre-trained Eagle small model; enabling the Eagle small model to establish a basic language generation capability in a plain text environment; s2, extracting a predetermined number of samples in a ShareGPT data set finely adjusted by an LLaVA model instruction, and migrating the capability of an Eagle small model by using a transition mode; s3, after the draft model is subjected to two-stage training, on the basis of the improved draft mechanism design, differential processing is carried out according to the characteristic difference between the text and the visual mode; according to the method, the performance of speculative decoding in a multi-modal task can be improved, the acceptance length of each verification of the target model is improved, and respective adaptation of the text lexical elements and the visual lexical elements in the draft model is realized.

The invention relates to a video question answering method and system based on multi-level alignment. The method comprises the following steps: generating multi-level visual features including global frame visual features and local frame visual features; generating multi-level text description according to the multi-level visual features, wherein the multi-level text description comprises global object description and local object description; and according to the generated multi-level visual features and the multi-level text description, establishing alignment between a visual mode and a text mode at an object level, a frame level and a video level, training a language model, and performing video question and answer by using the trained language model. According to the method, alignment between visual and text modes is established among object-level, frame-level and video-level multi-mode information, advanced performance can be obtained even if a small pre-training data set and few trainable parameters are adopted, and the method has wide practical value and application scenes.

A system is provided.SOLUTION: A system comprising: means for storing information; means for extracting key claims from the information using generative artificial intelligence to analyze the information; means for quantifying the key claims; and means for visually displaying the quantified key claims.SELECTED DRAWING: Figure 1