78 results about "Bi modal" patented technology

The application discloses a kind of light weight small target detection method and system based on bimodal channel reconstruction and adaptive fusion, the method includes: in RGB feature branch and IR branch respectively constructs a new light weight convolution module, and then obtains double branch light weight convolution module;Respectively extract visible light image and infrared image, input the obtained RGB and IR feature map to multimodal channel reconstruction and adaptive fusion module, generate bimodal fusion features for target detection;Introduce SD loss to construct total loss function to train light weight small target detection model, obtain final detection result for image test;The application can accurately extract small target features under different scales while maintaining light weight, highlight key target features in multimodal, suppress redundant information within modal, enhance the detection accuracy and efficiency of model to multimodal target in complex scene.

The application discloses a kind of based on multi-modal feature fusion sentiment analysis method and system, comprising: through Bi-GRU capture context relationship between text modal, speech modal and image modal each other, while based on cross-modal attention mechanism, text modal, speech modal and image modal are combined two by two, obtain the interactive sentiment representation between text-image, text-speech and image-speech modal, through the multi-head attention mechanism of regular term, text modal, speech modal and image modal are carried out joint sentiment representation, obtain the interactive sentiment representation of three kinds of modal, finally single modal, double modal and three modal emotion feature cascade are classified finally emotion.The application solves the problem that feature information is not enough rich due to the modeling of context information in the existing multi-modal sentiment analysis algorithm, also solves the information limited problem when using single-head attention mechanism for feature learning and the feature information redundancy problem existing in multi-head attention mechanism.

This invention discloses an open-set radar interference pattern recognition method, apparatus, and electronic device. The method includes: matched filtering of the radar received signal to construct a dual-modal input of a one-dimensional range sequence and a two-dimensional time-frequency matrix; feature extraction via a dual-branch network and adaptive fusion based on cosine similarity; introducing cross-modal consistency regularization during the training phase, constructing pseudo-unknown samples by shuffling intra-batch modes, and combining energy constraint loss to compress the energy of known classes and increase the energy of unknown classes to form a clear decision boundary; and adaptively setting a threshold based on the energy distribution of the validation set during the inference phase to achieve accurate identification of known interference and effective rejection of unknown interference. This invention overcomes the performance degradation defects of traditional closed-set methods in the face of unknown interference, and improves the generalization ability, robustness, and engineering practicality of radar interference recognition in complex electromagnetic environments.

This invention discloses a wired dual-modal imaging capsule endoscopy system, belonging to the field of medical device technology. The system includes a capsule module, a transmission and control module, a dual-modal imaging module, and a signal analysis module. The capsule module is used to achieve 360° circumferential scanning imaging of the esophageal mucosa. The transmission and control module ensures stable transmission of optical and electrical signals and precise control of capsule movement. The dual-modal imaging module integrates an optical coherence tomography submodule and a photoacoustic imaging submodule to acquire tissue morphology information and microvascular distribution information, respectively. The signal analysis module processes the dual-modal signals. This invention achieves three-dimensional high-resolution integrated imaging of the esophageal mucosa without the need for exogenous labeling agents, significantly improving the detection accuracy of early esophageal cancer and precancerous lesions.

The application provides a multi-modal caries depth intelligent interpretation method fusing intraoral photos and X-ray films, and through standardized oral multi-modal medical image and pathological data synchronous collection, spatial registration and pixel-level normalization, the spatiotemporal consistency of clinical images and histopathological data is realized, the application utilizes a double-channel convolutional neural network and a lightweight LSTM, applies a physical constraint gate unit to perform channel re-labeling on a double-modal shallow feature map, generates a re-labeled feature map, and after splicing, outputs a caries depth grading interpretation result through a decoding network, so that the accuracy and consistency of caries depth grading interpretation are effectively improved.

The application discloses an emotion recognition method based on online cross-modal knowledge distillation, comprising the following steps: acquiring electroencephalogram and electrocardiogram original signals and windowing and cutting; constructing electroencephalogram and electrocardiogram student models, extracting intermediate features from each modal data segment through an encoder, and obtaining non-normalized prediction output through a classifier; constructing a teacher probability distribution through a joint encoder fusion; introducing adaptive contrast loss to align the cross-modal intermediate features, introducing distillation loss to constrain the prediction probability distribution of each modal to align with the teacher probability distribution; synchronously optimizing new student model parameters through online collaborative training; and performing actual inference prediction based on the student model after training. The application combines double modal signals to make up for the defects of single modal information, excavates the complementarity of modes, realizes dynamic generation of teacher supervision signals and real-time collaborative learning of modes through online distillation, does not increase test calculation overhead, effectively improves the recognition accuracy, model robustness and generalization ability, and has good application prospect.

The application belongs to the technical field of intelligent traffic and driving safety monitoring, and specifically discloses a driver fatigue detection method and system based on multi-modal data fusion. First, a heterogeneous fusion system of facial visual features, bracelet features and vehicle driving behavior features is constructed. Then, the extracted multi-modal features are subjected to individual standardization processing based on individual sober benchmarks. Next, three fatigue detection models are constructed, two of which receive bimodal input and one of which receives multi-modal input. Real-time multi-modal information is collected, and according to the failure or recovery state of the visual mode / bracelet mode, adaptive switching is performed between the three fatigue detection models, and finally the switched model is used for fatigue state detection. The application solves the problems of poor cross-driver generalization performance and insufficient detection robustness in complex environments from the root, and realizes accurate identification and monitoring of the fatigue state of the driver.

The application provides a multi-modal dynamic tactile encryption method, a tactile encryption skin, a system and an application, which comprises the following steps: a dynamic tactile force acts on a tactile encryption epidermis, image data of an initial state of the tactile encryption epidermis and an image data generated in a dynamic tactile action process are collected through a tactile encryption subcutaneous tissue; based on the image data, a position cell tensor, a velocity tensor and a depth tensor generated before and after the encryption epidermis granular layer is subjected to a tactile stimulation are obtained; based on the position cell tensor, the velocity tensor and the depth tensor, a double-modal dynamic tactile feature or a multi-modal dynamic tactile feature based on a dynamic force-velocity-depth coupling is constructed; and the double-modal dynamic tactile feature or the multi-modal dynamic tactile feature is subjected to multi-level standard encryption based on a self-defined initial key to obtain a final dynamic multi-modal tactile process encryption result. The application has high specificity, which includes spatial and temporal features; the encryption process is dynamic, contains the whole process of biological tactile action, and has high reliability.

A computing system is disclosed that includes a processor and memory. The memory stores instructions that, when executed by the processor, cause the processor to perform several acts. The acts include generating a prompt that is to be input to a generative language model. The prompt includes conversational input set forth by a user. The acts further comprise providing the prompt as input to the generative language model, and receiving conversational output from the generative language model, where the generative language model generated the conversational output based upon the prompt. Additionally, the acts comprise receiving an indication that the user has performed an interface mode change action and updating a search engine results page (SERP) to provide information related to the conversational output generated by the generative language model. The acts further comprise presenting the updated SERP to the user on a client computing device.

This invention discloses a multimodal behavior recognition method based on modal credibility scheduling and conflict reconstruction, comprising the following steps: acquiring video and skeletal modal data of behavior samples, completing time alignment and standardization preprocessing to obtain time-synchronized bimodal input tensors; feature encoding the bimodal input tensors respectively to obtain global semantic features and behavior category prediction probability distributions for each modality; calculating the credibility weights of video and skeletal modalities based on the uncertainty of the prediction probability distributions; calculating the comprehensive conflict intensity of the bimodal prediction distributions to generate continuous conflict adjustment factors; modulating the credibility weights using the conflict adjustment factors to complete the adaptive fusion of bimodal probabilities, and outputting the final behavior recognition result. This method solves the technical problems in existing multimodal behavior recognition technologies, such as the lack of explicit quantitative modeling of modal reliability and the lack of active intervention mechanisms for cross-modal prediction conflicts, leading to insufficient recognition accuracy and system stability in complex scenarios.

The application provides a cross-modal dataset construction and automatic label annotation method, which comprises the following steps: obtaining an initial cross-modal dataset containing labeled and unlabeled sample pairs, initializing a pseudo label storage, extracting double-modal features from the unlabeled sample pairs through a modal encoder, outputting single-modal class probability distribution by a classifier, fusing the features to obtain cross-modal joint features, outputting joint class probability distribution by a shared classifier, taking the maximum value of the joint probability as the confidence score, taking the L1 distance between the double-single-modal probabilities as the deviation degree between the modes, screening candidate samples that meet the threshold condition, calculating the weight based on the confidence and the deviation, performing exponential moving average on the joint class probability and the smoothed pseudo label of the previous cycle to generate the weighted smoothed pseudo label of the current cycle and update the storage, constructing a hybrid supervision signal with the real label and the pseudo label of the candidate sample, minimizing the total loss function, and iteratively optimizing the modal encoder and the classifier.

The application discloses a video emotion recognition method based on multi-modal, relates to the technical field of computer vision, deep learning and emotion computing, and comprises the following steps: constructing a micro-motion label system, performing secondary labeling on a CAER data set based on the micro-motion label system, and labeling micro-motion labels; constructing a dual-modal feature extraction network, including a face macro-emotion encoder for extracting macro facial features and a time-series micro-motion feature encoder for extracting time-series micro-motion features; the face macro-emotion encoder outputs a Logits vector for face emotion classification for the face in the video; the time-series micro-motion feature encoder outputs a Logits vector for micro-motion emotion classification for the body in the video; and the dual-modal posterior Logits are fused, and the final emotion category is output based on the fused Logits vector. The application significantly improves the accuracy and robustness of video emotion recognition.

This invention proposes a method and system for determining the classification of tunnel surrounding rock based on a dual-modal fusion model, relating to the field of tunnel engineering technology. Addressing the problems of poor accuracy and applicability in existing surrounding rock classification technologies, this invention collects drilling parameters, tunnel face images, and level labels, and performs preprocessing. Based on the drilling parameters, a multi-dimensional initial feature set is constructed and key drilling parameter feature sets are obtained through screening. Dimensionality reduction processing of the tunnel face images yields key image feature sets, which are then used to determine the fusion feature set. Using the fusion feature set as input and the level labels as output, a feature-level fusion classification model is trained. Using the key drilling parameter feature set and key image feature set as input, combined with the level labels, two single-modal classification models are trained and then co-validated and fused to obtain a decision-level fusion classification model. The tunnel data to be classified is then input into either the feature-level or decision-level fusion classification model to determine the classification result. This invention offers high classification accuracy and applicability.

This invention discloses a method and system for evaluating the appearance quality of tobacco leaves by fusing hyperspectral and digital images. The method includes: acquiring hyperspectral and RGB images of tobacco leaves to obtain raw data; slicing the raw data to obtain a multimodal dataset, including hyperspectral image patches and corresponding RGB image patches; reducing the dimensionality of the hyperspectral image patches; extracting features from the hyperspectral image patches using multi-scale convolution, residual blocks, and channel attention strategies; extracting features from the RGB image patches using feature-level patch embedding, Transformer, and multi-head attention mechanisms; and performing dual-modal feature fusion and quality classification based on the hyperspectral and RGB features. This invention's method and system for evaluating the appearance quality of tobacco leaves by fusing hyperspectral and digital images can evaluate the appearance quality of dual-modal tobacco leaf data based on hyperspectral and digital images, achieving highly accurate prediction.

A missing modal multi-modal learning method based on quality perception prompt modulation. It relates to the technical field of multi-modal learning, aiming at the problem that the existing missing modal multi-modal learning method ignores the reliability difference of modal information, leading to the continuous propagation of low-quality information in the feature extraction and fusion process, affecting the robustness and prediction accuracy of the model, the following scheme is proposed: detecting the missing type of the input sample, generating the missing modal feature and completing the missing modal, combining the task correlation evaluation, decision influence evaluation and intrinsic truth evaluation to obtain the comprehensive quality score, and modulating the prompt vector of the pre-trained visual language encoder according to the comprehensive quality score, then the double modal feature is weighted and fused according to the quality perception, and the task prediction is completed. The present application is suitable for multi-modal learning tasks such as image-text classification, sentiment analysis, visual question answering, medical auxiliary diagnosis and the like under the condition of missing modal.

The present application relates to the technical field of robot navigation, in particular to a puncture navigation interaction system and method based on offline voice and gesture recognition fusion, which completes voice collection and recognition locally through an offline voice recognition module, without network transmission, eliminating the risk of patient privacy leakage and the influence of network delay, and ensuring real-time response in the surgical environment; the gesture recognition module adopts a hands-off collection method, so that the surgical operator can input instructions without touching any physical device, strictly meeting the sterile operation specification; the modal fusion decision unit performs consistency verification and priority scheduling on the dual-modal recognition results, outputs a single effective control instruction, effectively avoids instruction conflicts and misoperations, and significantly improves the interaction fault tolerance; the control unit converts the instruction into a driving signal and controls the puncture navigation robot to perform the corresponding action, realizing non-contact, high-reliability and low-delay human-computer interaction, and comprehensively guaranteeing the safety, accuracy and sterility of the puncture navigation surgery.

The application relates to the technical field of image target detection, and particularly relates to a weakly supervised remote sensing image target detection method based on multi-modal pseudo label guidance and adaptive fusion. Through a cross-modal adaptive relationship fusion module, double-modal features are dynamically fused; and through a cross-modal pseudo supervision label mapping module, RGB and depth pseudo labels generated and aggregated are used, a unified structure loss and a depth double loss are proposed to supervise and train a main branch and an auxiliary branch, so as to solve problems such as insufficient utilization of multi-modal information, large pseudo label noise and insufficient global and local information fusion under weak supervision.

This invention discloses a dance rehabilitation training system integrating EEG and EEG sensors. The system includes an EEG headband and EEG sensors, each with built-in EEG and EEG acquisition modules. The dance rehabilitation training process includes: device wearing and initialization; personalized physiological baseline acquisition; simultaneous acquisition and preprocessing of dual-modal signals; multimodal feature fusion and emotion recognition, obtaining emotional state through dual-model fusion and smoothing; personalized safety threshold decision-making and intervention level determination; multi-channel dance intervention execution; emergency stress handling mechanism; closed-loop iteration and real-time control; and finally, generation of an evaluation report. This invention solves the problems of static intervention parameters, low feedback accuracy, and easy triggering of stress responses in traditional dance training by using dual-modal signal fusion, personalized safety threshold modeling, and closed-loop real-time control, achieving real-time, precise, and safe personalized rehabilitation intervention.

This invention relates to the field of fault diagnosis technology, specifically to a fault diagnosis method and system based on dual-modal GAF coding and cross-modal fusion. The method includes the following steps: acquiring vibration signals; generating GASF and GADF images with complementary physical features using GASF and FFT-based GADF methods respectively; extracting deep feature sequences using two independent feature extraction networks, and then enhancing and fusing them. During fusion, a monotonically decreasing constraint is used to progressively evolve the model from shallow global exploration to deep local focus; outputting fault diagnosis results based on the fused features, and simultaneously outputting three dimensions of visual evidence: a Grad-CAM heatmap of the coding layer and its correspondence with the signal's physical features; a heatmap of attention weights in the feature layer and CKA values; and the distribution of modal fusion weights in the decision layer. This invention, through multi-level interpretability analysis, provides complete and traceable evidence for the diagnostic results, making it applicable to safety-critical scenarios such as aero-engines and wind turbine gearboxes.

The application discloses a synthesis method and application of a bimodal probe based on Cu 2+ imaging, and a structure of the bimodal probe based on Cu 2+ imaging is as follows: the bimodal probe based on Cu 2+ has high sensitivity and selectivity, and can interact with Cu 2+ under the action of Cu 2+ , and the photoacoustic signal is enhanced by 3-5 times. After the interaction with Cu 2+ , the hydrophobicity of the bimodal probe is enhanced, the interaction between the bimodal probe and proteins is further enhanced, the rotation-related time of the bimodal probe is changed, and the magnetic resonance signal is enhanced by 1-2 times. Therefore, the probe can be used as a developing agent of magnetic resonance imaging or photoacoustic imaging, and is applied to monitoring fluctuation of Cu 2+ in a living body, so as to provide a solution for in-vivo detection of metal ions.

This invention relates to the field of signal processing technology, and provides a time-frequency dual-modal alignment method and apparatus for multivariable time-series signals. It utilizes a cross-scale time-series feature extraction network to extract discriminative features containing both time and frequency information from the multivariable time-series signal using cross-variable attention and cross-time attention. Then, it employs a codebook sharing time-frequency feature prototypes across domains for dual-modal vector quantization alignment based on spectral consistency, thereby eliminating domain offset through explicit frequency domain constraints. Furthermore, it uses an adaptive pseudo-label optimization strategy based on channel mutual information to suppress noise channel interference and improve the confidence of the target domain pseudo-label. Finally, under an end-to-end unified framework, it jointly optimizes the global alignment loss, local alignment loss, mutual information weighted maximization of the confusion matrix loss, and source domain cross-entropy loss to determine the target domain pseudo-label, deeply adapting to the physical characteristics of the time-series signal and simultaneously solving the problems of multivariable coupling, long-term time dependence, and time-frequency feature collaborative alignment.

The application provides a concrete crack analysis method and system based on multi-modal image fusion, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a visible light image and a structured light depth image of the same concrete structure surface; performing spatial alignment on the visible light image and the structured light depth image to obtain a dual-modal aligned image set; extracting a texture high-frequency tensor and a surface normal gradient tensor of the dual-modal aligned image set; and adaptively fusing the texture high-frequency tensor and the surface normal gradient tensor to obtain a fused feature tensor. By fusing the visible light and structured light depth features, the application constructs a triangular reference net to correct the surface distortion, realizes crack geometric correction and continuous skeleton extraction, and accurately quantifies the opening width, thereby solving the problems of projection distortion error and skeleton fracture.

This invention belongs to the field of environmental perception and target detection technology for unmanned surface vessels. Specifically, it discloses a multi-band fusion target detection method for environmental coupling knowledge distillation on unmanned surface vessels. This method simultaneously collects multi-band perception data, hull motion data, and marine environment data, and sequentially calculates inter-frame registration recursive compensation coefficients, dual-modal feature enhancement weights, and dynamic distillation weights of multi-teacher networks to complete knowledge distillation, multi-modal feature fusion, and target detection using a lightweight student network. This invention achieves dynamic parameter linkage throughout the detection process, improves the stability of target detection in complex sea surface scenarios, and adapts to the deployment requirements of unmanned surface vessel edge computing platforms.

This invention discloses an automatic identification method and system for vessels emitting black smoke, aiming to solve the problem of rapid and accurate identification of such vessels. The method includes: recording initial parameters of a zoomable camera and acquiring channel video; capturing black smoke events using a multi-feature fusion black smoke detection algorithm; adjusting the camera via PID control and recording the vessel's 30-second navigation trajectory and heading; and filtering and matching identity information based on an "AIS + hull coding" dual-modal system, combined with vessel quantity and heading, supplemented by trajectory similarity and size error constraints to improve accuracy. The system includes modules such as a zoomable camera, an AI computing power unit, and a controller, integrating black smoke detection, trajectory tracking, identity matching, and data uploading. This invention achieves automatic identification of vessels emitting black smoke with strong real-time performance and high accuracy, enabling rapid evidence collection and providing technical support for channel vessel supervision.

This invention discloses a remote sensing rotating target detection method and system based on cross-modal image fusion, belonging to the field of remote sensing target detection technology. The method first acquires pixel-level spatially registered visible light and synthetic aperture radar images, which are then preprocessed and separated into dual-modal feature streams. These streams are input into a dual-branch backbone network to extract multi-scale features. Cross-modal joint dynamic fusion is used for low and medium scales, while cross-modal hybrid expert fusion is used for high semantic scales, resulting in three-scale fused features. After enhancement by multi-scale aggregation units and fusion with a neck network, these features are input into a rotating target detection head, outputting a rotated bounding box and category prediction. The detection result is obtained by combining rotation non-maximum suppression. A joint loss function is constructed for end-to-end optimization. This invention achieves accurate detection of targets in all weather and all attitudes in remote sensing scenarios, effectively improving detection robustness and positioning accuracy under complex backgrounds and multiple scattering conditions.

This invention discloses a continuous control method and device based on adaptive deep reinforcement learning, belonging to the field of intelligent control and robotics. The invention proposes a dynamic policy optimization framework integrating five innovative technologies: a dual-modal target network that fuses the outputs of the primary and secondary target networks through loss-driven dynamic weight allocation to suppress target value oscillations; an adaptive soft update algorithm that dynamically adjusts the target network update rate based on the reward change rate; a three-level dynamic experience storage architecture that combines value bias-driven priority sampling to improve the utilization rate of key samples; multi-scale feature calibration technology that performs differentiated normalization based on the differences in feature distribution at each level of the network; and an environment-aware exploration policy generator that drives adaptive exploration noise by real-time sensing of system dynamic parameters. This invention achieves an average policy performance improvement of 35%, a convergence speed improvement of over 40%, a control accuracy improvement of 25%, and a sample efficiency improvement of 2.1 times in 11 complex physical control tasks.

This invention discloses a dual-modal fusion wavelet-enhanced six-image hyperspectral classification method and system. The method includes: acquiring spectral and RGB image data of the six images; extracting features from the spectral and RGB image data respectively; enhancing the feature-extracted data using a learnable wavelet enhancement module; flattening and stitching the enhanced data; fusing the stitched feature sequences to obtain a fused feature sequence; embedding location information and category labels into the fused feature sequence, and feeding it into a Transformer encoder for global context modeling and classification. This invention's dual-modal fusion wavelet-enhanced six-image hyperspectral classification method and system achieves the fusion and utilization of hyperspectral imaging, near-infrared spectroscopy, and digital image features through feature-level and data-level fusion, fully utilizing the spatial-spectral information of multiple data sources, leveraging the advantages of various data, and effectively overcoming the limitations of single-modal methods.

This invention relates to the fields of medical device technology and signal processing algorithm technology. Specifically, it relates to an in vitro cardiac contractility dynamic enhancement system and method based on R-wave peak dual-modal sensing. The problem addressed by this invention is the technical issues of poor safety, high oxygen consumption, insufficient precision, and lack of automated closed-loop control in isolated cardiac mechanical perfusion methods. To solve these problems, this invention provides an in vitro cardiac contractility dynamic enhancement method based on R-wave peak dual-modal sensing, comprising: obtaining an initial threshold and an initial R-wave template library based on real-time cardiac physiological signals within an initial target time period; obtaining the true R-wave peak based on waveform verification of candidate R-wave peaks within the target time period and the initial threshold; calculating the stimulation delay based on species-preset parameters and real-time cardiac physiological signals; and selecting the stimulation mode and adjusting the energy parameters based on changes in the real-time cardiac physiological signals.