1737 results about "Multi modal data" patented technology

This disclosure provides a method for assessing the health status of bearings. The method includes: receiving multimodal data and prompting information about the bearing; performing feature extraction on the multimodal data using a first set of agents to generate a feature set about the multimodal data; performing a health assessment on the bearing using a second set of agents based on the feature set and prompting information about the multimodal data to generate multiple health assessment results about the bearing; and combining the multiple health assessment results using a third set of agents to generate a health assessment report about the bearing. According to this bearing health status assessment method, users can input multimodal data to perform bearing health status assessments, thereby improving the flexibility of the bearing health status assessment method. Furthermore, the bearing health status assessment method of this disclosure, by leveraging the reasoning capabilities of a large language model, can better understand user questions, thereby providing better maintenance suggestions and other outputs.

This application belongs to the field of autonomous driving technology, specifically relating to a multimodal data processing method and apparatus for end-to-end autonomous driving. The method includes: extracting multi-scale features from camera images and LiDAR bird's-eye view images through multiple feature extraction stages of an encoder; performing a feature fusion operation on the image features and LiDAR features of each feature extraction stage after each stage; after the final stage, segmenting at least a first output stream for trajectory prediction from the split LiDAR features; performing a feature enhancement operation on the first output stream and vehicle state features to generate enhanced features; and decoding the enhanced features to generate a sequence of future trajectory points for the vehicle. This application significantly improves inference efficiency and enhances environmental robustness while maintaining perception depth.

This invention provides a method and system for macrophage morphology recognition that integrates multimodal data. The method includes acquiring a time-lapse imaging sequence of live macrophage cells; segmenting and tracking individual cells using a probabilistic model of cell contour evolution and intracellular texture flow to obtain motion trajectories and continuous morphological contour sequences; acquiring behavioral features based on the motion trajectory; obtaining morphological features based on the morphological contour sequences; calculating local field influence features based on the behavioral and morphological features of neighboring cells within a neighborhood search radius; constructing a cell interaction graph structure based on the three types of features combined with intercellular Euclidean distance, motion direction correlation, and morphological features of cells at both ends; inputting the cell interaction graph into a trained graph attention network; and determining whether each macrophage is of subtype M1 or M2 based on the output.

The application discloses a kind of cattle contactless multi-modal intelligent monitoring early warning device and monitoring early warning method.The device includes the passage of being arranged in the side of cowshed, mobile acquisition mechanism, multi-modal acquisition unit, edge data processing unit, multi-modal data fusion engine, health assessment model, early warning module and remote management platform.Mobile acquisition mechanism can slide along the passage, rotatingly connected installation box below sliding seat, end push rod can make installation box deflection towards the direction of cattle travel, realize the rapid acquisition when normal traffic;When judging cattle exception, sliding seat moves to the top of cattle and follows movement, carries out whole course close synchronous acquisition.The application fuses color image, depth image, called audio and environmental parameters, combines Transformer model to evaluate health score and risk level, realizes contactless, automated monitoring, avoids cattle stress, improves early abnormal identification accuracy.

The present application relates to the field of natural resource investigation and geographic information processing, and discloses a multi-modal data fusion and ground object boundary identification processing method for field verification; current phase remote sensing images of a region to be verified, reference vector map patches, historical land attribute data, positioning trajectory data of a field verification terminal and a field photo collection correlation field template are acquired to construct a patch-level multi-modal basic data set; candidate ground object segmentation results and change area results are extracted to determine suspected problem patches; a boundary uncertainty band is generated for the boundary of the suspected problem patches and the to-be-verified boundary segments are divided; shooting guidance is triggered during the field verification process, field photos are collected, coarse registration and fine registration with local remote sensing images are completed; edge features of the field photos are back-projected to a map coordinate system to form a field confirmation boundary point set, the boundary of the suspected problem patches is corrected, verification confirmation patches are obtained, and patch-level verification evidence data is formed.

This invention relates to the field of intelligent technology for identifying hidden dangers on reservoir and river / lake embankments, specifically to a method for intelligent identification and risk assessment of embankment hidden dangers based on multimodal perception using a walking robot. The method involves a quadruped robot equipped with multiple sensor modules autonomously inspecting reservoir and river / lake embankment areas to collect multimodal data from the embankment surface. This multimodal data is then transmitted to a data processing center. Based on this data, the types and locations of hidden dangers on the embankment surface are identified. These hidden danger types include cracks, seepage or piping, settlement, and structural deformation. The types and locations of these hidden dangers are quantitatively analyzed to obtain quantitative results. Based on these results, a risk assessment report, a visual map, and an early warning are generated. This invention improves the spatial coverage accuracy and temporal synchronization of data acquisition, and possesses higher sensitivity and complementarity in identifying hidden danger types such as embankment cracks, seepage, and deformation, providing a perceptual foundation for subsequent analysis.

This invention discloses a method for accurate identification of the moment of an accident in a new energy vehicle based on multimodal data fusion. The method acquires vehicle motion sensor data, dynamic parameters, and visual image data, and then performs the following parallel processing: The physical layer processes the data to obtain motion features and compares them with preset trigger conditions; when a suspected collision is detected, it outputs the physical layer trigger time and acceleration amplitude; the dynamic layer processes the data based on the deviation between the vehicle's dynamic model and its actual motion state, outputting a vehicle trajectory status label; the visual layer processes the data to detect targets and analyze their motion state; when visual evidence of a collision is detected, it outputs the visual layer trigger time and the type of collision object; the fusion layer performs spatiotemporal alignment of the three layers' outputs and makes a comprehensive judgment, outputting the fused accurate collision trigger time and accident scene classification information. This invention effectively filters false alarms and improves the detection rate of low-speed accidents through multimodal fusion, achieving accurate identification of the moment of an accident and scene reconstruction.

The present application relates to a kind of intelligent interview evaluation and feedback system based on multi-modal data fusion, specifically relates to data processing field, by meta-learning mechanism dynamic perception interview scene and generate initial fusion weight, subsequently utilize the complex interaction relationship between modalities modeled by graph neural network to carry out fine-grained correction to weight, so as to significantly improve the accuracy and scene adaptability of multi-modal evaluation, further introduce reinforcement learning, link evaluation decision and long-term performance of talents, continuously optimize weight generation strategy, ensure that evaluation standard and business goal are aligned, finally, through closed-loop iteration mechanism, make the whole scheme can be updated automatically according to new data and performance feedback, continuous evolution, with strong self-optimizing ability and long-term robustness, realize the fundamental change from static rule to dynamic intelligent decision.

This invention proposes a method and system for judging tunnel defects based on large-scale model identification of tunnel point cloud data, belonging to the field of tunnel engineering inspection. The method includes: acquiring the original 3D point cloud data of the tunnel and preprocessing it into a standardized point cloud dataset; segmenting the standardized point cloud dataset and constructing image multimodal data pairs; inputting the image multimodal data pairs and dedicated prompting instructions into the constructed defect detection large-scale model to obtain preliminary identification and evaluation results; mapping the results back to the original 3D point cloud coordinate system through a view projection matrix to construct a 3D defect point cloud; constructing a temporal-spatial knowledge graph based on the 3D defect point cloud, generating early warning information when preset risk conditions are met; and generating tunnel defect detection results based on the 3D defect point cloud and early warning information. This invention achieves full-process automation and intelligence in tunnel defect detection, significantly improving detection efficiency and accuracy.

The application discloses a medical data processing method and product based on multi-stage transfer learning and multi-modal data collaborative fusion, relates to the technical field of medical data processing and artificial intelligence, and adopts MedicalNet medical special pre-training weights to initialize a classification model backbone network; a three-stage progressive fine-tuning framework is constructed on the basis, field adaptive coarse classification fine-tuning and target task fine classification are performed, and clinical structured data is introduced in the third stage; after high-dimensional image features are reduced in dimension and low-dimensional clinical features are increased in dimension through an adaptive multi-branch multi-level MLP architecture, mid-term deep fusion is performed, the application can effectively mine the complex relationships such as complementation, correlation and cooperation of multi-modal heterogeneous data, improve the lung disease classification precision and model generalization capability, and significantly inhibit the small sample overfitting phenomenon.

This invention discloses a multimodal data security sharing method based on information security. Addressing the problems raised in previous technologies, such as data becoming ineffective when it leaves a controlled domain, and the inability of subsequent dynamic control attempts to be implemented due to the lack of a trusted, data-co-existing execution endpoint, and the inability of security policies to learn and adjust based on real-time feedback during data usage, this invention proposes the following solution: policy-driven multimodal data preprocessing and labeling. For the raw multimodal data to be shared, a sensitive information identification model corresponding to the modality is invoked for analysis. This invention solves the industry problem of data loss of control after leaving the domain, promotes real-time collaboration and evolution of security capabilities, drives the continuous evolution of policies and identification models, and constructs a unified framework and modality-adaptive cross-modal data security management solution. It transforms complex security operation and maintenance problems into simple policy definition problems, significantly reducing the technical threshold and operational costs of secure sharing.

This invention discloses a mixed reality data processing and interactive response method, apparatus, and system. The method includes: acquiring corresponding spatial feature points in VR and AR environments; optimizing the solution of rigid body transformation matrices to achieve sub-millimeter-level spatial alignment; continuously monitoring alignment errors during virtual-real fusion rendering, triggering a repositioning process when errors exceed a threshold; evaluating user operations through multimodal data fusion and generating real-time AR correction guidance; and dynamically adjusting rendering parameters and resource preloading based on visual attention focus to ensure end-to-end latency and interactive feedback latency are both below set thresholds. The apparatus includes modules for feature point acquisition, spatial alignment calculation, error monitoring and repositioning control, multimodal data interface, and real-time rendering control. The system includes a server and a user interaction terminal, supporting intelligent training and adaptive interactive optimization for virtual-real fusion. This application addresses problems such as low virtual-real spatial alignment accuracy, high interactive latency, and insufficient personalized adaptation.

This application discloses an open-world 3D target detection method and apparatus, comprising: acquiring multimodal data of a target scene; performing 3D target detection on 3D point cloud data to obtain 3D candidate targets; inputting the geometric features of each 3D candidate target into an out-of-distribution target classifier to determine whether the 3D candidate target belongs to a known category set and to identify unknown targets; projecting the 3D candidate targets corresponding to the unknown targets onto 2D image data, obtaining the 2D image region, and inputting it into a visual language model, using natural language prompts to guide the visual language model to output the semantic category name of the unknown targets; fusing the semantic category name and the spatial location information of the 3D candidate targets to generate open-world 3D detection results. This application can effectively identify unknown 3D targets in an open world and generate open-world 3D detection results that combine spatial positioning and semantic description.

This disclosure provides a multi-slice data processing method, apparatus, electronic device, and storage medium. The multi-slice data processing method includes: acquiring multimodal data of multiple target slices; clustering the multimodal data based on an analysis unit for each target slice to determine multimodal labeled data, whereby the multimodal labeled data describes the association between the target slices and target label data; extracting features from the multimodal data using an initial feature extractor based on a preset neural network model to obtain a multimodal feature matrix; performing label prediction on the multimodal feature matrix using a label prediction unit based on the preset neural network model to obtain predicted label data; training the preset neural network model based on the target label data and the predicted label data; and obtaining a target feature extractor for multi-slice feature extraction of multiple target slices based on the trained preset neural network model. The embodiments of this application provide a method suitable for multi-slice data processing.

This invention discloses a method for autonomously constructing an evolvable knowledge graph by integrating a multimodal large-scale model, relating to the fields of artificial intelligence and knowledge graph technology. The method includes: collecting raw multimodal data from text, images, audio, and structured tables; using a pre-trained multimodal understanding large-scale model to perform unified semantic encoding on the raw multimodal data, generating modality-independent deep feature vectors to form an initial multimodal feature pool; performing cross-modal clustering analysis on the vectors in the feature pool, grouping semantically similar vectors into the same feature cluster, with each feature cluster defined as a candidate knowledge concept node; constructing an initial concept relationship network based on the co-occurrence relationship and feature similarity between candidate concept nodes; iteratively optimizing and evolving the network according to preset graph quality evaluation indicators, outputting the final evolvable knowledge graph. This invention achieves the autonomous and unified construction and evolution of a knowledge graph from multimodal data.

A foundation model for performing molecular-level tasks by learning multimodal data in the form of one-dimensional text and two-dimensional graphs, and a system therefor, according to an embodiment of the present invention, enable various molecular-unit tasks such as chemical reaction prediction, molecular attribute prediction, and natural language description generation to be effectively processed through a single foundation model. In addition, it is possible to increase prediction accuracy of the model by maximizing utilization of two-dimensional molecular graph information, and automatically generate, on the basis of statistical sparsity, high-quality descriptive text that emphasizes core and distinctive features of each molecule.

The application discloses a kind of multi-task wisdom robots based on multi-modal data, comprising: personalized interaction module obtains scene image and obtains face image by face recognition, and judge whether it is student, if yes, the identity information of student is obtained, and corresponding historical data is obtained;Face expression extraction is carried out based on face image and generates face label and conversation theme in combination with historical behavior data;Conversation content is generated based on face label, conversation theme and historical data, whether student is in speaking state is judged, if yes, conversation is stopped, and new conversation content is generated;Multi-task module pre-set task mode, regional function and schedule, obtain current time and current location, obtain current task mode and collect student dynamic data, generate evaluation report based on multi-task data, at least including student dynamic data and historical data.The efficient and personalized conversation and multi-task service with student are realized by the application.

The present application relates to the technical field of metrological calibration, and a PH automatic titration metrological calibration method and system based on visual perception, comprising: confirming a titration metrological calibration environment based on a titration metrological calibration instruction, drying a glass instrument to be calibrated based on a pre-constructed drying device, collecting internal and external contour features of the dried glass instrument based on a visual sensing unit, performing three-dimensional space modeling on an instrument contour multi-modal data set based on an intelligent recognition unit, identifying instrument specification parameters based on an instrument three-dimensional model, calculating an actual liquid volume based on a titration end point data set, performing deviation analysis based on the actual liquid volume and the instrument specification parameters, and automatically sorting the glass instrument to be calibrated based on a metrological calibration evaluation report. The present application can improve the accuracy and efficiency of PH automatic titration metrological calibration.

This invention provides a driver state monitoring and intelligent interaction method based on multimodal data, belonging to the field of intelligent driving assistance technology. The method includes: simultaneously acquiring the driver's electroencephalogram (EEG) signals, at least one other physiological signal, and vehicle state data to form a multimodal data stream; preprocessing and extracting features from the multimodal data stream to obtain a multimodal feature vector for state recognition; inputting the multimodal feature vector into a preset multimodal state recognition model to generate a recognition result characterizing the driver's current state; and triggering corresponding intelligent interactive feedback or vehicle control commands based on the recognition result. This invention, through multimodal signal fusion, comprehensively assesses the driver's state from three dimensions: neurophysiology, behavioral performance, and vehicle control, significantly improving the accuracy and reliability of state recognition.

The application discloses a multi-modal feature collaborative generation analysis method and system for tumor survival prediction, and relates to the technical field of computer vision. The method comprises the following steps: acquiring multi-modal data and preprocessing to obtain a training data set; learning a first feature mapping relationship between multiple modes based on complete mode samples, and training a generator based on missing mode samples and the first feature mapping relationship to obtain a target generator, which generates virtual coding features of the missing mode; acquiring partial mode medical data of a target object, inputting the partial mode medical data of the target object into the target generator to generate virtual missing mode coding features of the target object; extracting at least one other mode coding feature from the partial mode medical data, fusing the virtual missing mode coding features and the at least one other mode coding feature, and determining a survival prediction result of the target object according to the fused features. The application improves the accuracy and interpretability of tumor survival prediction.

This invention discloses a method and system for digital evaluation of rural tourism based on multimodal data. The method includes: collecting multimodal data from rural tourism scenarios to construct a multimodal dataset; encoding the features of each modality to obtain corresponding modal feature vectors; performing cross-modal fusion processing on the modal feature vectors to generate fused feature representations; applying semantic alignment constraints to the modal feature vectors; constructing a feature-evaluation mapping model and a weight determination model; jointly training the feature-evaluation mapping model and the weight determination model by jointly optimizing the objective function; based on the trained model, performing inference on the real-time collected multimodal data to obtain and output the comprehensive digital evaluation result of rural tourism; and simultaneously updating the model online based on newly accessed multimodal data. This invention can significantly improve the accuracy of digital evaluation of rural tourism.

The application discloses a multi-modal perception robot elevator control method and related equipment, the method comprises the following steps: obtaining an elevator task and current radar data, performing path planning according to the current radar data and preset floor map data to obtain an elevator path; controlling the robot to go to a floor elevator position according to the elevator path, performing multi-modal scanning on the floor elevator position, obtaining multi-modal environment data and performing fusion analysis to determine elevator state information; performing decision analysis according to the elevator state information and the obtained elevator task to generate robot control instructions, and controlling the robot to execute the elevator task according to the generated robot control instructions; obtaining multi-modal data through multi-modal scanning to perform fusion analysis, determine the elevator state information, and generate control instructions based on the elevator state information, and control the robot to execute the elevator task. The application embodiment can improve control reliability and reduce implementation cost. The application can be widely applied in the field of robot control technology.

This invention relates to the field of coal quality testing technology, specifically to an online method and system for detecting coal quality entering the furnace; it includes: a multimodal data acquisition step: acquiring microwave scattering parameters and surface element characteristic spectra using a microwave tomography sensor array and a surface spectrometer, respectively; a physical field inversion and modeling step: reconstructing the complex permittivity distribution map based on the scattering parameters to establish a three-dimensional physical distribution model including density and moisture; a field-spectrum coupling deduction step: using the characteristic spectra as boundary constraints, and combining them with the complex permittivity distribution map to deduce the internal chemical element distribution; and a volume-weighted quantization step: performing volume-weighted integration on the three-dimensional physical model and chemical distribution to output comprehensive coal quality parameters; this invention eliminates the detection blind spots of single technologies and significantly improves the representativeness and safety of full-section detection of coal entering the furnace.

The application discloses an abnormal driving behavior recognition method and system based on multi-modal data fusion, and particularly relates to the technical field of multi-vehicle cooperative driving safety control, and is used for solving the problem that the existing cooperative driving system cannot convert the abnormal behavior risk of a driver into a cooperative control instruction at the vehicle fleet level; the abnormal state of the driver is recognized by collecting and fusing multi-modal sensor data of the vehicle; when the vehicle is a lead vehicle of a vehicle fleet, the comprehensive risk of the abnormal state to the safety of the vehicle fleet is evaluated based on the abnormal state through probabilistic evolution simulation; the expected effect of different vehicle fleet control strategies is simulated according to the risk evaluation result, and the best strategy is selected; vehicle fleet risk information containing the strategy is generated and sent to all following vehicles through vehicle-to-vehicle communication; the whole process from single-vehicle driver state monitoring to vehicle fleet level cooperative risk prevention and control is realized, and the overall safety of the vehicle fleet system is improved.

The application relates to the technical field of computer vision, and discloses an integrated identification and positioning method and system for farmland weed plant protection operation, which comprises the following steps: synchronously triggering a multi-modal sensor to form a time-aligned multi-modal data group; identifying image data to output crop pixel coordinates and weed pixel coordinates; searching a carrier attitude from pose data according to a time stamp carried by the image; predicting the pose of the carrier after a delay according to a full-link processing delay to obtain predictive crop operation coordinates and predictive weed operation coordinates; transmitting an instruction carrying the predictive weed operation coordinates to an execution mechanism through a real-time bus; and completing a target action in an execution window triggered by a unified clock source timing signal according to the predictive weed operation coordinates. The application can eliminate the influence of random delay jitter on positioning accuracy, provides a crop avoidance reference for target operation, and realizes the dual goals of accurate weed positioning and effective crop protection.

The application discloses a kind of collaborative robot system and service process for active power grid marketing, belong to artificial intelligence and electric power marketing service technical field.System includes: robot perception interaction end, cloud intelligent analysis hub and dynamic customer memory bank, robot perception interaction end deploys lightweight vision model, real-time identification customer emotion, body state and explicit feature, and initiatively trigger service;Cloud intelligent analysis hub uses power grid knowledge enhanced large language model, carries out depth demand analysis and individualized strategy generation to multi-modal data;Dynamic customer memory bank stores customer historical interaction record and portrait information in the form of knowledge graph, realizes the continuity and memory of service.The application is through edge small model+cloud big model collaborative architecture, general AI capability and power grid business knowledge are deeply fused, so that robot can actively identify customer demand, provide marketing service, improve customer experience and marketing conversion rate of electric power business hall.

The application relates to the technical field of drug efficacy prediction, in particular to a multi-modal data prediction model for diabetes gene therapy response, which comprises an unmedicated data collection module, a medicated data collection module and a data prediction module.The unmedicated data collection module collects unmedicated data samples of a user; the medicated data collection module collects data samples of the user after taking medicine; the data samples comprise explicit data and implicit data; the explicit data is body information and medicine information related to blood glucose change; and the implicit data is blood glucose values corresponding to the explicit data; the data prediction module inputs fused features into a convolutional network model to generate drug efficacy prediction values.The application generates a prediction residual corresponding to the explicit data after taking medicine through a blood glucose value prediction module, identifies blood glucose fluctuation characteristics caused by non-drug factors by using the prediction residual, separates the net influence of medicine on blood glucose, and thus improves the accuracy of drug efficacy prediction.

The invention discloses an FPSO (Floating Production Storage and Offloading) intelligent directional spraying method based on multi-sensor fusion, which is suitable for fire prevention and control of an FPSO upper module. The method comprises the following steps: firstly, dividing protection sub-areas, arranging infrared, ultraviolet, visible light and attitude sensors, matching with mixed nozzles and networking; establishing a module coordinate system, and completing sensor calibration and multi-modal data registration; and a multi-task CNN fire identification model is constructed, and fire source positioning, type discrimination and spreading prediction are realized. When the system operates, image stabilization is performed through hull attitude compensation, fusion feature input models are extracted, and personalized fire extinguishing strategies are generated; and generating a spraying instruction through space calculation and the like, driving the nozzle to spray in a directional and constant-flow manner, and dynamically optimizing the strategy based on real-time feedback until the fire is extinguished. Point-to-point fire extinguishing is achieved, water stain loss and fire fighting water waste are reduced, and the accuracy and efficiency of fire fighting in the marine environment are improved.

The application discloses a kind of children's rare lung disease multi-modal intelligent screening method, system and equipment, belong to medical artificial intelligence technical field.The method includes: obtaining the medical image data, clinical text data and corresponding visit time information generated by the children at multiple time points, and the multi-modal data is standardized;Based on the multi-modal time series data after processing, construct cross-modal time series logic diagram, the cross-modal time series logic diagram is used to represent the association between different modal data and its evolution characteristics with time change;The cross-modal time series logic diagram is input into the children's rare lung disease screening model for analysis, and the risk prediction result of suffering from rare lung disease is output.The screening model includes multi-modal feature extraction module, cross-modal feature fusion module and global comprehensive prediction module, by the joint modeling of image feature, text feature and time series information, realize the intelligent screening of children's rare lung disease.The application can overcome the problem of multi-modal information fragmentation and insufficient use of time series association in the prior art, and is suitable for the auxiliary screening and risk assessment scene of children's rare lung disease.

The present application relates to the technical field of device control, and in particular to a feed quality real-time regulation method and system based on multi-modal data, which mainly includes constructing a feed quality evaluation model, determining whether the current feed is normal based on a quality index, outputting a device regulation index based on the frequency and the deviation rate of the data corresponding to the current target device through a device regulation model, sorting the device regulation index, and outputting the regulation. The above method gives a relatively objective sorting result, and the control system or the staff can adjust the parameters of the corresponding device according to the current sorting result, can complete the regulation in the shortest and most reasonable time, realize the continuous production of the production line, improve the detection efficiency and realize the relatively effective regulation, and at the same time, avoid the traditional static regulation mode, and adjust the current feed production in real time.