920 results about "Multimodal data" patented technology

This disclosure provides a method for assessing the health status of bearings. The method includes: receiving multimodal data and prompting information about the bearing; performing feature extraction on the multimodal data using a first set of agents to generate a feature set about the multimodal data; performing a health assessment on the bearing using a second set of agents based on the feature set and prompting information about the multimodal data to generate multiple health assessment results about the bearing; and combining the multiple health assessment results using a third set of agents to generate a health assessment report about the bearing. According to this bearing health status assessment method, users can input multimodal data to perform bearing health status assessments, thereby improving the flexibility of the bearing health status assessment method. Furthermore, the bearing health status assessment method of this disclosure, by leveraging the reasoning capabilities of a large language model, can better understand user questions, thereby providing better maintenance suggestions and other outputs.

This invention provides a method and system for online monitoring and quality assessment of additive manufacturing processes based on multi-sensor fusion, relating to the technical field of additive manufacturing. The method includes: constructing a multimodal spatiotemporal dataset; extracting features from each physical modality data to obtain an independent feature representation set; inputting the independent feature representation set into a preset dynamic cross-modal attention network to calculate the mutual information correlation matrix and perform feature weighted reconstruction to obtain a deep fusion feature vector; inputting the deep fusion feature vector into a preset quality mapping model to output quality assessment index values ​​representing defect probability and defect classification; constructing a three-dimensional quality digital twin model; and outputting online monitoring and early warning commands and global quality assessment results based on the three-dimensional quality digital twin model. This invention completely eliminates the problem of spatiotemporal registration misalignment of multimodal data, achieving high-precision deep feature fusion and voxel-level panoramic three-dimensional quality space mapping and tracing.

This application belongs to the field of autonomous driving technology, specifically relating to a multimodal data processing method and apparatus for end-to-end autonomous driving. The method includes: extracting multi-scale features from camera images and LiDAR bird's-eye view images through multiple feature extraction stages of an encoder; performing a feature fusion operation on the image features and LiDAR features of each feature extraction stage after each stage; after the final stage, segmenting at least a first output stream for trajectory prediction from the split LiDAR features; performing a feature enhancement operation on the first output stream and vehicle state features to generate enhanced features; and decoding the enhanced features to generate a sequence of future trajectory points for the vehicle. This application significantly improves inference efficiency and enhances environmental robustness while maintaining perception depth.

This invention relates to the field of wetland ecological regulation technology, specifically a method for optimizing the ecological restoration of small wetlands based on multimodal data. The method includes: collecting multimodal monitoring information such as remote sensing image sequences, time-series data of physical and chemical parameters, and ecological sample data of the target area; constructing a dynamic three-dimensional ecological field; assimilating and fusing multi-source data using a spatial grid as a basis to form a comprehensive ecological state attribute value with timestamps; utilizing an improved ecological cellular automata algorithm, combined with an ecological restoration intervention library and the attribute differences of adjacent grids, completing the synchronous evolution calculation of the ecological state of the entire grid, and generating a spatialized ecological evolution field; selecting spatially connected stable grid clusters and implementing connectivity enhancement and structural optimization; and outputting restoration layout, implementation timeline, and engineering quantity configuration content that matches site conditions. This method achieves spatiotemporal integration of multi-source heterogeneous ecological data, refines the accuracy of regional ecological evolution prediction, and enhances the ability to regulate the spatial structure of wetlands.

This invention discloses a sleep stage identification method, comprising: a multimodal data acquisition step; a preprocessing step, which normalizes the acquired physiological signals and segments them according to a set duration to obtain several original signal segments; a Williams RGB compression step, which maps the original signals to an RGB image to obtain a Williams RGB energy map; a feature extraction step, which inputs the Williams RGB energy map into a neural network model for feature extraction; and a comparison and classification step, which compares the feature values ​​with RGB standard thresholds to determine the corresponding sleep stage. This sleep stage identification method avoids directly processing the complex frequency domain features of physiological signals, thus reducing computational complexity. Three physiological signals within a certain time period only need to be represented by a single pixel in the image, achieving physiological signal compression and dimensionality reduction, further reducing computational load. This invention also effectively utilizes medical knowledge graphs to provide interpretable evidence for the sleep stage identification decision-making process.

This invention provides a method and system for macrophage morphology recognition that integrates multimodal data. The method includes acquiring a time-lapse imaging sequence of live macrophage cells; segmenting and tracking individual cells using a probabilistic model of cell contour evolution and intracellular texture flow to obtain motion trajectories and continuous morphological contour sequences; acquiring behavioral features based on the motion trajectory; obtaining morphological features based on the morphological contour sequences; calculating local field influence features based on the behavioral and morphological features of neighboring cells within a neighborhood search radius; constructing a cell interaction graph structure based on the three types of features combined with intercellular Euclidean distance, motion direction correlation, and morphological features of cells at both ends; inputting the cell interaction graph into a trained graph attention network; and determining whether each macrophage is of subtype M1 or M2 based on the output.

This invention relates to the field of data processing and discloses a method, system, and electronic device for generating multimodal data from multiple dimensions. The method includes: acquiring raw data from at least two dimensions; integrating and synchronizing the raw data from each dimension; annotating the data characteristics of different dimensions using corresponding analysis methods; extracting semantic information and structured features of each dimension to obtain annotation results for each dimension; analyzing each annotation result using a corresponding analysis model to obtain modal analysis results; assigning appropriate weights to each modal analysis result based on the credibility, historical accuracy, and importance of the raw data; fusing the modal analysis results using a weighted fusion method to generate multimodal data; and generating final multimodal data after iteration. This application achieves the technical effects of multimodal data alignment, annotation, fusion, and generation of new data.

This invention discloses a multifunctional home robot, comprising: a robot body; a central control system disposed on the robot body, the central control system being configured to perform multimodal data fusion and intelligent decision-making; a multimodal sensing unit electrically connected to the central control system for collecting user physiological data, behavioral data, and environmental data; and a life-assistance execution unit connected to the central control system for performing various operational tasks in the home environment according to instructions from the central control system. This multifunctional home robot solves the problem of single-function home service devices by introducing a dual-core architecture of a "life-assistance execution unit" and an "adaptive physiotherapy execution unit." The robot can not only perform household chores such as floor cleaning and tidying up, but also provide professional physiotherapy services. More importantly, the central control system enables the scenario-based linkage between the two.

This invention relates to the field of intelligent technology for identifying hidden dangers on reservoir and river / lake embankments, specifically to a method for intelligent identification and risk assessment of embankment hidden dangers based on multimodal perception using a walking robot. The method involves a quadruped robot equipped with multiple sensor modules autonomously inspecting reservoir and river / lake embankment areas to collect multimodal data from the embankment surface. This multimodal data is then transmitted to a data processing center. Based on this data, the types and locations of hidden dangers on the embankment surface are identified. These hidden danger types include cracks, seepage or piping, settlement, and structural deformation. The types and locations of these hidden dangers are quantitatively analyzed to obtain quantitative results. Based on these results, a risk assessment report, a visual map, and an early warning are generated. This invention improves the spatial coverage accuracy and temporal synchronization of data acquisition, and possesses higher sensitivity and complementarity in identifying hidden danger types such as embankment cracks, seepage, and deformation, providing a perceptual foundation for subsequent analysis.

This invention discloses a method for accurate identification of the moment of an accident in a new energy vehicle based on multimodal data fusion. The method acquires vehicle motion sensor data, dynamic parameters, and visual image data, and then performs the following parallel processing: The physical layer processes the data to obtain motion features and compares them with preset trigger conditions; when a suspected collision is detected, it outputs the physical layer trigger time and acceleration amplitude; the dynamic layer processes the data based on the deviation between the vehicle's dynamic model and its actual motion state, outputting a vehicle trajectory status label; the visual layer processes the data to detect targets and analyze their motion state; when visual evidence of a collision is detected, it outputs the visual layer trigger time and the type of collision object; the fusion layer performs spatiotemporal alignment of the three layers' outputs and makes a comprehensive judgment, outputting the fused accurate collision trigger time and accident scene classification information. This invention effectively filters false alarms and improves the detection rate of low-speed accidents through multimodal fusion, achieving accurate identification of the moment of an accident and scene reconstruction.

This invention discloses a method for semantic encoding / decoding and reliable transmission of multimodal data oriented towards channel polarization. The method includes: acquiring anchor points and target modal semantic features of multimodal data; calculating the reliability of polarization subchannels; asymmetrically mapping the two types of features to different reliability intervals according to priority and encoding and transmitting them; the receiver preferentially decodes anchor point features to reconstruct cross-modal semantic prior vectors; using the SCL algorithm to decode and split the target features, reconstructing the local semantic features of the current split path and calculating the cross-modal semantic distortion with the prior vector; and dynamically generating a joint path metric based on polarization feature parameters for list pruning, breaking the cyclic redundancy check red line, blocking retransmissions, and directly outputting the path with the minimum semantic distortion. This invention breaks down the separation between the physical and semantic layers, avoiding erroneous pruning and retransmissions caused by local noise in deep fading channels, and achieving semantic coherence and reliable transmission in complex environments.

This invention proposes a comprehensive agricultural four-condition monitoring system based on big data and its early warning and prediction method. It belongs to the field of smart agriculture technology. The method includes: performing multi-source sensor spatiotemporal alignment processing on the agricultural four-condition monitoring area to generate a spatiotemporally synchronized multi-dimensional monitoring dataset of soil moisture, insect infestation, seedling condition, and climate; constructing an agricultural causal knowledge graph based on this dataset; dynamically and adaptively adjusting thresholds according to the agricultural causal knowledge graph and initiating multimodal data acquisition to obtain the original agricultural four-condition monitoring data and corresponding spatiotemporal coordinate data; performing joint noise reduction processing on the original agricultural four-condition monitoring data using physical models and data-driven methods to generate high-quality agricultural four-condition feature data; and using this early warning and prediction method of the comprehensive agricultural four-condition monitoring system based on big data, the accuracy of agricultural risk early warning can be greatly improved, allowing farmers to promptly grasp potential risks in agricultural production.

This invention discloses a multimodal data security sharing method based on information security. Addressing the problems raised in previous technologies, such as data becoming ineffective when it leaves a controlled domain, and the inability of subsequent dynamic control attempts to be implemented due to the lack of a trusted, data-co-existing execution endpoint, and the inability of security policies to learn and adjust based on real-time feedback during data usage, this invention proposes the following solution: policy-driven multimodal data preprocessing and labeling. For the raw multimodal data to be shared, a sensitive information identification model corresponding to the modality is invoked for analysis. This invention solves the industry problem of data loss of control after leaving the domain, promotes real-time collaboration and evolution of security capabilities, drives the continuous evolution of policies and identification models, and constructs a unified framework and modality-adaptive cross-modal data security management solution. It transforms complex security operation and maintenance problems into simple policy definition problems, significantly reducing the technical threshold and operational costs of secure sharing.

This invention discloses a mixed reality data processing and interactive response method, apparatus, and system. The method includes: acquiring corresponding spatial feature points in VR and AR environments; optimizing the solution of rigid body transformation matrices to achieve sub-millimeter-level spatial alignment; continuously monitoring alignment errors during virtual-real fusion rendering, triggering a repositioning process when errors exceed a threshold; evaluating user operations through multimodal data fusion and generating real-time AR correction guidance; and dynamically adjusting rendering parameters and resource preloading based on visual attention focus to ensure end-to-end latency and interactive feedback latency are both below set thresholds. The apparatus includes modules for feature point acquisition, spatial alignment calculation, error monitoring and repositioning control, multimodal data interface, and real-time rendering control. The system includes a server and a user interaction terminal, supporting intelligent training and adaptive interactive optimization for virtual-real fusion. This application addresses problems such as low virtual-real spatial alignment accuracy, high interactive latency, and insufficient personalized adaptation.

This application discloses an open-world 3D target detection method and apparatus, comprising: acquiring multimodal data of a target scene; performing 3D target detection on 3D point cloud data to obtain 3D candidate targets; inputting the geometric features of each 3D candidate target into an out-of-distribution target classifier to determine whether the 3D candidate target belongs to a known category set and to identify unknown targets; projecting the 3D candidate targets corresponding to the unknown targets onto 2D image data, obtaining the 2D image region, and inputting it into a visual language model, using natural language prompts to guide the visual language model to output the semantic category name of the unknown targets; fusing the semantic category name and the spatial location information of the 3D candidate targets to generate open-world 3D detection results. This application can effectively identify unknown 3D targets in an open world and generate open-world 3D detection results that combine spatial positioning and semantic description.

This disclosure provides a multi-slice data processing method, apparatus, electronic device, and storage medium. The multi-slice data processing method includes: acquiring multimodal data of multiple target slices; clustering the multimodal data based on an analysis unit for each target slice to determine multimodal labeled data, whereby the multimodal labeled data describes the association between the target slices and target label data; extracting features from the multimodal data using an initial feature extractor based on a preset neural network model to obtain a multimodal feature matrix; performing label prediction on the multimodal feature matrix using a label prediction unit based on the preset neural network model to obtain predicted label data; training the preset neural network model based on the target label data and the predicted label data; and obtaining a target feature extractor for multi-slice feature extraction of multiple target slices based on the trained preset neural network model. The embodiments of this application provide a method suitable for multi-slice data processing.

The application discloses a kind of based on multimodal data fusion's spherical fruit quality detection method, including mobile platform moves to the coordinate position of detected plant;Real-time identification fruit's space three-dimensional coordinates by the visual auxiliary positioning device set on mobile platform;Mechanical arm control system is controlled to move by mechanical arm, so that flexible mechanical paw moves to the space coordinates of detected fruit;Flexible mechanical paw carries out multiple directions'snatching action to detected fruit, and multiple directions'sensed fruit tactile data are dynamically collected by multiple-point array flexible film pressure sensor, and multiple directions'sensed fruit image data are collected to sensed fruit by industrial camera;By multimodal data processing unit, the deep semantic features and dynamic response characteristics of multiple directions'sensed fruit are extracted respectively, the deep semantic features and dynamic response characteristics are spliced and fused after input to MLP multilayer perception machine and are fused and analyzed, and the quality evaluation result of sensed fruit is obtained.

This invention discloses a method for autonomously constructing an evolvable knowledge graph by integrating a multimodal large-scale model, relating to the fields of artificial intelligence and knowledge graph technology. The method includes: collecting raw multimodal data from text, images, audio, and structured tables; using a pre-trained multimodal understanding large-scale model to perform unified semantic encoding on the raw multimodal data, generating modality-independent deep feature vectors to form an initial multimodal feature pool; performing cross-modal clustering analysis on the vectors in the feature pool, grouping semantically similar vectors into the same feature cluster, with each feature cluster defined as a candidate knowledge concept node; constructing an initial concept relationship network based on the co-occurrence relationship and feature similarity between candidate concept nodes; iteratively optimizing and evolving the network according to preset graph quality evaluation indicators, outputting the final evolvable knowledge graph. This invention achieves the autonomous and unified construction and evolution of a knowledge graph from multimodal data.

A foundation model for performing molecular-level tasks by learning multimodal data in the form of one-dimensional text and two-dimensional graphs, and a system therefor, according to an embodiment of the present invention, enable various molecular-unit tasks such as chemical reaction prediction, molecular attribute prediction, and natural language description generation to be effectively processed through a single foundation model. In addition, it is possible to increase prediction accuracy of the model by maximizing utilization of two-dimensional molecular graph information, and automatically generate, on the basis of statistical sparsity, high-quality descriptive text that emphasizes core and distinctive features of each molecule.

This invention discloses a method and system for digital evaluation of rural tourism based on multimodal data. The method includes: collecting multimodal data from rural tourism scenarios to construct a multimodal dataset; encoding the features of each modality to obtain corresponding modal feature vectors; performing cross-modal fusion processing on the modal feature vectors to generate fused feature representations; applying semantic alignment constraints to the modal feature vectors; constructing a feature-evaluation mapping model and a weight determination model; jointly training the feature-evaluation mapping model and the weight determination model by jointly optimizing the objective function; based on the trained model, performing inference on the real-time collected multimodal data to obtain and output the comprehensive digital evaluation result of rural tourism; and simultaneously updating the model online based on newly accessed multimodal data. This invention can significantly improve the accuracy of digital evaluation of rural tourism.

This invention belongs to the field of artificial intelligence technology. The embodiments of this invention provide a fully automated interview assistance method, system, and device based on a large language model. First, it acquires recruitment requirements, initial interview questions, and multimodal data of job seekers from the recruitment end to construct a multimodal dynamic knowledge base. This multimodal dynamic knowledge base includes content dynamic mechanisms, structure dynamic mechanisms, and iterative dynamic mechanisms, used to update the knowledge base content and sub-question sequences, update the structure of the knowledge graph in the multimodal dynamic knowledge base, and optimize sub-question generation and retrieval strategies, respectively. Then, based on the multimodal dynamic knowledge base, it determines the job seeker's structured question set, and subsequently determines the job seeker's interview assistance strategy. This invention, through three update mechanisms, makes knowledge application more closely aligned with actual interview scenarios, overcoming the shortcomings of traditional static and rigid knowledge bases.

This invention discloses a method and system for feature extraction of multi-layer stacked cardboard boxes based on 3D vision. Addressing the problem that existing technologies cannot accurately quantify surface anomalies and multi-dimensional stacking features of cardboard boxes in complex stacking scenarios, this invention acquires the original point cloud and RGB images of the stacking scene; after preprocessing, a height distribution histogram is constructed and clustered to obtain initial point cloud clusters; warped anomalies are eliminated through curvature analysis and normal vector consistency to obtain accurate upper surface point clouds; principal component analysis is used to obtain the length, width, and normal vector direction of the cardboard box; basic geometric features, damage features, occlusion relationships, overhang / overlap features, tilt features, and multi-layer interlacing features are extracted by combining RGB images; finally, the feature vector of each cardboard box is output in a structured manner. This invention achieves refined point cloud processing, effectively eliminates warped points, systematically quantifies complex stacking features, and integrates multimodal data, significantly improving the ability to identify and extract features from cardboard boxes in complex logistics scenarios.

The application discloses a kind of green ecological monitoring management systems based on knowledge graph, including multimodal data reading module: generate first preprocessed data and second preprocessed data;Multimodal feature extraction module: first preprocessed data is input to ShuffleNet model, and second preprocessed data is input to RoBERTa model;Ecological space topology construction module: constructs ecological space topology atlas;Space vector generation module: based on two-stage space mapping mechanism, generates visual space vector and text space vector;Space probability model construction module: constructs space probability model;Cross-modal space inference and matching module: execute cross-modal causal consistency inference;Ecological knowledge graph generation module: realize the knowledge graph storage and update of multimodal entity pair.The application realizes higher precision multimodal fusion and more complete knowledge graph construction effect in complex ecological monitoring scene.

This invention relates to the field of coal quality testing technology, specifically to an online method and system for detecting coal quality entering the furnace; it includes: a multimodal data acquisition step: acquiring microwave scattering parameters and surface element characteristic spectra using a microwave tomography sensor array and a surface spectrometer, respectively; a physical field inversion and modeling step: reconstructing the complex permittivity distribution map based on the scattering parameters to establish a three-dimensional physical distribution model including density and moisture; a field-spectrum coupling deduction step: using the characteristic spectra as boundary constraints, and combining them with the complex permittivity distribution map to deduce the internal chemical element distribution; and a volume-weighted quantization step: performing volume-weighted integration on the three-dimensional physical model and chemical distribution to output comprehensive coal quality parameters; this invention eliminates the detection blind spots of single technologies and significantly improves the representativeness and safety of full-section detection of coal entering the furnace.

The application relates to the technical field of drug efficacy prediction, in particular to a multi-modal data prediction model for diabetes gene therapy response, which comprises an unmedicated data collection module, a medicated data collection module and a data prediction module.The unmedicated data collection module collects unmedicated data samples of a user; the medicated data collection module collects data samples of the user after taking medicine; the data samples comprise explicit data and implicit data; the explicit data is body information and medicine information related to blood glucose change; and the implicit data is blood glucose values corresponding to the explicit data; the data prediction module inputs fused features into a convolutional network model to generate drug efficacy prediction values.The application generates a prediction residual corresponding to the explicit data after taking medicine through a blood glucose value prediction module, identifies blood glucose fluctuation characteristics caused by non-drug factors by using the prediction residual, separates the net influence of medicine on blood glucose, and thus improves the accuracy of drug efficacy prediction.

The present application relates to big data risk control technical field, especially to a kind of multimodal data's business travel exception capture method and system, method includes: the multimodal data in business travel service is synchronously acquired, and the standardization preprocessing is carried out to multimodal data;Feature extraction is carried out to the multimodal data after standardization preprocessing;Based on the transaction statistical features, time series behavior features and semantic correlation features extracted, a dynamic heterogeneous entity relationship graph is constructed;On the dynamic heterogeneous entity relationship graph, an anomaly detection model based on graph neural network is used, the embedding representation of the nodes is learned, and the abnormal score of the target business travel order is calculated;According to the comparison between abnormal score and adaptive risk threshold value.The present application effectively breaks through the limitation of traditional anomaly detection relying on single data source and static rules, and realizes the capture of business travel exception of multimodal data.

The application discloses a kind of based on multimodal machine learning's bottom mud repair method and system, it is related to environmental remediation technical field, wherein method includes: obtaining satellite remote sensing data of target area, in-situ sensor data and laboratory data are composed of multimodal data;Based on multimodal data, pre-processing and feature selection are sequentially carried out, to obtain key feature set;Based on key feature set, historical pollution data and ecological sensitive area location information are combined to generate risk-cost trade-off suggestion;Based on risk-cost trade-off suggestion and multi-objective optimization algorithm, to generate optimized repair scheme;Based on optimized repair scheme, repair task is executed in target area, and the state data of target area in execution process is obtained;Based on state data, to adjust optimized repair scheme.Pollution dynamic diagnosis, real-time optimization and automated accurate execution of repair scheme are realized, and then the efficiency of complex water bottom mud treatment is improved.

This disclosure proposes a method for synthesizing multimodal data for decision-making attribution explanation in the mining industry, comprising the following steps: First, extracting causal logic from industry texts to construct a structured rule base; second, collecting on-site multimodal data and filtering out data pairs strongly correlated with rules through scenario-based weight modeling and cross-modal association mechanisms; third, using a multimodal generative adversarial network with rule constraint mechanisms and combined with rule mutation technology to synthesize multimodal data with complete logical chains and covering various complex working conditions; finally, ensuring data quality and achieving self-evolution of the knowledge base through a three-level verification and rule feedback mechanism, and storing qualified data in a hybrid storage database. This invention solves the problems of complex causal relationships and scarce high-quality training samples in the mining field, and can automatically and in batches generate logically rigorous and scenario-diverse multimodal data, improving the training efficiency and generalization ability of attribution decision models.

This invention relates to the field of image processing technology and discloses a multimodal data fusion method for online quality monitoring of plastics. The method includes: simultaneously acquiring a reference optical image of the plastic surface and a temporal infrared thermal distribution image containing the current frame and the previous frame infrared image; calculating a spatial temperature gradient matrix; correcting the geometric affine matrix based on the displacement vector normal component determined by the dense optical flow field, constructing a spatiotemporal compensation mapping matrix to achieve alignment of heterogeneous pixel coordinate systems, and generating a structure-guided tensor; applying morphological opening operations to remove low-frequency background thermal fluctuations from the structure-guided tensor to obtain a pure-state high-frequency thermal gradient matrix; adjusting the diffusion coefficient using the pure-state high-frequency thermal gradient matrix as a constraint, applying anisotropic diffusion filtering to the reference optical image, and extracting defect regions. This invention achieves physical-level decoupling between optical artifacts and real defects, eliminates feature mapping deviations caused by thermal conduction hysteresis, and improves the signal-to-noise ratio of image feature extraction.

The present application relates to the technical field of train dispatching, in particular, the present application relates to a kind of train service work order priority dynamic scheduling method and system based on multi-modal fusion, real-time acquisition work order multimodal data in the present application, fusion structured field, fault text semantics and visual features by cross-modal semantic alignment model, generate work order feature vector, combine basic weight rule set to calculate initial priority;Monitoring work order time limit difference, emergency order superposition incremental emergency promotion coefficient with difference reduction;Based on the number of load work order, work hour saturation and mobile state calculation master resource utilization rate, high load area inhibits the priority of non-emergency work order, and is preferentially assigned to idle master;Finally, according to work order priority weight global ordering, combine master real-time state and position, generate optimal allocation instruction execution scheduling, realize multi-modal information and resource state coordination, balance work order time limit and regional load, improve scheduling efficiency and resource utilization rate.