47 results about "Modal method" patented technology

The invention relates to the technical field of deep learning and defect detection, and discloses a multi-modal fusion defect perception and identification method based on deep learning, and the method comprises the steps: 1, obtaining multi-modal data: obtaining the multi-modal data of a target object through a collection device, and obtaining the multi-modal data of the target object; comprising visual data and non-visual data, and the data format covers two-dimensional images, three-dimensional point clouds, time series data and the like. According to the multi-modal fusion defect perception and identification method based on deep learning, the advantages of visual data and non-visual data can be integrated through multi-modal data fusion, the characteristics of a target object are described more comprehensively, information loss caused by single-modal data is reduced, the accuracy of defect detection is improved, and in industrial part detection, the detection efficiency is improved. By fusing the image and the point cloud data, defects such as cracks and holes on the surface of the part can be identified more accurately, and the detection precision is obviously improved compared with that of a single-mode method.

The invention discloses an intelligent tea withering control system based on multi-modal feature fusion and time sequence prediction. The system is composed of a multi-modal feature extraction module, a time sequence modeling module, a transfer learning module and an intelligent decision control module, a Transform attention mechanism is adopted to construct a cross-modal fusion framework, RGB images, hyperspectrum and time sequence information are fused, and accurate recognition and trend prediction of the withering state are achieved. According to the system, an adaptive attention fusion network is designed, optimal fusion of images, spectrums and grade information is realized through dynamic distribution of modal weights, and the recognition accuracy and stability are remarkably improved. The classification accuracy of 180 verification samples reaches 93.33%, and is improved by 15.93%-34.83% compared with that of a single-mode method. And cross-environment self-adaption is realized through fusion transfer learning, and the performance is improved by 6.7%-14.3%. The intelligent decision engine optimizes temperature and humidity parameters based on a multivariable coupling control theory, the control precision reaches + / -0.5 DEG C and + / -2.0%, and the response time is 2.5 seconds.

The invention discloses a ship propulsion shafting fault diagnosis method based on multi-modal attention fusion, and relates to the technical field of ship fault diagnosis. According to the method, multi-modal data such as vibration parameters, lubricating oil parameters and cooling water parameters of key parts of a ship propulsion shafting are synchronously collected and preprocessed; a modal specific feature extraction strategy is adopted, a multi-modal attention fusion mechanism including intra-modal self-attention, inter-modal cross attention and adaptive dynamic weight distribution is designed, and heterogeneous modal features are effectively integrated. Compared with a single-mode method and a traditional feature splicing method, the method has the advantage that the diagnosis accuracy is remarkably improved. The method has high robustness, and can still maintain high diagnosis accuracy even under the condition that part of sensors fail.

The invention relates to the technical field of data processing, in particular to a classroom teacher teaching performance description method, model and system based on multi-modal data fusion and a storage medium. By introducing a multi-modal data fusion strategy, cooperative processing of classroom teacher visual information, audio information and text information is realized, and the accuracy and time sequence continuity of teacher target perception are significantly improved. Compared with a traditional single-mode method, the method not only can extract the posture, expression and action characteristics of the teacher from the visual mode, but also can recognize the voice emotion and the side language signal from the audio mode, and achieves the full-dimensional description of the teaching behavior of the teacher in combination with the text semantics. The objective of the invention is to solve the problem of how to perform multi-dimensional evaluation on teaching performance of a classroom teacher based on data of multiple modalities.

The invention provides an image fuzzy detection method based on frequency domain analysis and deep learning fusion. The method comprises the following steps: S1, frequency domain feature extraction and quantification; s2, spatial domain feature extraction and modeling; s3, carrying out multi-modal feature fusion; s4, joint optimization and post-treatment are carried out; s5, outputting and verifying; through complementarity design of frequency domain and deep learning, complex fuzzy detection requirements of static images and video streams are covered, high efficiency and reliability are verified in industrial quality inspection, video conferences and other scenes, energy attenuation characteristics caused by global blur are accurately captured through frequency domain analysis, motion blur and out-of-focus blur are effectively distinguished, and the method is suitable for large-scale popularization and application. According to the method, local texture degradation of deep learning network modeling, dynamic track abnormity analysis of an optical flow network and complex scenes covering static images and video streams are realized through a bidirectional feature fusion mechanism, the mAP of mixed fuzzy detection is effectively improved compared with a single-mode method, and dynamic fuzzy and static out-of-focus fuzzy are effectively distinguished.

The invention discloses a news public opinion analysis method based on multi-modal graph representation learning, and belongs to the field of multi-modal deep learning. Constructing a graph structure from the text and image features by adopting a graph topology extraction and enhancement module; a cross-modal multi-scale feature fusion module is adopted to align and integrate features of texts and images in a multi-scale manner, and the cross-modal multi-scale feature fusion module comprises an ASPP and a global attention mechanism module; a final expansion mode feature, a single-mode text mode feature and an image mode feature are obtained through an image topology extraction and enhancement module and a cross-mode multi-scale feature fusion module and are input into a classifier together to obtain respective probabilities, and finally a news public opinion classification result is obtained through weighted fusion. The performance of the DCPNet on a multi-mode data set is superior to that of a traditional single-mode method and other existing multi-mode irony detection methods, and the huge potential of the DCPNet in practical application is proved.

The disclosure relates generally to methods and systems for real time video driven human 3-dimensional (3-D) posture estimation during physical activities. Conventional techniques do not exploit temporal information, they do not give smooth transition of postures over time. Furthermore, the techniques that exploit the temporal information suffer from higher time requirements due to two state computations. The present disclosure solves the technical problems in the art with the methods and systems for real time video driven human 3-D posture estimation during physical activities. The present invention discloses a smart-phone camera based automatic posture monitoring system designed with an auto-encoder based architecture. The disclosed auto-encoder based cross-modal method uses monocular video (2-D image sequences) from a single low-end mobile device (for example, smart-phone camera) for estimating human 3-D posture in real time (˜5 fps) with high accuracy (less than 1 cm error per joint location).

The invention relates to the technical field of multi-modal document analysis, and discloses a multi-modal document analysis method and device, electronic equipment and a storage medium. The method comprises the steps that picture elements in a document are automatically recognized and extracted through a third-party library module, placeholders are generated in combination with picture positions, and the placeholders are stored in the third-party library module; and then semantic understanding and text generation are carried out by using a multi-modal large model, so that the visual content is converted into a structurable information text. The method has the advantages that by introducing the multi-modal large language model, unified analysis of multi-modal information such as texts, pictures, tables and flow charts is achieved, layout semantics and logic structures of documents are reserved, fusion analysis of image-text content is achieved, the integrity and the intelligent level of document analysis are remarkably improved, and the method is suitable for large-scale popularization and application. Compared with a traditional single-mode method, the method has the advantages that unified understanding and structured expression can be carried out on multi-mode content, and richer and more accurate structured results can be output.

The present application relates to a kind of cross-modal methods for visual identification in large-scale point cloud map, comprising: based on the RGB image data and point cloud data collected in global map, construct data set;The data set is converted cross-modal;Using converted data set, the preset network model is trained, and cross-modal positioning model is obtained;Wherein, the preset network model includes: multi-scale feature encoder, cascaded cross attention module and projection converter;RGB image data not included in the data set is input into the cross-modal positioning model, and the position of sensor in global map is obtained.The present application can significantly improve the image-to-point cloud cross-modal location recognition accuracy and stability in unknown indoor and outdoor environment, and simultaneously due to its lightweight design, increase the feasibility of the cross-modal location recognition device in practical application deployment.

Disclosed are integrated DFOS / DAS systems, methods, and structures that employ a large-scale pretrained recognition model we refer to as an “acoustic-language model”, which is pretrained with natural-language supervision (“contrastive language-audio pretraining”. The acoustic-language model comprises two primary components: an acoustic encoder and a text encoder. These encoders are pretrained using a cross-modal approach on a vast dataset of acoustic features (such as images created from log Mel spectrograms) and their corresponding textual captions. When acoustic features and / or languages are input into their respective encoders within the model, they generate corresponding embedding vectors. Both embedding vectors are then linked in a joint multimodal space using linear projections. The acoustic classification tasks using this model are executed by assessing the similarity between the acoustic and language embedding vectors, essentially evaluating the maximum similarity between the acoustic features and the events described in a specific language.

The application belongs to the field of material data, and discloses a simulation data generation method of metal material multi-modal fusion. The method comprises the following steps: collecting data, extracting features to obtain a feature vector, fusing the feature vector through a multi-modal method to obtain a metal material multi-modal feature training data set, constructing a metal material simulation data diffusion model and a diffusion model loss function meeting physical condition constraints, training the diffusion model, and outputting high-quality simulation metal material data meeting the basic physical law of the material. The simulation data generation method of metal material multi-modal fusion provided by the application introduces physical constraints, so that the generated simulation data meets experimental data from a statistical perspective and meets the basic physical law. The method can effectively generate simulation data of metal materials under the condition of limited data, significantly reduces the experimental and calculation costs, and simultaneously fuses numerical and image modal data, which is more in line with the organization-property evolution relationship of the material.

The present invention discloses a multimodal crack detection method, medium, and device. The multimodal method includes intensity and range modes. The method comprises extracting intensity crack features and range crack features simultaneously, introducing two sets of decoupling adapter modules to fine-tune the SAM2 encoder; inputting the intensity crack features, range crack features, and their fusion features into the SAM2 decoder; employing a gradient conflict resolution method to mitigate gradient conflicts, thereby obtaining three updated gradients. Finally, all updated gradients are summed to generate a final gradient; and the updated gradients are used to update the parameters of the intensity and range encoders and decoder, ultimately generating a crack detection map.

The invention relates to a cross-modal method for visual identification in a large-scale point cloud map, and the method comprises the steps: building a data set based on RGB image data and point cloud data collected in a global map; performing cross-modal conversion on the data set; training a preset network model by using the converted data set to obtain a cross-modal positioning model; wherein the preset network model comprises a multi-scale feature encoder, a cascade cross attention module and a projection converter; and inputting the RGB image data which are not included in the data set into the cross-modal positioning model, and obtaining the position of the sensor in the global map. According to the invention, the precision and stability of cross-modal position recognition from images to point clouds in unknown indoor and outdoor environments can be remarkably improved, and the deployment feasibility of the cross-modal position recognition device in practical application is improved due to the lightweight design.

The invention discloses a multimodal transport method and system suitable for a salt mist environment, and the method comprises the steps: building a multimodal transport salt mist adaptive transport capacity grid model with city / hub nodes as vertexes and connected road sections as edges; configuring a salt mist adaptation device at the node, and marking the node protection capability, the availability time window and the supply capability; meanwhile, a model hypothesis is set, and cargo transportation integrity, facility availability, transportation rules and a parameter acquisition mode are determined; defining a multi-dimensional decision variable, a parameter and a state quantity; incorporating the salt mist environment influence into cost composition, and constructing a salt mist correlation total cost model; constructing a mixed integer programming model containing path connection and flow conservation, protection capability constraint, energy / endurance constraint and corrosion / exposure constraint by taking the minimum total cost as a target; and designing an improved hybrid immune-particle swarm dual optimization algorithm, and carrying out optimization solution, optimal path output, mode selection and protection parameter configuration through hybrid coding, security repair operators and a memory mechanism.

The present application relates to a fatigue gait pattern recognition device and method based on a multi-modal sensor, multi-modal sensor data is collected based on a multi-modal sensor, gait segmentation is performed after preprocessing, data samples after gait segmentation are obtained, including multi-modal sensor data and corresponding data features; an improved feature fusion model is constructed, and the model is trained to be stable with data samples; multi-modal sensor data is collected, and the preprocessed data is input into the trained model, and the fatigue gait pattern recognition result is output; the device includes two groups of multi-modal sensors for collecting multi-modal sensor data, and the synchronous transceiver device sends the corresponding two groups of multi-modal sensor data after matching; the controller acquires the multi-modal sensor data and recognizes the fatigue gait pattern. The present application realizes high-precision classification of fatigue gait pattern, and the accuracy is improved by 15%-20% compared with traditional single-mode method; based on dynamic threshold classification and multi-dimensional feature analysis, user-specific rehabilitation suggestions are generated, and walking ability and safety are improved.

The invention discloses a special vehicle task state recognition method based on cross-modal alignment, and the method comprises the steps: constructing an image-audio cross-modal retrieval model, and achieving the deep interaction of image features and audio features through the hierarchical extraction of features; performing multi-granularity space-time correlation analysis on vehicle appearance features in the image and siren spectrum features in the audio by using a forward and backward sequence-to-sequence model two-way architecture, and dynamically screening out cross-modal redundant information; based on attention weight adaptive fusion effective features, outputting a special vehicle task state discrimination result and cross-modal confidence assessment; according to the invention, accurate identification of the task state of the special vehicle in a complex environment is realized, and good technical support is provided for automatic driving to avoid the special vehicle which is executing the task in intelligent traffic; the problems that traditional single-mode recognition is insufficient in reliability in a complex environment, and an existing cross-mode method is rough in feature alignment granularity and poor in dynamic adaptability are solved.

The invention discloses a scene identification method and system based on vision-point cloud multi-modal fusion, and the method comprises the steps: collecting a visual image and laser radar point cloud data in a target scene, carrying out the preprocessing, and extracting corresponding visual image features and point cloud features; generating a global feature, a visual modal local feature and a point cloud modal local feature in the target scene; performing global retrieval in a preset scene database by utilizing the global features to obtain a plurality of candidate positions and corresponding global matching scores; performing local matching and geometric verification on the candidate positions based on the visual modal local features and the point cloud modal local features to obtain a local matching score of each candidate position; and reordering the candidate positions according to the global matching score and the local matching score, and outputting an optimal scene recognition result. The method solves the problems of poor feature alignment, insufficient modal fusion, weak complex scene adaptability and data redundancy in the existing multi-modal method.

The invention discloses a multi-modal false comment recognition method and system. The method comprises the steps that text data and behavior data are preprocessed to generate word embedding vectors and behavior feature vectors; inputting the word embedding vectors into a plurality of multi-scale space weighted convolution modules in parallel to generate multi-scale semantic features; inputting the multi-scale semantic features into a multi-scale context aggregation module to generate a semantic feature fusion vector; inputting the semantic feature fusion vector into a BiLSTM-Attention module to generate a semantic enhancement vector; and after the behavior feature vector and the semantic enhancement vector are spliced, inputting into a graph Laplacian layer for classification, and outputting a false comment identification result. False comment identification is carried out through a multi-modal method, the defects of a traditional single-modal method are overcome, and the identification accuracy and generalization ability are improved.

The present invention discloses a method for simulating the dynamic response of a rotating flexible beam with partially covered active constrained layer damping (ACLD) based on sliding mode control (SMC). A portion of the rotating flexible beam is covered with a piezoelectric damping layer and discretized using the assumed modal method. The influence of high-order coupling terms is considered. Based on the dynamics of rigid-flexible coupled multi-body systems and sliding mode control theory, the rigid-flexible coupling dynamic equations of the partially covered ACLD rotating flexible beam based on SMC are established. The generalized α method is used to solve the dynamic equations of the partially covered ACLD rotating flexible beam system based on SMC, and a lateral displacement-time curve diagram of the end of the partially covered ACLD rotating flexible beam based on SMC is obtained. The present invention provides a new control model for the vibration control of a rotating flexible beam structure, which has a better control effect on the vibration suppression of the rotating flexible beam structure.

Disclosed are integrated DFOS / DAS systems, methods, and structures that employ a large-scale pretrained recognition model we refer to as an "acoustic-language model", which is pretrained with natural-language supervision ("contrastive language-audio pretraining". The acoustic-language model comprises two primary components: an acoustic encoder and a text encoder. These encoders are pretrained using a cross-modal approach on a vast dataset of acoustic features (such as images created from log Mel spectrograms) and their corresponding textual captions. When acoustic features and / or languages are input into their respective encoders within the model, they generate corresponding embedding vectors. Both embedding vectors are then linked in a joint multimodal space using linear projections. The acoustic classification tasks using this model are executed by assessing the similarity between the acoustic and language embedding vectors, essentially evaluating the maximum similarity between the acoustic features and the events described in a specific language.

The invention discloses a spectrum library optimization method and system based on sensitivity analysis, and belongs to the field of semiconductor optical measurement, and the optimization method comprises the steps: obtaining a simulation spectrum; carrying out sensitivity analysis on preset library building parameters based on the simulated spectrum, and obtaining the sensitivity of each library building parameter; obtaining a parameter value range of each library building parameter, and obtaining a division weight of each library building parameter based on the sensitivity of each library building parameter and the parameter value range of each library building parameter; calculating the grid division number of each library building parameter based on the division weight of each library building parameter; grid points are obtained according to the grid division number of each library building parameter, each grid point corresponds to a set of geometric parameters, and a parameter network is obtained; and geometric parameters corresponding to each grid point in the parameter network are simulated through a Fourier modal method, and an optimized spectrum library is generated. The spectrum library sample obtained through the optimization method is balanced in distribution and small in deviation, and the precision and stability of the neural network model are improved.

This invention discloses a dual-modal fusion wavelet-enhanced six-image hyperspectral classification method and system. The method includes: acquiring spectral and RGB image data of the six images; extracting features from the spectral and RGB image data respectively; enhancing the feature-extracted data using a learnable wavelet enhancement module; flattening and stitching the enhanced data; fusing the stitched feature sequences to obtain a fused feature sequence; embedding location information and category labels into the fused feature sequence, and feeding it into a Transformer encoder for global context modeling and classification. This invention's dual-modal fusion wavelet-enhanced six-image hyperspectral classification method and system achieves the fusion and utilization of hyperspectral imaging, near-infrared spectroscopy, and digital image features through feature-level and data-level fusion, fully utilizing the spatial-spectral information of multiple data sources, leveraging the advantages of various data, and effectively overcoming the limitations of single-modal methods.

The present disclosure provides a multi-modal method for classifying a thyroid nodule based on ultrasound (US) and infrared thermal (IRT) images. Based on ultrasound and infrared thermal images and in combination with a multi-modal learning method, the present disclosure provides an adaptive multi-modal hybrid (AmmH) model which is composed of three parts: an intra-modal hybrid encoder (HIME), an adaptive cross-modal encoder (ACME), and a multilayer perceptron (MLP) head. The HIME is capable of modeling a global feature while extracting a local feature. The ACME is capable of customizing personalized modality-weights according to different cases and performing information interaction and fusion of inter-modal features. The MLP head classifies a fused feature obtained. The method enables the AmmH model to automatically classify a thyroid nodule of a subject based on ultrasound and infrared thermal images of the subject, providing a doctor with an objective and accurate classification result to assist diagnosis.

The present disclosure concerns a methodology that allows a user to “orbit” around a model on a specific axis of rotation and view an orthographic floor plan of the model. A user may view and “walk through” the model while staying at a specific height above the ground with smooth transitions between orbiting, floor plan, and walking modes.

The invention relates to the technical field of image classification, in particular to a visual text prompt-based multi-modal network few-sample image classification method, which comprises the following steps of: acquiring an image, a learnable prompt and a manual prompt; obtaining a frequency domain prompt fusion feature map; obtaining an original image feature vector; inputting the original image and the frequency domain visual fusion feature map into an image encoder to obtain a map frequency domain prompt fusion feature vector; respectively inputting the learnable prompt and the manual prompt into a text encoder to obtain a learnable prompt feature vector and a manual prompt feature vector; calculating a similarity score between the original image feature vector and the manual prompt feature vector; calculating a similarity score of the graph frequency domain prompt fusion feature vector and the learnable prompt feature vector; and constructing the regularization of the minimum divergence to the consistency among the vision, the text prompt and the distribution. According to the method, the problem of coordination and integration of different modal information and consistency of cross-modal semantics in an existing multi-modal method is solved.

The invention belongs to the technical field of electric digital data processing, and particularly relates to an expressway scene-oriented interpretable hierarchical reasoning multi-modal method and system, and the method comprises the steps: obtaining multi-modal data, and carrying out the preprocessing of the multi-modal data; performing feature coding and semantic fusion on the preprocessed multi-modal data to form uniform features; performing hierarchical recursive collaborative reasoning on the unified features, including performing global semantic understanding and long-term reasoning through a high-level part to obtain a final semantic state; data updating, fine-grained calculation and supplementary inference are carried out through the low-layer part; a natural language answer is generated through an encoder based on the output of the high-level part and the low-level part under the condition of the final semantic state, and the contribution proportion of the natural language answer is calculated by calculating the interpretation distribution of the high-level part and the low-level part; after interpretability analysis is carried out on the natural language answers, causal activation mapping calculation is carried out on each generated result, and a visual interpretation graph is obtained.

The invention relates to a modal method and geometric precision beam mixed three-dimensional large deformation monitoring method for a beam structure, belongs to the technical field of structure health monitoring, solves the problems of high generalized strain calculation complexity of the structure and too high requirement on the mounting position of a sensor in the prior art, and comprises the following steps: S1, initializing calculation; mathematical representation is carried out on the geometric configuration of the to-be-monitored beam structure, and a global coordinate system and a local coordinate system are established; s2, obtaining generalized strain of a beam center reference line of the to-be-monitored beam type structure based on a modal superposition method; s3, calculating a spatial position vector of a beam center reference line; s4, calculating a spatial position vector of any point on the to-be-monitored beam structure after deformation; and S5, outputting the obtained spatial position vector of the to-be-monitored beam structure after deformation as a monitoring result, and providing the monitoring result to a structure instability risk early warning program.

The invention relates to an industrial video anomaly recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting a multi-view video, sensor data and equipment operation data into a visual language model, obtaining a depth feature representation, carrying out the multi-modal depth fusion of the depth feature representation through an industrial equipment knowledge graph, and obtaining an industrial equipment anomaly recognition result. A multi-modal fusion sequence is obtained; performing global context information enhancement on the features of the multi-modal fusion sequence through a space-time diagram to obtain final feature representation; performing video anomaly detection on the final feature representation, and outputting a video anomaly detection result; multi-view videos, sensor data and equipment operation data are integrated, multi-modal feature extraction is realized by using a visual language model, deep fusion is performed in combination with an industrial equipment knowledge graph, global context information is enhanced by means of a space-time diagram, the problem of incomplete feature representation of a traditional single-modal method is effectively solved, and the method is suitable for large-scale popularization and application. The method has the advantage that the industrial video anomaly detection accuracy and the system robustness are improved.

This invention discloses a frequency-domain modal method for determining the stability of vehicle-to-grid oscillations in new energy systems, applicable to converter impedance measurement. Specifically, it involves: acquiring the topology and component parameters of an electrified railway traction power supply system; performing frequency scanning on the single-input single-output frequency-domain impedance obtained through measurement or mathematical modeling in the frequency band to be analyzed, forming the system frequency-domain node admittance matrix at each frequency; performing eigenvalue decomposition to obtain the amplitude-frequency curve of the modal admittance or modal impedance; determining the oscillation frequency based on the extreme values ​​of the curve; determining the system stability based on the real and imaginary parts of the modal admittance or impedance at the oscillation frequency; and calculating the participation factor of each node in the unstable oscillation mode. This invention is applicable to low-frequency oscillation stability analysis of traction power supply systems with arbitrary topologies that do not contain three-phase converters. The SISO frequency-domain impedance of the single-phase voltage source converter interface power supply and locomotive load can be directly measured through impedance scanning without the need for dq-domain conversion.

The invention relates to the technical field of engineering machinery, and provides a pipeline management method based on modal displacement and electronic equipment, a multi-body dynamics-fluid pressure coupling simulation model is constructed based on a hydraulic excavator working device pipeline system, and target position parameters of a lug are obtained through a free modal method and a modal displacement algorithm. The multi-mode superposition effect principle is applied to layout optimization of the engineering machinery pipeline lug, accurate optimization of the position of the pipeline lug is achieved by modeling and analyzing the coupling vibration problem of a multi-order mode, and the problem of stress concentration caused by resonance is effectively avoided. Furthermore, actually-measured acquisition data are acquired through load spectrum acquisition, dynamic response analysis is performed to execute dynamic monitoring and early warning, and a dynamically-optimized and continuously-monitored closed-loop system is established through comparative analysis of simulation and actually-measured data, so that the simulation precision is improved, dynamic adjustment and early warning of actual working conditions are realized, and the working efficiency is improved. And the applicability and the reliability of the system are obviously improved.