16846 results about "Multi modal data" patented technology

The invention relates to the technical field of intelligent operation and maintenance, and discloses an intelligent operation and maintenance method fusing multi-modal data and active learning, and the method comprises the steps: deploying a hierarchical Internet of Things equipment network in a target operation and maintenance region, collecting a heterogeneous operation and maintenance data set, and generating an operation and maintenance feature data set through employing a unified building operation and maintenance service framework cooperating with a plurality of MCPs; inputting the operation and maintenance feature data set and the user feedback information into a deep learning intention recognition model for intention recognition and demand analysis to obtain structured user demand data and an operation and maintenance task priority sequence; according to the operation and maintenance feature data set and the structured user demand data, performing real-time evaluation on the equipment operation state to obtain fault risk early warning data; autonomous decision analysis is carried out through an agent type artificial intelligence engine, an equipment maintenance scheme and a resource scheduling scheme are generated, and then the problems that in traditional operation and maintenance, fault prediction is single, the model adaptability is poor, and prediction results are difficult to convert into effective decisions are solved.

The invention provides an industrial environment monitoring and accident prediction method fusing multi-modal data, and relates to the technical field of data processing, and the method comprises the steps: carrying out the semantic collection and causal association preprocessing of multi-modal heterogeneous data collected in real time through constructing a dynamic industrial knowledge graph; a customized deep learning model is adopted to extract deep abstract features of each mode, and weak signals and potential risks are accurately represented and uncertainty is quantified; a high-fidelity digital twin model is utilized to drive a deep reinforcement learning algorithm, and dynamic optimization and verification are performed to generate a multi-level and multi-target preventive intervention strategy combination; an intervention strategy is executed through an edge-end-cloud three-layer collaborative intelligent architecture, and online learning and system sustainable evolution are realized by using a closed-loop data feedback mechanism. According to the method, the sensing and early warning capability of the early weak and complex abnormal state of the industrial environment can be remarkably improved, the accident evolution path is accurately predicted, and credible explanation is provided.

The invention belongs to the technical field of multi-modal data processing, and particularly relates to a multi-modal document retrieval enhancement generation method based on a large model, which comprises the following steps: receiving query content input by a user for a multi-modal document; processing the query content by adopting an embedded model, generating a query vector representing user query semantic information, and storing the query vector in a vector database; analyzing the multi-modal document to obtain long text information, segmenting the long text information into data blocks by adopting a recursive partitioning strategy, and numbering and marking the data blocks; carrying out vectorization processing on the data blocks by adopting an embedded model to generate document vectors, storing the document vectors into a vector database, and constructing a hierarchical index structure; retrieving in a vector database based on the query vector, and returning a retrieval result; and processing a retrieval result by utilizing a large language model to generate response content conforming to the query intention of the user. According to the method, the multi-modal document can be effectively analyzed and processed, and the accuracy and comprehensiveness of analysis are improved.

The invention provides a home abnormal state signal detection method and system based on multi-modal sensing, and relates to the technical field of detection, and the method comprises the steps: collecting human body motion, acoustics and environment parameters through a millimeter wave radar, an acoustic sensor and an environment sensor, inputting the parameters into a deep fusion network, a spatio-temporal attention mechanism and a multi-scale convolutional neural network are used to extract a spatio-temporal feature sequence, and joint probability features are formed in combination with an acoustic feature spectrum analysis result. And inputting the joint probability features into a time sequence knowledge graph, and obtaining scene adaptive features after processing by a graph attention network and a double-flow auto-encoder network. And finally, the multi-task learning network is combined with the risk propagation neural network and the recursive neural network to carry out anomaly prediction and risk level evaluation, and is combined with historical early warning information to output an early warning strategy. According to the invention, multi-modal data can be effectively fused, the accuracy and reliability of home abnormal state detection are improved, and a more accurate risk assessment and early warning strategy is provided.

The invention discloses a system and a method for task planning of an intelligent robot with a body based on multi-dimensional situation awareness, and particularly relates to the technical field of task planning of the intelligent robot with the body, space-time alignment is carried out on asynchronous heterogeneous data generated by a multi-modal sensor channel, and cross-modal space-time features are extracted through a cross-modal feature fusion network; a fusion situation matrix is generated, dynamic causal modeling is used to update association strength among the multi-modal data, an anti-factual reasoning engine is used to identify and trace abnormities, and an abnormities traceability result is output; dynamically adjusting the reliability weight of each sensing channel through a multi-modal credibility evaluation model by using the fusion situation matrix and an abnormal traceability result; and on the basis of the reliability weight, inputting the fusion situation matrix into a real robot dynamic model and a digital twin virtual model, executing collaborative predictive control, and starting an adaptive rule evolution mechanism when a safety score is lower than a threshold value, thereby solving the problem of fusion matrix distortion in dynamic obstacle avoidance and precise grabbing tasks.

A decentralized event detection and geolocation system is disclosed, utilizing a dynamically adaptable Spontaneously Emergent Geolocation Network (SEGNet) comprising mobile, stationary, and other devices equipped with multi-sensor capabilities. The system operates by obtaining sensor data from distributed devices, applying trained models to detect potential events of interest (EOIs), and dynamically forming device networks for collaborative geolocation and event validation. The system employs synchronization mechanisms and geolocation techniques, including Time Difference of Arrival (TDOA) and / or Angle of Arrival (AoA), to determine event locations through multimodal data fusion and iterative validation processes. The architecture supports multiple operational modes, including decentralized, centralized, and hybrid configurations, enabling operation in connectivity-limited environments and enhanced processing when server access is available. The system is designed to accommodate dynamic task delegation and real-time threshold adjustments based on environmental conditions, providing reliable event detection and geolocation across diverse scenarios.

The invention belongs to the technical field of bridge diagnosis, and discloses a bridge crack intelligent diagnosis system based on multi-modal data fusion. By fusing multi-source data such as visual images, sound wave detection and vibration signals, comprehensive perception and characterization of crack features are realized; constructing a bridge crack characteristic spectrum diagram by adopting a cross-modal feature extraction and heterogeneous feature coding technology; generating a crack evolution situation map based on space-time correlation analysis and knowledge graph construction; the robustness of the system in a complex environment is improved through environmental adaptability feature enhancement and multi-scale characterization; constructing a bridge safety risk hypergraph in combination with multi-dimensional risk analysis and multi-agent collaborative diagnosis; analyzing and revealing a crack evolution mechanism by applying a causal relationship; and finally, through dynamic fusion and uncertainty quantification, a crack intelligent diagnosis comprehensive report is generated. According to the system, the limitation of traditional single-mode diagnosis is broken through, and dynamic prediction and accurate risk assessment of fracture evolution are realized.

The present invention provides a system and method for causality-augmented generative intelligence capable of autonomously discovering non-obvious actionable insights from heterogeneous and multimodal data sources. The system integrates a data ingestion unit for semantic and temporal harmonization of structured and unstructured datasets, a causal inference processor for constructing a dynamically evolving directed causal knowledge representation using perturbation-based validation, a latent representation processor that combines multimodal semantic embeddings with causal parameters to generate fused latent vectors, and a generative insight processor utilizing causally constrained generative reasoning to synthesize hypotheses anchored to verified cause-effect dependencies. A validation processor performs counterfactual assessment and observational verification to ensure retention of only those insights that remain consistent with causal ground truth.

The invention relates to a knowledge graph construction method and system based on a large language model, and the method and system achieve the automatic construction and dynamic maintenance of a knowledge graph through multi-modal data fusion, reinforcement learning and comparative learning joint optimization, teacher-student model knowledge migration, time sequence dynamic analysis and an incremental updating mechanism. Constructing a reinforcement learning framework, and taking accuracy and integrity as reward indexes to train a large model to extract an entity relationship; a large-scale knowledge graph is used as a teacher model, and conflict resolution and semantic alignment of newly added knowledge and an existing graph are realized through a graph attention network; a verification rule is dynamically generated based on historical data and domain knowledge, and relation periodicity and mutation points are detected in combination with Fourier transform and a CUSUM algorithm; and finally, generating a traceable knowledge graph through incremental updating and version control. And the multi-modal data processing precision, the entity relationship extraction dynamic adaptability and the knowledge graph maintenance efficiency are improved.

The invention discloses a multi-element sales planning agent system and method, and aims to improve the intelligence and precision of sales planning. The system comprises a collection module, an analysis module, an optimization module, a creation module and a generation module. The collection module is used for receiving multi-modal data such as marketing targets and extracting key marketing elements. The analysis module is used for generating a target user portrait and extracting marketing strategy analysis data. And the optimization module is used for calculating a medium putting weight by utilizing reinforcement learning and generating a medium strategy scheme. And the creation module generates a propagation theme and marketing content by adopting a generative artificial intelligence technology. And the generation module predicts a delivery effect by using a machine learning model and dynamically optimizes a medium strategy and a content scheme. Through multi-modal data fusion, intelligent analysis and optimization, closed-loop processing from data acquisition to marketing execution is realized, the marketing decision-making efficiency is improved, and brand promotion accuracy and market adaptability are enhanced.

The invention discloses an agent autonomous decision control method based on multi-modal data fusion. The method comprises the following steps: S1, synchronously collecting multi-source heterogeneous data; s2, dynamic weight adaptive fusion is carried out; s3, generating a task-driven decision; and S4, performing autonomous decision closed-loop optimization. According to the method, through dynamic weight distribution and space-time correlation modeling, the problems of heterogeneity and environment adaptation in multi-modal data fusion are solved; furthermore, a risk-sensitive reinforcement learning framework and a closed-loop feedback mechanism are combined, so that full-link cooperative control from data fusion, strategy generation to optimization execution is realized. In the mechanism level, the method breaks through the limitations of static fusion, single-target optimization and offline training, can adapt to a dynamic environment, ensures that the intelligent agent is in a complex scene such as noise interference, illumination abrupt change and task emergency switching, and meets the requirements of decision-making efficiency, safety and environment robustness at the same time.

The embodiments of the present disclosure belong to the technical field of data processing. Provided is a multimodal scenario risk determination method based on a generative AI large language model. The method specifically comprises: step 1, acquiring multimodal data to form a target data set, wherein the multimodal data comprises visual data and text data; step 2, using an ALBEF algorithm to extract key features corresponding to the target data set, and fusing the key features into a comprehensive scenario representation; and step 3, on the basis of a preset safety index and a large language model, evaluating a risk degree corresponding to the comprehensive scenario representation, comparing the risk degree with a risk threshold, and determining whether the scenario corresponding to the comprehensive scenario representation is a high-risk scenario. By means of the solution in the present disclosure, a high-risk scenario can be rapidly recognized and identified, so as to provide a basis for taking emergency measures, thereby enhancing the real-time response capability.

The invention relates to the technical field of multi-modal data processing, and discloses a multi-modal sensor fusion inspection method and system, and the method comprises the steps: collecting the multi-modal original data of power equipment through a multi-modal sensor in an inspection robot, and constructing a feature vector set; performing adaptive weight calculation on the multi-modal sensor according to the feature vector set to obtain a sensor weight set; carrying out conflict identification and resolution on the multi-modal original data to obtain a fusion data set; performing abnormal feature extraction on the power equipment based on the fused data set to obtain an abnormal feature set; and carrying out routing inspection trajectory optimization based on the abnormal feature set to obtain a target routing inspection path sequence, and carrying out equipment state joint prediction in combination with historical equipment routing inspection data to obtain an equipment fault prediction result. And thus, more accurate equipment state joint prediction is realized.

The invention relates to the technical field of energy scheduling, in particular to a multi-modal data fusion method and system based on energy scheduling and a storage medium. The method comprises the following steps: obtaining multi-modal data, and carrying out abnormal fluctuation feature extraction to obtain a space-time fusion abnormal feature labeling set; deducing a multi-objective optimization path according to the space-time fusion abnormal feature labeling set to obtain a dynamic scheduling decision map; performing edge node game equilibrium calculation according to the dynamic scheduling decision map to obtain a trusted scheduling verification chain; performing digital twinborn constraint optimization on the trusted scheduling verification chain to obtain a closed-loop scheduling digital twinborn body; compiling a dynamic scheduling instruction set based on the closed-loop scheduling digital twin to obtain an anti-disturbance energy scheduling strategy library; and obtaining real-time energy supply and demand data, and performing scheduling deviation tracing on the real-time energy supply and demand data according to the anti-disturbance energy scheduling strategy library to obtain an energy distribution decision. According to the invention, the efficiency and reliability of energy scheduling can be improved.

The invention discloses a trajectory optimization method based on a multi-source heterogeneous positioning data fusion algorithm, and relates to the technical field of intelligent navigation and high-precision positioning, multi-modal data are acquired through a multi-modal sensor array, positioning redundancy of scenes such as tunnels and indoor scenes is enhanced, a weight distribution strategy is dynamically adjusted through an Actor-Critic network architecture, and the positioning accuracy is improved. The state space input comprises an environment semantic tag, a historical error sequence and a real-time noise variance, the output action space is continuous weight distribution of each data source, a multi-target reward function optimization strategy is combined, scene adaptability is realized, a local SLAM map, inertial navigation error parameters and a weight distribution strategy are shared in real time based on a V2X protocol, and the real-time performance of the system is improved. According to the method, a single device accumulative error is compensated by using adjacent vehicle data, a terminal locally trains an error compensation model, parameters are uploaded to a cloud end through differential privacy encryption, the cloud end adopts a FedAvg algorithm to aggregate a global model and issue the global model, the error difference between devices is inhibited, and dynamic road network updating and scene differentiation model distribution are supported at the same time.

The present invention relates to a railway anomaly detection method and system based on multi-modal data fusion. The method comprises: separately encoding acquired modal data in a railway environment, and concatenating encoded modal data features, wherein the modal data comprises a one-dimensional vibration signal, two-dimensional image data, and 3D point cloud information; performing automatic weight classification on the concatenated multi-modal data features on the basis of an attention mechanism to obtain a weighted feature vector fused with multi-modal information; and adding a location code to the feature vector and using same as an input of an SAM encoder to obtain a segmentation result, and determining an anomaly condition of a railway on the basis of the segmentation result. The present invention achieves high monitoring accuracy, a real-time response capability, and specificity.

The invention relates to the field of semiconductor packaging material defect detection, in particular to a semiconductor packaging material defect detection method based on deep learning, which comprises the following steps: acquiring a surface image, and extracting a two-dimensional contour and a feature point set; preprocessing the image, and separating a packaging material main body area; constructing a two-dimensional defect identification model based on Transform, and outputting a two-dimensional detection result; scanning suspected and unknown defect areas to obtain three-dimensional point cloud data, and extracting geometric and texture features; fusing two-dimensional and three-dimensional data through a space-time alignment model; utilizing the multi-modal fusion model to output defect positions and types; and evaluating the defect importance based on the material node connectivity and the stress distribution, and generating a visual detection report. According to the invention, high-precision detection of semiconductor packaging material defects is realized, the defect identification rate, the positioning precision and the detection efficiency are improved through multi-modal data fusion and a deep learning model, and a visual report can be generated based on material structure quantification defect importance.

Multi-modal models learn a joint latent space for relating data points across different modalities. To more effectively learn multi-modal models with reduced training requirements and greater benefit from limited multi-modal training data, a multi-modal model may be trained with fixed or pre-trained unimodal encoders that generate data representations in respective latent spaces. The multi-modal model is trained to learn a shared latent space while fixing the unimodal encoders, enabling training without storing the unimodal encoders in memory. Limited multi-modal data may also be augmented by generating synthetic data between commonly-labeled pairs in the respective modality's latent spaces. The effect of data diversity can also be determined by generating a diverse data set with respect to the data points in latent space, enabling measurement of performance of the multi-modal model on limited training data.

The invention relates to the technical field of LED display, solves the problem that the existing LED display defect detection and parameter adjustment technology is lack of multi-modal information fusion and intelligent process control capability and is difficult to meet the quality control requirement of a high-precision display product, and provides an LED display defect prediction and process adjustment method and system based on multi-modal fusion. The method comprises the following steps: performing multi-modal data fusion processing on optical image data, electrical test data and thermal infrared imaging data corresponding to a to-be-tested LED display screen to obtain fused data; inputting the fused data into a pre-trained defect recognition model to obtain a defect recognition result; according to a process parameter adjustment strategy corresponding to the defect identification result, adjusting the original process parameter to obtain a target process parameter; and according to the target process parameters, process flow correction processing is carried out, and a qualified LED display screen is produced. According to the method, the defect identification precision is improved, and the quality control requirement of high-precision LED display screen production is met.

The invention relates to a network public opinion intelligent classification and emergency decision-making system based on multi-modal fusion and dynamic evolution, and belongs to the field of network public opinion monitoring and big data analysis and artificial intelligence. The system comprises a multi-source data acquisition and preprocessing module used for crawling multi-modal data, constructing a propagation path map after preprocessing, and identifying key propagation nodes; the multi-dimensional classification engine module is used for carrying out conflict intensity quantification on public opinion events and dynamically updating a rule word bank to keep the adaptability of a conflict intensity quantification model; the event graph construction and anomaly detection module is used for constructing a public opinion propagation path and public opinion event generality logic chain mode, monitoring public opinion propagation speed and giving an alarm; the stakeholder dynamic risk assessment module is used for finely classifying network public opinion participants, providing a basis for differential propagation intervention and simulating public opinion evolution to carry out risk simulation; and the intelligent decision-making and emergency response module executes different levels of emergency measures based on the risk index according to the hierarchical response strategy.

The invention discloses an equipment intelligent safeguard system based on an offline large model, and relates to the field of equipment management and safeguard, and the system comprises a collection module which collects an equipment surface defect image and a time sequence signal through an unmanned plane, collects a text log, processes the defect image, the time sequence signal and the text log, and generates a multi-modal data set; the feature fusion module is used for performing feature extraction on the multi-modal data set and fusing the three features to generate a multi-modal feature; the updating module is used for extracting a fault triple from the semantic vector, constructing a Bayesian causal graph according to the fault triple, dynamically updating node confidence and generating a knowledge graph; the maintenance strategy module is used for matching the multi-modal features with a knowledge graph and positioning a fault root cause; and the lightweight model deployment module is used for pre-training a lightweight model in an edge server, accelerating parameter aggregation by using a quantum annealing algorithm, adjusting and optimizing a global large model according to the aggregated parameters, and generating a strategy in combination with reinforcement learning.

The invention discloses an AI agent design platform decision optimization method based on big data, and particularly relates to the field of artificial intelligence, comprising multi-modal data sensing layer construction, a streaming feature calculation engine, a dynamic index fusion center and an adaptive decision matrix. According to the method, accurate synchronous monitoring of the utilization rate of hardware resources and dynamic collaborative optimization of heterogeneous computing units are achieved, and the resource scheduling efficiency in a complex computing scene is remarkably improved; knowledge system degradation caused by long-term learning is effectively prevented, and the continuous reliability of a cognitive system is ensured. The provided multi-dimensional decision credibility verification system is fused with interpretability penetration analysis, environment coupling modeling and logic drift detection technologies, the limitation of a traditional single credibility index is broken through, the risk prediction and fault-tolerant capability of the decision process is remarkably enhanced, and a full-dimensional safety decision guarantee system is constructed for an intelligent agent.

The invention provides a multi-agent task collaboration method, device and equipment and a storage medium, and the method comprises the steps: carrying out the deep fusion and unified semantic coding of a collected multi-modal data set through a multi-modal large language model, and obtaining a high-dimensional cross-modal feature embedding and semantic representation file, the task requirement mapping module is used for enabling local task requirements of multiple agents to correspond to cross-modal semantics to obtain task requirement semantic mapping, and task division and time arrangement are carried out; when multiple agents execute tasks, key data and operation results are sampled in real time and compared with high-dimensional semantic representation, a concept offset detection result is obtained, and when it is detected that the concept drifts progressively, the multi-modal large language model is dynamically adjusted. According to the method, the communication and cooperation efficiency among multiple agents is enhanced by using a large language model, and task allocation and collaborative decision are optimized through task demand semantic mapping; and concept drift detection and a dynamic adjustment mechanism are introduced, so that the long-term adaptability in a complex dynamic environment is improved.

The invention relates to the technical field of poultry breeding monitoring, and discloses a chicken flock state inspection monitoring system and method. The method comprises the following steps: firstly, collecting chicken flock visual images, sound signals, environment temperature and humidity and individual activity track data, and constructing a multi-modal data set through labeling and preprocessing; different modal data features are extracted and fused; training a self-supervised contrast learning model to generate a discrimination model, and optimizing hyper-parameters in combination with a genetic algorithm; collecting data in real time, calculating a health state probability, generating an abnormal score, and dynamically updating an early warning threshold value; and if the abnormal score exceeds a threshold value, grading early warning and abnormal positioning are carried out. The system comprises a data acquisition and preprocessing module, a feature extraction module, a feature fusion module and the like. According to the method, the health of the chicken flocks is accurately monitored by using multi-modal data, dynamic early warning and model adaptive optimization are realized, the breeding benefits are improved, and the disease risk is reduced.

The invention provides an intelligent rehabilitation training method and system based on artificial intelligence and virtual reality, a user wears an intelligent wearable device to collect multi-modal data such as electroencephalogram, myoelectricity, physiological features and motion signals, the multi-modal data is preprocessed and then input into an artificial intelligence training model, and key features are extracted and fused by using a graph convolutional network and an attention mechanism algorithm. And constructing a digital twinborn model by using the fusion features, performing real-time dynamic mapping and predictive simulation, and generating a customized training scheme by means of a reinforcement learning algorithm in combination with a rehabilitation target and a physical state of the user. A user is trained in the virtual reality interaction model, the system monitors actions and physiological states in real time, the digital twin model synchronously acts, and the scene is dynamically adjusted. After training, the rehabilitation effect is evaluated according to the physiological indexes, the motion data and the twinning optimization analysis result, and an optimization training scheme and a digital twinning model are fed back. Precision, individuation and intelligentization of rehabilitation training are achieved, and the training effect and quality are improved.

Multi-modal models learn a joint latent space for relating data points across different modalities. To more effectively learn multi-modal models with reduced training requirements and greater benefit from limited multi-modal training data, a multi-modal model may be trained with fixed or pre-trained unimodal encoders that generate data representations in respective latent spaces. The multi-modal model is trained to learn a shared latent space while fixing the unimodal encoders, enabling training without storing the unimodal encoders in memory. Limited multi-modal data may also be augmented by generating synthetic data between commonly-labeled pairs in the respective modality's latent spaces. The effect of data diversity can also be determined by generating a diverse data set with respect to the data points in latent space, enabling measurement of performance of the multi-modal model on limited training data.

The invention belongs to the technical field of industrial equipment monitoring, and discloses an industrial production equipment monitoring and early warning system based on the Internet of Things and edge intelligence, and the system collects multi-modal data in real time through a distributed sensor network, and carries out the real-time processing and feature extraction through the edge intelligence technology. Data fusion and optimization are carried out through a cloud deep learning model, a self-adaptive early warning and decision module dynamically adjusts a threshold value and optimizes an operation strategy, an environmental adaptability optimization module ensures the stability of equipment under extreme conditions, and an equipment health management and collaborative maintenance module realizes cross-equipment collaborative maintenance. The remote monitoring and interaction module provides man-machine interaction type control, and the cross-device collaborative learning module improves the model performance through federal learning. According to the system, the accuracy of equipment monitoring, the timeliness of early warning and the intelligent level of maintenance are remarkably improved.

The invention provides a gas leakage time-space correlation early warning method and system based on multi-modal data fusion, and relates to the technical field of data processing, and the method comprises the steps: collecting real-time concentration data and a historical leakage accident database through an NB-IoT combustible gas sensor network disposed at a key node of a gas pipe network; according to the real-time concentration data and a historical leakage accident database, a UTM projection coordinate system is adopted to carry out registration on geographic space data, a three-dimensional space mapping model of a pipe network topological structure is established, timestamp alignment is carried out on all dynamic monitoring data streams through an NTP clock synchronization protocol, and a time-space correlation standardized data set is formed. According to the invention, full-chain intelligent management of the gas leakage risk from sensing, prediction to disposal is realized.

The invention discloses a PCBA circuit board welding spot detection method based on multi-modal data fusion, and relates to the technical field of electronic manufacturing quality detection.The PCBA circuit board welding spot detection method comprises the steps that a distributed sensing network is constructed, multi-modal data are collected, welding spot information is obtained in an omnibearing mode, and time-space alignment of the multi-modal data is carried out; performing feature extraction on the multi-modal data, dynamically weighting each modal feature through an attention mechanism, and highlighting key defect characterization; a welding spot spatial topological graph is constructed by using a graph neural network, and a spatial relationship between welding spots is modeled. By integrating optical, X-Ray, thermal, mechanics, electricity and other multi-dimensional data, the information limitation of single-mode detection is broken through, the complementation of different mode data is utilized, the attention mechanism is combined to dynamically weight each mode feature, the complex defect is accurately identified, the graph neural network is utilized to model the welding spot space topological relation, the associated defect is further captured, and the defect detection accuracy is improved. And the defect classification accuracy is improved.

The invention discloses an archive knowledge base construction and retrieval method and system based on multi-modal data fusion. The method comprises the steps that heterogeneous archive data are cleaned, image features are extracted through CNN, text features are extracted through Transform, audio is converted into text and then subjected to similarity, a unified feature vector is generated, and metadata is constructed according to archive code association; creating a graph database instance, defining nodes and relationship types, importing entities and relationships, and storing feature vectors and metadata; the features are mapped to a high-dimensional shared semantic space, positive and negative sample pairs are constructed to update embedded layer parameters, self-attention is used in modalities, a shared attention mechanism is used between modalities, weights are adjusted according to archive features, and unified knowledge representation is generated; segmenting the steering quantity of the multi-modal data, storing the steering quantity into a database, and adopting hierarchical indexing and optimizing as required; related document fragments are retrieved through RAG technology vectors, answers are generated with the help of a large language model, and session feedback is provided. The file retrieval efficiency and accuracy are improved.