35218 results about "Multi modality" patented technology

The invention discloses a power transmission and distribution production task cooperation system and method based on an intelligent agent, and relates to the technical field of power distribution production task scheduling, the system comprises six modules: a natural language input interaction module processes a user instruction and multi-modal information, and generates structured data; the electric power field knowledge enhancement analysis module establishes mapping from a natural language to business data; the dynamic interaction context memory module stores historical interaction data and generates a context feature vector through a bidirectional LSTM and an attention mechanism; the intelligent task scheduling and conflict resolution module is used for disassembling instructions into sub-tasks, dynamically evaluating priorities in combination with three-dimensional indexes and resolving resource conflicts; the agent task execution and cooperation module drives agents to execute tasks according to priorities and synchronize states in real time; the system closed-loop feedback optimization module analyzes the execution log and automatically updates model parameters; according to the system, the problems of term analysis deviation, strategy staticization and insufficient self-optimization capability of a traditional scheduling system are solved.

The invention provides a medical decision-oriented multi-modal data dynamic fusion and labeling method and system, and relates to the technical field of medical decision-making, and the method comprises the steps: collecting medical images, electronic medical records and physiological monitoring data of a patient, extracting multi-modal features such as a focus form, a disease type, a symptom, a treatment record and a physiological index, and carrying out the fusion and labeling of the multi-modal features; and constructing a dynamic fusion labeling model. According to the model, multi-modal features are coded, similarity is calculated, a feature association graph is constructed, feature aggregation is carried out, labeling is carried out in combination with a medical knowledge base, and a dynamic labeling vector is generated. And inputting the dynamic labeling vector into a medical decision support model, and based on similar case analysis and deep reinforcement learning, generating a diagnosis and treatment suggestion containing a treatment scheme and early warning information. By dynamically fusing multi-modal data and an online updating mechanism, the accuracy and efficiency of medical decision making can be improved, and personalized diagnosis and treatment are realized.

The invention relates to a dynamic data pipeline construction method based on artificial intelligence and multi-modal data processing, and belongs to the technical field of data fusion treatment and artificial intelligence. According to the method, a multi-modal deep learning model is adopted, semantic analysis is performed on business documents, user requirements and function specification instructions, and key indexes, data requirements and logic constraints are extracted; constructing a multi-modal knowledge graph, performing unified representation and feature extraction on structured and unstructured data resources, and performing embedded learning by using a graph neural network; based on a deep semantic matching model and a hybrid recommendation system, realizing accurate matching of business requirements and data resources; modeling generation of a data processing flow into a Markov decision process by adopting a reinforcement learning strategy, and automatically generating an optimized data processing workflow; the adaptability of the model to new business requirements is improved by utilizing a meta-learning algorithm; and dynamic scheduling and execution of tasks are realized through a workflow engine. According to the invention, the data processing efficiency can be improved.

The invention provides an industrial environment monitoring and accident prediction method fusing multi-modal data, and relates to the technical field of data processing, and the method comprises the steps: carrying out the semantic collection and causal association preprocessing of multi-modal heterogeneous data collected in real time through constructing a dynamic industrial knowledge graph; a customized deep learning model is adopted to extract deep abstract features of each mode, and weak signals and potential risks are accurately represented and uncertainty is quantified; a high-fidelity digital twin model is utilized to drive a deep reinforcement learning algorithm, and dynamic optimization and verification are performed to generate a multi-level and multi-target preventive intervention strategy combination; an intervention strategy is executed through an edge-end-cloud three-layer collaborative intelligent architecture, and online learning and system sustainable evolution are realized by using a closed-loop data feedback mechanism. According to the method, the sensing and early warning capability of the early weak and complex abnormal state of the industrial environment can be remarkably improved, the accident evolution path is accurately predicted, and credible explanation is provided.

The invention discloses an auditing decision support system and method based on a dynamic knowledge graph, relates to the technical field of computers, and aims to solve the problems that auditing data are heterogeneous and complex, risk identification is not timely and causal interpretation is lacked. According to the system, multi-modal audit data is collected in real time through a streaming event processing framework, and a dynamic audit knowledge graph with timeliness weight is constructed. Based on a graph calculation engine and cross-domain rule mining, identifying a high-frequency risk mode, and generating a risk conduction path graph; further fusing a multi-modal graph attention network, identifying and positioning abnormal entities, and outputting abnormal nodes and risk links thereof; and finally, the abnormal node embedding representation is dynamically updated through the time sequence diagram attention network, an interpretable audit causal map is generated in combination with a structural causal model, and closed-loop support from data acquisition and risk identification to interpretive audit decision is realized. The intellectualization and transparency of audit decision making are improved, and an efficient and traceable decision making basis is provided for a complex audit scene.

The invention discloses a small unmanned aerial vehicle inspection data management method, relates to the technical field of unmanned aerial vehicle inspection, and is used for solving the problem that unmanned aerial vehicle inspection data management is not clear. According to the invention, through a high-precision positioning technology and timestamp synchronization, multi-modal data collected by an unmanned aerial vehicle is associated with an equipment position, space-time consistency of inspection data is realized, multi-modal feature fusion and dimension reduction processing are carried out, unified feature representation is generated, the precision and efficiency of data analysis are improved, historical data and real-time collected data are combined, and the accuracy and efficiency of data analysis are improved. Equipment states are monitored in real time, fault modes are identified and classified, risks are evaluated, fault development trends are predicted through a time sequence prediction model, maintenance suggestions are generated, the timeliness of fault diagnosis is improved, based on fault detection results, routing inspection task priorities and path planning are dynamically optimized, an optimal routing inspection path is generated, and a flight path is corrected; the intelligence and the precision of the inspection task are improved, and the efficiency and the safety of the unmanned aerial vehicle inspection system are remarkably improved.

A network mapping behavior anomaly detection method and system based on machine learning is provided. The method includes: collecting dual-source traffic data, generating a structured log data set through dual-source log fusion engine; performing subgraph matching calculation to obtain a mapping behavior deviation degree; generating communication data containing a watermark identifier in a session corresponding communication path; verifying whether attack events carry the watermark identifier; generating a network mapping behavior anomaly detection report. According to the disclosure, an adaptive attack behavior model is constructed through a multi-modal feature vector based on structured logs and a graph protocol mapping rule base, so that the cognitive robustness to protocol camouflage and path drift is fundamentally enhanced, a real-time verification chain of detection results is built, and traditional passive detection is transformed into self-proof active defense through cross verification of watermark carrying state and behavior trajectory.

The invention relates to the technical field of industrial automation control, and discloses a multi-machine collaborative industrial robot intelligent scheduling system and an application method.The system comprises a data acquisition module for acquiring real-time motion data through a multi-modal sensor; the feature fusion module is used for generating environment dynamic feature data based on graph neural network fusion data; the strategy generation module is used for generating collaborative strategy parameters by utilizing a collaborative reinforcement learning model; the trajectory optimization module is used for constructing a multi-target trajectory optimization model to plan an optimal synchronous trajectory; the motion control module is used for establishing a layered motion control model to realize cooperative motion control; a dynamic resource allocation module can also be included to optimize resource allocation. The application method sequentially executes corresponding steps according to the modules of the system. According to the method, the environment can be accurately sensed, the cooperation strategy can be optimized, the efficient track can be planned, accurate motion control and reasonable resource allocation can be realized, and the cooperative motion control capability and the production efficiency of the industrial robot in the complex environment can be effectively improved.

The invention discloses a decision-making method and model for off-line reinforcement learning and continuous on-line fine tuning, and the method comprises the steps: obtaining original text-image pair data and historical track data, and obtaining an enhanced state sequence through feature matching verification and standardization processing; performing sequence prediction by using a multi-level attention mechanism, and obtaining pre-training model parameters through knowledge distillation and task adaptability enhancement; on the basis of stability evaluation and plasticity control, online knowledge migration is carried out on the model, and optimized model parameters are obtained by combining experience playback optimization; and finally, performing multi-dimensional performance evaluation and resource optimization. According to the method, through multi-modal feature fusion, dynamic knowledge migration and an adaptive optimization mechanism, the problems of insufficient feature representation, disastrous forgetting, low optimization efficiency and the like are effectively solved, and the learning efficiency and generalization ability of the model are improved.

The invention provides a home abnormal state signal detection method and system based on multi-modal sensing, and relates to the technical field of detection, and the method comprises the steps: collecting human body motion, acoustics and environment parameters through a millimeter wave radar, an acoustic sensor and an environment sensor, inputting the parameters into a deep fusion network, a spatio-temporal attention mechanism and a multi-scale convolutional neural network are used to extract a spatio-temporal feature sequence, and joint probability features are formed in combination with an acoustic feature spectrum analysis result. And inputting the joint probability features into a time sequence knowledge graph, and obtaining scene adaptive features after processing by a graph attention network and a double-flow auto-encoder network. And finally, the multi-task learning network is combined with the risk propagation neural network and the recursive neural network to carry out anomaly prediction and risk level evaluation, and is combined with historical early warning information to output an early warning strategy. According to the invention, multi-modal data can be effectively fused, the accuracy and reliability of home abnormal state detection are improved, and a more accurate risk assessment and early warning strategy is provided.

The invention provides a notarization intelligent question and answer customer service method and system based on a knowledge graph, and relates to the technical field of information, and the method comprises the steps: collecting multi-source heterogeneous data in the notarization field, forming an entity set and a relationship set, combining to form an initial knowledge graph, carrying out a dynamic update mechanism based on time decay, and supplementing to form a dynamic knowledge graph. Carrying out vectorization expression to obtain an atlas vector; receiving a multi-modal problem, determining fusion features, performing deep semantic understanding, obtaining semantic features, determining a user intention, and forming a structured query expression; based on the user intention and the structured query expression, carrying out multi-hop reasoning in the dynamic knowledge graph, carrying out similarity calculation by utilizing graph vectors, determining a reasoning path in combination with the user intention, carrying out knowledge association reasoning along the reasoning path, obtaining a preliminary reasoning result, carrying out interpretability analysis on the preliminary reasoning result, and obtaining a final reasoning result; and generating a reasoning chain and a confidence score, forming a detailed reasoning result, and finally generating a personalized answer.

The invention belongs to the technical field of knowledge retrieval, and discloses a multi-modal large model-based dynamic vector knowledge base construction and retrieval method, which comprises the following steps of: obtaining a multi-source heterogeneous modal data set, and carrying out preprocessing and modal standardization processing on the multi-source heterogeneous modal data set to obtain a standardized multi-modal data set; performing feature extraction and semantic vector representation generation by using the pre-trained multi-modal large model, and constructing a multi-modal knowledge vector set; semantic association analysis and hierarchical clustering are carried out on the multi-modal knowledge vector set, and a structured vector knowledge base is constructed; performing semantic similarity calculation and relation modeling on the vector knowledge base to form a vector relation network; intention analysis and vector representation are performed based on mixed modal query information input by a user, and efficient similarity retrieval is realized in combination with a vector relation network; dynamic optimization is carried out through user feedback, personalized retrieval result adjustment is achieved, and the problem of limitation of a traditional retrieval system during multi-modal data processing is effectively solved.

The invention discloses an intelligent customer service system based on an AI large model, and belongs to the technical field of intelligent customer service systems. Through the multi-modal interaction technology, the flexibility of the system and the user experience are improved, the data type is intelligently recognized and correspondingly processed, the language style and emotional intonation of the response content are automatically adjusted according to the mood and intention of the user, the knowledge base integrates static data and dynamic interaction data, and the user experience is improved. Real-time updating and comprehensiveness of knowledge content are ensured, rapid retrieval is performed based on a knowledge base, accurate reply is performed in combination with a large language model, high-quality answers and solutions are generated through rule reasoning and model generation, state information of each round of interaction of a user is recorded and managed by constructing a dialogue state tracker, and the user experience is improved. And precise tracking and updating of the context are realized. For a complex task, the system splits a user request into a plurality of sub-tasks, and the sub-tasks are executed in sequence according to the priority and the dependency relationship, so that the high efficiency and continuity of task processing are ensured.

The invention discloses a system and a method for task planning of an intelligent robot with a body based on multi-dimensional situation awareness, and particularly relates to the technical field of task planning of the intelligent robot with the body, space-time alignment is carried out on asynchronous heterogeneous data generated by a multi-modal sensor channel, and cross-modal space-time features are extracted through a cross-modal feature fusion network; a fusion situation matrix is generated, dynamic causal modeling is used to update association strength among the multi-modal data, an anti-factual reasoning engine is used to identify and trace abnormities, and an abnormities traceability result is output; dynamically adjusting the reliability weight of each sensing channel through a multi-modal credibility evaluation model by using the fusion situation matrix and an abnormal traceability result; and on the basis of the reliability weight, inputting the fusion situation matrix into a real robot dynamic model and a digital twin virtual model, executing collaborative predictive control, and starting an adaptive rule evolution mechanism when a safety score is lower than a threshold value, thereby solving the problem of fusion matrix distortion in dynamic obstacle avoidance and precise grabbing tasks.

An intelligent pulmonary nodule grading method and system based on multi-modality feature fusion. The method comprises: using clinical multi-modality data of a physical examination crowd to acquire an ROI and a VOI of a pulmonary nodule on the basis of a chest CT examination image and an examination report, designing a multi-task feature extraction network based on an attention mechanism, and acquiring radiomics features and deep-layer image features from the ROI and the VOI; designing a cross-modality feature fusion method based on graph representation learning, designing a multi-modality information extraction method, acquiring a specific feature representation and graph structure of each modality, and then performing feature representation and graph structure fusion; and providing a pulmonary nodule grading GCN model optimization and clinical verification method based on self-supervised learning, so as to implement fine grading of pulmonary nodule malignancy having only slight differences. A new way is provided for the design of a fine-grained classification algorithm; and in the aspect of solving a clinical problem, provided is a classification algorithm optimization method and clinical verification scheme in which generative self-supervised learning and comparative self-supervised learning are organically combined.

The invention discloses a target identification tracking method and system based on multi-source fusion imaging, and the method comprises the steps: obtaining and preprocessing multi-modal environment monitoring data, carrying out the image enhancement based on the preprocessed multi-modal environment monitoring data, and obtaining the image-enhanced multi-modal environment monitoring data; performing multi-modal feature extraction on the image enhancement multi-modal environment monitoring data, and introducing a cross attention mechanism to fuse the extracted multi-modal features to construct cross-modal fusion features; constructing a cascade target recognition model, and inputting cross-modal fusion features to perform target recognition on the current monitoring scene; tracking a scene recognition target in the current monitoring scene to generate scene target tracking information, and storing the scene target tracking information in a preset track memory pool; and when a tracking target disconnection condition occurs during target tracking, obtaining a mismatched tracking trajectory, and performing tracking trajectory disconnection repair in combination with the trajectory memory pool. Therefore, the identification accuracy and tracking reliability of the target in the monitoring scene are improved.

The invention discloses a territorial space planning environment influence monitoring method and device. The method comprises the following steps: collecting and preprocessing multi-source environment monitoring data; performing spatio-temporal feature extraction on the preprocessed multi-source environment monitoring data by using a self-supervised spatio-temporal diagram mask transfer attention network to obtain initial spatio-temporal features; performing fusion processing on the initial spatial-temporal characteristics based on a multi-mode diffusion converter to obtain fusion characteristics; calculating an environmental impact evaluation index based on the fusion feature; and based on a preset visualization template and an early warning rule, generating a dynamic monitoring result including a space-time evolution diagram, a trend analysis diagram and early warning information according to the environmental influence evaluation index. Through deep learning and multi-modal data fusion technologies, the precision and efficiency of environmental impact monitoring are improved, and accurate evaluation and timely early warning of environmental impact in the land space planning implementation process are realized.

The invention belongs to the technical field of visual special effects, and discloses an AI visual special effect dynamic generation system fusing multi-modal perception. Through cooperative work of six core modules of multi-modal input processing, importance analysis, parameter configuration, parameterized rendering, parameter optimization and rendering and output, deep fusion of special effects and contents is realized. An image sequence, audio data and scene parameters can be processed at the same time, a multi-modal feature data set is constructed, a scene key visual area is scientifically recognized, a visual importance distribution map is generated, and the special effect parameter configuration and rendering process is guided. An improved neural radiation field technology and a physical constraint model are adopted to ensure the consistency and reality of the special effect at different visual angles; and a multi-dimensional quality evaluation and parameter automatic optimization mechanism is introduced, so that the visual expressive force and the artistic value of the special effect are guaranteed. The creation threshold is remarkably reduced, the cooperative expression ability of the special effect and the content is improved, and the special effect becomes a powerful tool for enhancing the narration and enhancing the emotion.

The embodiment of the invention provides a scene space three-dimensional model dynamic modeling method based on multi-modal data, and the method comprises the steps: fusing the feature information of a video sequence, point cloud data and IMU data, and building a space-time corresponding relation through employing a particle filtering algorithm. Scene semantic understanding is realized based on a Transform multi-modal scene analysis network, scene scores are calculated in combination with spatial density, depth occlusion and shape change features, and an optimal modeling algorithm is selected through a strategy network of deep reinforcement learning training. Meanwhile, scene dynamic features are analyzed by using a deep neural network and a graph neural network, adaptive grid subdivision and geometric constraint reconstruction of a dynamic region are realized, and a reconstruction threshold is dynamically adjusted according to a scene semantic type. According to the method, the three-dimensional reconstruction quality and efficiency of a large-scale complex scene are effectively improved.

The invention discloses a multi-modal AI-based power equipment anomaly detection method and system, and the method comprises the steps: synchronously collecting electrical, mechanical and thermal modal data of power equipment through an edge computing node, carrying out the load adaptive dynamic preprocessing, and uploading the data to a cloud end; the cloud constructs a multi-modal feature extraction network based on a structural causal model, analyzes a causal path between modals through a Bayesian network and performs weighted fusion on feature vectors; capturing device state mutation by using a gating attention mechanism, and updating the feature vector; executing time-space consistency verification of the equipment group to identify regional group abnormality and suppress single-point misinformation; generating an interpretable report containing an abnormal root cause analysis and priority ranking maintenance strategy; and establishing a closed-loop feedback mechanism to correct the cause and effect probability distribution of the Bayesian network model. The system comprises a multi-modal sensor array, an edge computing node and a cloud analysis platform, wherein the cloud analysis platform is integrated with a causal reasoning engine, a space-time consistency verification module and the like. According to the invention, by analyzing the multi-modal deep causal association, the method adapts to the dynamic change of the equipment, reduces the false alarm rate, generates an interpretable report, supports closed-loop self-optimization, and improves the anomaly detection accuracy and operation and maintenance decision efficiency of the power equipment.

The invention provides a live broadcast room content identification and intelligent distribution method and system based on multi-modal fusion, and relates to the technical field of machine learning, and the method comprises the steps: collecting video, audio and bullet screen data of a live broadcast room, and carrying out the feature extraction; performing cross-modal interaction, feature alignment and feature reconstruction on the multi-modal features by using the multi-modal feature fusion model to generate cross-modal fusion features; and carrying out content identification based on a pre-trained content classification model to obtain a category label and a time sequence feature. Furthermore, a live broadcast room content portrait and an association network are constructed, and a live broadcast room real-time score is generated and a live broadcast room display weight and a push strategy are dynamically adjusted in combination with user historical watching behavior data and a deep reinforcement learning model. Through multi-modal data fusion and deep reinforcement learning, accurate identification and personalized recommendation of the live broadcast room content are realized, and the user experience and the platform operation efficiency are effectively improved.

The invention discloses an aerospace intelligent manufacturing large model construction method, which comprises the steps of collecting original data, performing preprocessing and data association, and constructing an aerospace intelligent manufacturing database; establishing a knowledge acquisition and structured conversion assembly line, a multi-dimensional associated domain knowledge graph, a knowledge quality control system and a dynamic updating mechanism, and constructing a professional knowledge base; aligning the cross-modal manufacturing data to generate a corpus; combining base general large model pre-training, injecting terminology semantics and multi-modal association capability, and completing knowledge migration; based on the pre-trained aerospace intelligent manufacturing large model, constructing an aerospace manufacturing cognitive agent, and forming a complex engineering problem solving framework; professional ability is optimized through a two-stage progressive multi-task training strategy, and dynamic adaptation of a production environment is realized in combination with an online learning and incremental updating mechanism. The intelligent level of aerospace intelligent manufacturing is remarkably improved.

The invention discloses an exhibition hall robot visual language navigation method based on a large model, belongs to the field of exhibition hall robots, and aims at solving the problems existing in an existing exhibition hall robot. According to the method, an exhibition hall robot multi-mode scene sensing system senses a current scene, and candidate target objects are obtained through fusion features of RGB images and laser point clouds; extracting image features of all candidate target objects in the scene; according to the exhibition hall robot multi-mode environment understanding system, an exhibition hall robot receives a voice instruction, a visual language multi-mode large model is used for understanding a target object and a user intention, then high-dimensional semantic features of a target object text are extracted from the system, and the target object is found out from candidate target objects; according to the exhibition hall robot intelligent navigation system based on the multi-modal large model, an exhibition hall robot carries out path planning and autonomous movement according to a pose sequence generated by an intention, and self behaviors are dynamically adjusted according to a real-time changing scene provided by the exhibition hall robot multi-modal scene sensing system.

The invention relates to a business credit evaluation and supervision method based on a multi-modal coevolution algorithm, and the method comprises the steps: collecting and cleaning multi-modal data, guaranteeing the data quality, uniformly storing the multi-modal data as a structural database, extracting features through cross-modal comparative learning, and enabling the features to be embedded into a dynamic multi-modal knowledge graph, a modal sharing network is used to extract and fuse modal sharing and specific features, consistency of modal feature distribution is optimized through adversarial learning, a global co-evolution credit scoring model is initialized, a dynamic multi-objective algorithm is used to optimize accuracy, robustness and interpretability of the model, and a credit scoring result is obtained. A causal reasoning model is introduced to analyze driving factors of enterprise credit changes, a causal relationship is established, a closed-loop feedback system is designed, a credit evaluation model is updated, adaptability is enhanced, an interpretable credit evaluation report is generated based on a generative adversarial network, and transparency and credibility are enhanced. According to the method, the accuracy and adaptability of credit assessment are improved, and the robustness and interpretability of an assessment result are ensured.

The invention relates to the technical field of intelligent energy management, in particular to an energy consumption prediction and optimization system for energy-saving management and control, which comprises a data acquisition core module, a dynamic energy consumption prediction core module, an intelligent optimization control core module, a self-adaptive calibration core module, a user interaction core module and the like. The data acquisition module acquires energy consumption, equipment state and environment data from multiple sources; the dynamic energy consumption prediction module fuses improved time series decomposition and a multi-modal LSTM model to realize accurate prediction; the intelligent optimization control module is combined with strategies such as time-of-use electricity price and equipment linkage to generate an optimal instruction; the adaptive calibration module dynamically optimizes the model through Kalman filtering and incremental learning; the user interaction module supports visual display and strategy self-definition; in addition, the system is provided with an edge computing node to guarantee offline operation, an SM4 algorithm and a block chain technology are adopted to guarantee data security, and the system is compatible with various industrial protocols. The energy utilization efficiency is effectively improved, the operation cost is reduced, and the system safety and reliability are enhanced.

The invention provides an intelligent maintenance decision-making method and system for a numerical control machine tool based on digital twinning, and relates to the technical field of equipment maintenance, and the method comprises the steps: obtaining the structural data and simulation precision requirements of the numerical control machine tool, determining the optimal grid size and distribution parameters of a finite element model, obtaining operation conditions, and adding the operation conditions to a multi-physics coupling modeling module, generating a machine tool digital twinborn model, fusing the machine tool digital twinborn model with the physical field, generating a machine tool operation degradation model, and optimizing to obtain a self-evolution digital twinborn model; obtaining machine tool operation parameters, performing preprocessing to obtain multi-modal perception data, extracting multi-view potential features, determining potential correlation features, and obtaining a machine tool health state evaluation result; the method comprises the following steps: generating a multi-level fault prediction result, determining a candidate maintenance scheme, adding the candidate maintenance scheme and the multi-level fault prediction result into a knowledge graph construction module, generating a semantic association network, and solving to obtain an optimal maintenance decision in combination with a multi-objective optimization algorithm.

A multi-modal information tagging method, apparatus and device, and a storage medium and a product. An information tagging effect is effectively improved by means of determining visual feature information of visual information by means of a multi-modal tagging system, performing multi-modal fusion processing on the basis of the visual feature information, the visual information, recognized text information and descriptive text information to obtain image-text fusion features, determining image-text correlation information between the visual information and both the recognized text information and the descriptive text information, on the basis of the image-text fusion features and the image-text correlation information, determining a tagging result for information to be tagged, and performing multidimensional information tagging on the basis of multi-modal information fusion and cross-modal correlations.

The invention provides a network security penetration detection method and system based on artificial intelligence, and the method comprises the steps: firstly obtaining real-time network flow data containing multi-dimensional communication behavior records and protocol metadata of a target network system, and then extracting network behavior sequence features capable of describing communication behavior time relevance and a protocol interaction mode from the network flow data; inputting the network behavior sequence features into a pre-trained multi-modal threat prediction model, generating a potential attack vector prediction result containing an attack intention label, penetration path probability distribution and a threat level score, and dynamically verifying the potential attack vector prediction result based on a threat knowledge base, and finally, generating a self-adaptive defense strategy according to the penetration threat detection result, triggering defense interface deployment of a target network system, updating attack mode characteristics and a defense rule set of a threat knowledge base, and realizing intelligent and efficient network security penetration detection and defense.