2319 results about "Attention network" patented technology

The invention relates to a knowledge graph construction method and system based on a large language model, and the method and system achieve the automatic construction and dynamic maintenance of a knowledge graph through multi-modal data fusion, reinforcement learning and comparative learning joint optimization, teacher-student model knowledge migration, time sequence dynamic analysis and an incremental updating mechanism. Constructing a reinforcement learning framework, and taking accuracy and integrity as reward indexes to train a large model to extract an entity relationship; a large-scale knowledge graph is used as a teacher model, and conflict resolution and semantic alignment of newly added knowledge and an existing graph are realized through a graph attention network; a verification rule is dynamically generated based on historical data and domain knowledge, and relation periodicity and mutation points are detected in combination with Fourier transform and a CUSUM algorithm; and finally, generating a traceable knowledge graph through incremental updating and version control. And the multi-modal data processing precision, the entity relationship extraction dynamic adaptability and the knowledge graph maintenance efficiency are improved.

The invention discloses a machine vision-based resistor disc defect online detection system and a grading method, relates to the technical field of industrial machine vision detection, and solves the defect problems in the aspects of multi-scale defect dynamic perception, cross-level feature interaction and process adaptive optimization in the prior art. According to the scheme, metal reflection interference is inhibited through Retinex illumination correction and a combined denoising model; adopting a deformable convolution kernel and cavity space pyramid pooling to realize gradient entropy driving dynamic sensing of the multi-scale defect; constructing a bidirectional cross-layer attention network to realize early fusion of high-resolution details and high-level semantics; modeling local-global feature physical association based on a graph attention network and a self-supervised message passing mechanism; integrating reinforcement learning and a memristor random calculation unit to form a closed-loop parameter optimization system; according to the method, the multi-scale defect detection precision, the cross-modal feature fusion efficiency and the system adaptive capacity under complex working conditions are remarkably improved.

The invention relates to the technical field of fault prediction, and provides a fault prediction method for a multi-modal cross-attention enhancement graph neural network, and the method comprises the steps: collecting the data of equipment; performing adaptive enhancement and normalization processing on the image data, performing sliding window segmentation, standardization and noise suppression on a time sequence numerical signal, and performing semantic vectorization coding on a maintenance log text; extracting low-dimensional spatial features of image data by using the pruned lightweight convolutional neural network, connecting time sequence features of modeling time sequence numerical signals in series, extracting context semantic expressions of maintenance log texts, integrating the features into multi-modal data, alternately taking each modal feature as Query and the other modal features as Key and Value, and obtaining multi-modal data; calculating attention weight and performing weighted fusion; constructing a modal node weighted graph, and performing inter-node feature propagation through a multi-layer graph attention network; and a residual service life regression prediction module and a degradation level classification module are deployed in parallel, and fault early warning is completed through multi-task joint optimization.

The invention discloses a vehicle multi-modal trajectory prediction method based on an improved attention network, and belongs to the technical field of intelligent vehicle trajectory prediction, and the method comprises the steps: collecting historical trajectory data of a target vehicle and surrounding vehicles as an input sequence; secondly, constructing a vehicle multi-modal trajectory prediction model which comprises a motion feature extraction module, a space-time interaction module, a space-time fusion module and a trajectory output module; the motion feature extraction module uses a multi-scale convolution attention network and a gating circulation unit for processing, the space-time interaction module uses a dynamic graph attention network for extracting vehicle interaction information, and the space-time fusion module splices and fuses target vehicle motion features and space-time interaction features to obtain space-time fusion features; the track output module inputs the fusion features into a gating circulation unit, decodes the fusion features and then inputs the fusion features into a mixed density network, and multi-mode output of vehicle tracks is achieved; and finally, a proper loss function is selected for training, so that the prediction precision and the convergence speed of the model are improved.

The invention relates to the technical field of artificial intelligence, in particular to an artificial intelligence-based image segmentation and dynamic target recognition method, which comprises the following steps of: accurately positioning a candidate region through multi-modal space-time fusion and dynamic confidence coefficient screening; strengthening spatial-temporal feature expression in a layering manner through a multi-level feature decoupler, and generating a multi-dimensional feature enhanced spatial-temporal candidate region; through a deformable segmentation network, a deformation convolution kernel and edge motion matching loss are combined, joint optimization of a geometric boundary and motion continuity is realized, and the segmentation robustness of a flexible target is improved; through optical flow back propagation dynamic correction and confidence coefficient propagation, high-precision segmentation masks with consistent time and space are output; and through a target trajectory re-identification and completion mechanism driven by a graph attention network, and in combination with optical flow deformation prediction, stable tracking in a shielding scene is realized.

The invention relates to a time sequence knowledge graph multi-hop reasoning method and system oriented to the legal field. The method comprises the following steps: establishing a dynamic mapping relationship through a semantic association technology; extracting life cycle states of the legal provisions, and introducing a legal conflict detection algorithm; setting a revocation influence factor; determining time embedding representation, and introducing clause conflict matrix elements to construct a time sequence knowledge graph; constructing a space-time coupled completion model based on the graph attention network and the long short-term memory network to perform path completion; determining three selected agents, performing reinforcement learning, designing a reward function for each agent, constructing a collaborative arbitration mechanism in combination with a dynamic priority strategy and a weight adaptive algorithm, and performing multi-hop reasoning. By collecting multi-source legal data, the authority, the real-time performance and the relevance of the data are ensured; according to the method, the path reasoning capability in a low-coverage scene can be improved, so that a complex legal knowledge multi-hop path reasoning task can be accurately and efficiently completed.

The invention discloses a BIM (Building Information Modeling)-based hoisting construction safety monitoring and early warning system. The system comprises a terminal sensing layer which is used for collecting environmental parameters and personnel behavior data in a closed space in real time; the edge computing layer is used for carrying out cleaning, compression and encrypted transmission on original data by utilizing an explosion-proof edge computing gateway; the cloud collaboration layer is used for storing full data based on a BIM digital twinborn platform, constructing a'danger mode-construction feature-disposal measure 'three-dimensional meta-knowledge graph by adopting an MAML + + algorithm, meanwhile, coupling a physical mechanism data enhancement engine with a multi-physics field coupling model and a physical constraint generative adversarial network, generating virtual data conforming to mass conservation and energy conservation, and sending the virtual data to the cloud collaboration layer; performing mixed training with real data; according to the intelligent decision-making layer, a space-time adaptive threshold evolutionary algorithm encodes a space-time context through a graph attention network and Transform, an alarm threshold is dynamically optimized through deep reinforcement learning, meanwhile, a digital twin deduction engine calculates a shortest safety path in real time, and rescue resource allocation is optimized.

The invention provides a dam intelligent early warning method and system based on a depth time sequence attention network, and the method comprises the steps: carrying out the time alignment and sliding window segmentation of environment variables and historical displacement data of dam monitoring, extracting multi-scale statistical features, fusing the multi-scale statistical features with original features, carrying out the unified normalization processing of a spliced high-dimensional vector, and carrying out the calculation of the unified normalization processing; model input is generated; based on the constructed DSA-Net, carrying out local feature extraction, bidirectional time sequence modeling and key moment weighting on an input sequence, and jointly outputting horizontal and vertical displacement predicted values; fusing double deformation prediction results into a unified radial deformation index, and dynamically setting an early warning threshold value band according to a historical residual error to realize self-adaptive deformation early warning; quantitative evaluation is carried out on the model prediction precision, online prediction and threshold determination of real-time observation data are realized by using the qualified model and an adaptive threshold mechanism, abnormal early warning is triggered, and alarm information is recorded. According to the invention, deep coupling of deformation dimensions and dynamic threshold early warning are realized.

The invention relates to the technical field of automatic driving, and discloses an automatic driving lane changing trajectory planning method based on deep learning, and the method comprises the steps: carrying out the data collection and preprocessing of a multi-modal sensor; performing spatial feature extraction and time sequence modeling on the preprocessed multi-modal data by adopting a CNN-LSTM hybrid architecture, performing feature fusion through an attention mechanism, and outputting a first feature extraction vector; taking the detected vehicles as graph nodes to construct a traffic graph, learning an interaction relationship between the vehicles through a graph attention network and a message passing mechanism, and calculating a scene urgency score and a safety score; generating a lane changing decision based on the deep Q network and the strategy gradient; and generating a trajectory based on the generative adversarial network. The technical problems that an existing lane changing track planning method cannot adapt to the dynamic traffic environment, lacks the ability of understanding complex multi-vehicle interaction and is difficult to balance safety and urgent conflict requirements are solved, and intelligent, safe and efficient automatic driving lane changing track planning is achieved.

The invention, which relates to the technical field of payment security, discloses a payment-scene-oriented interaction intention identification and error correction system comprising an input analysis module, an intention simulation module, a dynamic decision module, a biological verification module, an audit evidence storage module, and a cross-scene knowledge migration module. According to the method, multi-modal data such as voice, texts, images and touch tracks are integrated, structured feature vectors are generated through a cross-modal attention network, the problem of incomplete single-modal coverage is solved, cross-modal data consistency verification is achieved based on a unified semantic tag system, and the reliability of input sources is graded by combining equipment fingerprints and geographic positions, so that the reliability of the input sources is improved. A high-risk transaction protection capability is enhanced, a generative adversarial network is utilized to construct a virtual attack sample library, attacks such as tampering with characters similar in shape and AI faking voiceprints are simulated, unknown threats are actively defended through cosine similarity matching, a user historical behavior statistical model is integrated, and known risks such as high-frequency small-amount transfer are passively intercepted. And a closed-loop incremental learning continuous optimization model is supported.

The invention discloses an industrial equipment fault detection method fusing a complex relation and space-time dependence, and belongs to the technical field of industrial anomaly detection, and the method comprises the steps: constructing a plurality of adjacent matrixes, carrying out the weighted fusion to form an enhanced adjacent matrix, and comprehensively and accurately describing the complex multi-dimensional relation between industrial equipment; designing a spatial-temporal feature extraction module, extracting spatial features in parallel by using a graph convolutional neural network and a random graph attention network, extracting time features through time convolution and a multi-head attention mechanism, and dynamically fusing the spatial-temporal features by means of a gating mechanism to generate graph-level features; a state judgment layer composed of a plurality of node-level binary classifiers and a voting mechanism are adopted to comprehensively judge classification results of all nodes, so that the stability and reliability of judgment of the overall state of the industrial control system are enhanced, and the risk of misjudgment is reduced; the problems of equipment relation modeling and multi-dimensional information fusion are effectively solved, features are extracted and fused more accurately, and the accuracy and adaptability of anomaly detection are improved.

The invention discloses a software multi-agent collaboration method and system based on a large language model, and the method comprises the steps: receiving natural language task description submitted by a user at the same time, carrying out the semantic understanding and intention recognition through a pre-trained large language model center, and generating a structured task element set; based on the structured task element set, the large language model center generates a task dependency graph through multiple rounds of reasoning, and the task dependency graph comprises a plurality of atomic subtasks, logic relations among the tasks and data flow constraints; according to a topological structure and resource demand characteristics of a task dependency graph, a double-layer graph attention network is adopted to dynamically match a professional agent with specific domain capability, and a distributed collaborative network is formed. Through the dynamic graph network scheduling and cross-domain semantic alignment mechanism, the problems that the multi-agent dynamic collaborative adaptation capability is insufficient and cross-domain semantic fusion is difficult are solved.

The embodiment of the invention relates to the technical field of artificial intelligence, and provides a linkage method and system of an apartment network and intelligent equipment, and the method comprises the steps: collecting network operation data and equipment operation data of a target apartment in real time; based on the equipment operation data, determining currently achievable candidate service scenes of the target apartment; fusing the network operation data and the equipment operation data into a joint operation map of the target apartment through the space-time diagram attention network; a multi-agent depth deterministic strategy gradient algorithm is adopted to carry out network resource allocation and equipment control decision making on the joint operation map, scene adaptive optimization is realized in combination with candidate service scenes, and a linkage strategy of a target apartment is obtained; and generating a decision instruction based on the linkage strategy, and adopting the decision instruction to realize remote control of the network equipment and the intelligent equipment. Through deep fusion of the space-time diagram attention network and multi-agent reinforcement learning, dynamic collaborative management and control of the network and equipment are realized, and the resource utilization rate and the service response speed are improved.

The invention provides a mountainous area tunnel construction safety intelligent monitoring and early warning method and system, and relates to the technical field of construction safety monitoring, and the method comprises the steps: collecting visible light and depth images of tunnel surrounding rock, and carrying out the segmentation and extraction of crack features through a depth attention network after image preprocessing and data fusion; extracting parameter time sequence data based on the crack spatial position and the type feature; determining fracture evolution characteristics and critical state parameters by combining wavelet transform and stress-rate coupling analysis; and adopting deep reinforcement learning to calculate the instability probability and generate early warning information. According to the invention, intelligent identification, instability prediction and risk early warning of tunnel surrounding rock cracks are realized, and the safety monitoring accuracy and early warning timeliness are improved.

The invention discloses an exoskeleton motion signal peak valley intelligent detection method and system, and relates to the technical field of computer assistance. The method is used for solving the problems of control delay and high misjudgment rate caused by large motion signal noise interference and inaccurate processing in an exoskeleton system. The method comprises the steps of firstly, suppressing motion artifact noise and generating a high-signal-to-noise-ratio preprocessing signal through multi-modal signal collaborative noise reduction and dynamic energy entropy segmentation, secondly, constructing a parallel convolution attention network to extract multi-modal features, and extracting peak and valley candidate points in combination with dynamic weight fusion and multi-scale differential detection; a peak valley point set is optimized based on a variable structure density sensing clustering algorithm and density gradient analysis, artifact interference is eliminated, finally, a multi-rule confidence model is constructed by fusing time sequence prediction of a bidirectional gating circulation unit and biomechanical correlation, and a threshold value is dynamically adjusted to trigger an exoskeleton joint assistance instruction. Closed-loop processing from signal acquisition to real-time control is realized, and peak valley detection precision and response speed are remarkably improved.

The invention relates to the technical field of safety management and risk prediction, in particular to an intelligent safety management and risk prediction method and system based on cloud computing. The method comprises the following steps: dynamically accessing multi-source heterogeneous data through a cloud platform, and forming unified event representation through time alignment and credibility labeling; constructing a hierarchical mixed probability safety twin model, updating dynamic parameters by adopting credibility weighted online variational Bayesian, and outputting a state interface by combining structural adaptation, cross-object graph regularization and physical constraint projection; mapping the twinborn state into a causal feature, constructing an intervening causal graph, generating causal embedding by using a credibility weighted attention network, simulating an intervention operation in an embedding space, and quantifying a risk probability; and generating a multi-candidate security policy, evaluating and sorting through a multi-objective utility function, executing an optimal policy, collecting feedback data, and updating the model and the policy. According to the method, credibility regulation and control, probability twinning and causal intervention are fused, and real-time intelligent decision making of an industrial safety scene is supported.

The invention relates to the technical field of power grid monitoring, in particular to a power operation risk identification method, system and device based on multi-modal data fusion and a storage medium. In order to solve the problems of multi-modal data splitting, topological constraint missing and the like in traditional power disturbance analysis, an improved BERT model is constructed, and electrical signal time-frequency features and text semantic information are mapped to a unified vector space through a multi-modal embedding mechanism; a time sequence attention mechanism is adopted to establish a time dependency relationship between signals and texts, and a graph attention network is combined to realize risk propagation modeling under power grid topology constraints; and collaborative optimization of disturbance classification, risk prediction and trend analysis is carried out through a multi-task learning framework. The technical problems that heterogeneous data fusion is difficult and risk identification precision is insufficient are effectively solved, accurate identification and intelligent early warning of electric power operation risks are achieved, and the safe operation level of a power grid is improved.

The invention discloses a method for reducing a large model illusion problem based on an RAG technology, and the method comprises the steps: extracting semantic entities, relation phrases and context features in a natural language query, and constructing a multi-source heterogeneous hypergraph; fusing the graph structure and sequence context information by using a graph attention network and a sequence perception network to form unified semantic representation; evaluating the illusion risk based on the confidence score, the evidence coverage rate and the semantic deviation index, and triggering reverse retrieval and fusion reinforcement; and the content is generated through causal consistency discrimination feedback control. According to the method, the illusion phenomenon of the generation result is remarkably reduced, and the method is widely applied to the field of intelligent question answering and information retrieval.

The invention discloses a hybrid expert and KAN-based cyclic attention network time sequence prediction method. The method comprises the following steps: S100, inputting time sequence data needing to be predicted; s200, constructing a graph structure by using an attention mechanism, learning basic correlation characteristics among variables of the input time sequence data through an adaptive and learnable graph convolutional network, and then performing global averaging and maximum pooling on the basic correlation characteristics along a time dimension to obtain complementary time domain statistical information, so as to provide effective time-space correlation characteristics for the follow-up process; s300, after feature learning is completed, collaborative modeling of the KAN and an attention mechanism is brought into full play, rapid and efficient time sequence modeling is carried out on data by adopting a cyclic attention network embedded based on the KAN, and a foundation is laid for subsequent time sequence prediction; and S400, establishing a hybrid KAN expert-based time sequence prediction network, and adaptively fusing differentiation prediction results by a gating mechanism. The time sequence prediction method is designed from the three aspects of feature learning, time sequence modeling and time sequence prediction.

The invention discloses a semantic recognition system and method based on a heterogeneous graph attention network and dynamic normalization, and belongs to the technical field of natural language processing and artificial intelligence. The system adopts a dual-channel architecture and comprises a general semantic channel and a domain semantic channel, semantic feature extraction is performed through a DIFF attention mechanism and a ToST statistical attention mechanism, and training stability is improved by adopting a DyT dynamic normalization module. Adaptive fusion of cross-channel semantic features is realized through a GeGLU gating mechanism, and high-precision semantic recognition is realized by combining an improved SimCSE + + comparison learning loss and a local minimization editing strategy of semantic perception. According to the method, the problems of inaccurate semantic expression, poor context adaptability and the like in the prior art are solved, the accuracy and applicability of cross-domain semantic recognition are remarkably improved, and the method can be widely applied to scenes of legal document processing, financial document analysis and the like.

The invention belongs to the technical field of remote sensing super-resolution images, and particularly relates to a remote sensing image super-resolution system and method based on an adaptive Mamba-attention network. Comprising a feature extraction module used for carrying out shallow feature extraction on an input low-resolution image to obtain shallow features; the multiple cascaded adaptive state space blocks are used for processing the shallow layer features to obtain reconstruction features; and the reconstruction module maps the reconstruction features to a target resolution space through sub-pixel rearrangement operation to obtain a high-resolution remote sensing image. High-frequency details and a low-frequency structure are cooperatively processed in a feature space by using the remote sensing frequency sensing modulation module, and high-resolution output is generated by combining sub-pixel rearrangement up-sampling, so that high-quality reconstruction of a complex remote sensing scene is realized.

The invention discloses an intelligent electric meter anomaly detection method and system based on a federal differential privacy and attention mechanism. The method comprises the following steps: firstly, collecting and regionally grouping intelligent electric meter data; then, a multi-level attention mechanism is adopted to extract regional specific initial features, and spatial correlation features are enhanced through fusion of a graph attention network and multi-head attention; carrying out distributed anomaly preliminary identification; for potential anomalies, user behavior log features are safely obtained and protected through a federated learning framework and an LDP mechanism; secondly, performing security aggregation and optimization on local model parameters which are trained by all parties and subjected to parameter-level differential privacy protection under a federated learning framework; local model fine tuning and secondary cross validation are carried out to confirm true abnormity; performing qualitative traceability on the abnormal event by utilizing multi-dimensional dynamic attention; and finally, dynamically adjusting a feature extraction strategy through federal feedback, and optimizing a data processing scheme in combination with resource awareness. According to the method, the problem of balance of precision, privacy protection and model adaptability in data anomaly detection of the intelligent electric meter is solved.

The invention discloses a cloud side-end cooperative task scheduling and efficiency optimization method and system for heterogeneous patrol resources, and relates to the technical field of intelligent scheduling and resource optimization. According to the method, accurate perception of a resource state is realized by constructing a digital twinborn and federated learning mechanism, resource contention conflicts are solved by adopting a space-time diagram attention network and multi-agent reinforcement learning, and multi-target optimization and trusted execution are realized in combination with a quantum genetic algorithm and a block chain smart contract. Finally, the stability of the system is verified through Lyapunov optimization, a complete scheduling system from resource perception and conflict resolution to steady state maintenance is formed, and the task scheduling efficiency and the system stability in the heterogeneous resource environment are remarkably improved.

The invention provides an industrial fault feature adaptive extraction and multi-mode detection system and method, and belongs to the technical field of industrial fault detection. Comprising the steps of collecting multi-source data of industrial equipment, performing timestamp alignment and processing on the multi-source data to obtain a standardized feature sequence, inputting the standardized feature sequence into a dynamic convolutional neural network, extracting signal local features through a deformable convolution kernel, calculating feature weights in combination with a self-attention mechanism, and screening feature channels to obtain feature vectors of all modes; a graph structure with modals as nodes and correlation as edges is constructed, cross-modal features are aggregated through a graph attention network, and global state descriptors fused with spatio-temporal information are generated; and performing time sequence modeling on the global state descriptor through a bidirectional LSTM network, outputting fault type probability distribution, and completing industrial equipment fault detection. According to the method, the problems of insufficient single-modal information, fixed feature extraction, low efficiency of multi-modal correlation modeling and lagging model updating in traditional industrial fault detection are solved.

The invention relates to a multivariate time series data abnormal root cause identification method based on a space-time cause and effect diagram, and belongs to the technical field of anomaly detection. According to the method, multi-window expansion causal convolution is adopted for multivariate time series data, mutual information screening is combined, and time embedding covering short-term mutation and long-time dependence at the same time is extracted; non-local space correlation is learned through multi-head self-attention, the directional causal intensity is measured through conditional entropy, and a sparse and interpretable space-time causal graph is generated through normalization-pruning; introducing a causal enhancement graph attention network on the space-time causal graph, and performing multiple rounds of causal propagation updating on node embedding; and calculating a root cause score by integrating the abnormal degree and the causal influence, identifying a key source node in an abnormal propagation path, and realizing accurate root cause positioning of the system abnormality. According to the method, the adaptability to the dynamic behavior mode and the capturing capability to the abnormal driving factor are enhanced, and the modeling precision and the root cause identification capability of the abnormal propagation process are improved.

The invention discloses an anesthesia early warning method based on multi-source signal fusion. The anesthesia early warning method comprises the following steps that physiological signals in the anesthesia process are collected in real time and preprocessed; a multi-modal Transform attention network is adopted to extract correlation features in the time sequence physiological signals, and multi-modal fusion features are obtained; predicting the fusion feature at the next moment by using a long-short-term memory neural network to obtain a prediction error; updating a noise and observation covariance matrix of the Kalman filter according to the prediction error; the dynamically adjusted Kalman filter calibrates the multi-modal fusion features in real time; according to the calibration characteristics, anesthesia depth and consciousness state prediction values are calculated in real time, and early warning is output. The real-time performance, the stability and the accuracy of anesthesia state monitoring are improved.

The invention discloses a drug resistance prediction method and system based on comparative learning and multi-modal fusion, and the method comprises the steps: firstly generating a molecular map and a molecular fingerprint based on the SMILES of a target drug, and extracting the molecular features of the drug through a comparative learning model constructed through combining a map attention network and a map convolution network; and then, acquiring protein expression, gene expression and metabolic expression data from the target tissue cells, extracting modal features through a deep convolutional network, a Transform encoder and a multi-dimensional attention network, and realizing adaptive fusion of the multi-modal features through a heterogeneous interactive attention mechanism. And finally, jointly inputting the fused multi-modal features and drug molecular features into a multi-layer sensor to realize high-precision prediction of the drug resistance of cells to drugs. By introducing a contrast learning and multi-modal feature fusion mechanism, the characterization capability and prediction precision of the model are effectively improved, and efficient and reliable support can be provided for drug screening and clinical decision making.

The invention provides a low-visibility environment pedestrian detection method based on an improved YOLOv8n model. A double-branch fusion attention network is adopted in a backbone network of an original YOLOv8n model, and a CBFuse module and a CBLinear module are introduced for feature fusion between different branches; a feature aggregation and calibration pyramid network is introduced, multi-scale feature fusion is carried out through an up-sampling module, a down-sampling module and a feature aggregation and calibration module, and the feature aggregation and calibration module carries out feature calibration and enhancement through a local attention mechanism, a global attention mechanism and a pixel attention mechanism; an adaptive task alignment detection head is introduced to execute a dynamic convolution mechanism, a task decomposition mechanism and a dynamic feature alignment mechanism; an improved YOLOv8n model is formed based on the improvement and serves as a foggy day pedestrian detection network model; according to the method, the detection accuracy and stability of the network in processing shielded and background complex images can be enhanced, and the boundary and detail features of a fuzzy target can be extracted more accurately in low-visibility environments such as foggy days and the like.

The invention belongs to the technical field of document content extraction, and particularly discloses a document content extraction method and system based on multimodal model collaboration, a terminal and a medium. Comprising the following steps: identifying the type of an input to-be-processed document, and judging the document type; on the basis of the type identification result, calling a multi-modal model to analyze the document content, and outputting space coordinates, visual features and semantic features of document elements; generating a content sequence according with a reading habit through a semantic sequence reconstruction algorithm; paragraph boundary detection, paragraph recombination and semantic association modeling of charts and texts are completed based on the multilayer attention network and the graph neural network; grammar error correction, format optimization and title hierarchy generation are carried out by using a large language model and a hierarchical classification network; and converting the identification result into a structured output file. According to the method, the processing requirements of different types of documents can be considered, and high-precision analysis and efficient output are realized under the scenes of complex layouts, multiple languages and formula tables.

A sparse processing method and apparatus for a sparse attention network, and an electronic device. The method specifically comprises: in a memory cell, using constraint conditions to construct a masked sparse representation (S301), wherein the constraint conditions are: the dimension of a mask matrix is [B,a,S], B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in the S dimension represents a masked starting row of each column in the mask matrix; and a calculation unit acquiring the masked sparse representation from the memory cell, using the masked sparse representation to perform sparse processing on input data, and storing the sparse processing result into the memory cell (S302). According to the method, the constraint conditions are used to construct the masked sparse representation, so that memory consumption can be reduced from a quadratic order of the sequence length to a linear order of the sequence length, thereby remarkably reducing memory requirements during large model training, and improving the training efficiency.