1327 results about "Text graph" patented technology

The invention discloses a cross-modal information fusion and retrieval method and system driven by semantic comprehension, and the method comprises the steps: obtaining text, image and audio original data, and extracting an initial feature set of each modal through a deep neural network; dynamically distributing each modal weight coefficient based on an attention mechanism, and performing weighted fusion on the initial feature set to obtain cross-modal fusion feature representation; through a cross-modal semantic association analysis model, high-dimensional semantic association features are extracted from the fusion feature representation, and semantic enhancement feature vectors are generated; constructing a cross-modal semantic graph network based on the vector, complementing missing modal features, and generating an optimized multi-modal feature set; and inputting the optimized feature set and the query sample into a contrast learning model, calculating a semantic similarity score, and generating a cross-modal retrieval result sorting list according to the score.

The invention discloses an intelligent file classification and retrieval method and system, and the method comprises the steps: carrying out the unified format conversion and feature extraction of text, image and audio data according to the original data format of a file through a multi-modal data fusion algorithm, and obtaining a standardized file feature data set; based on a deep learning adaptive classification model, performing classification processing on the standardized archive feature data set to generate multi-level archive classification labels; analyzing the semantic relationship between the archive classification labels through an enhanced semantic network construction technology, generating a semantic association map, and optimizing the semantic association map to obtain an optimized semantic association map; and according to a retrieval request input by a user, performing semantic matching and correlation sorting by utilizing the optimized semantic correlation graph, and outputting archive information most related to the retrieval request. By utilizing the embodiment of the invention, the relevance and accuracy of retrieval can be improved, and modernization and intellectualization of archive management are promoted.

The invention relates to a brilliance large-scale model culture knowledge generation and retrieval method combined with a knowledge graph. The method comprises the steps that entity extraction, relation modeling and dynamic completion processing are carried out according to multi-modal culture data, a culture knowledge graph is constructed in combination with a fuzzy time reasoning algorithm and a cross-modal alignment mechanism, and the multi-modal culture data comprises an unstructured culture text, image data and space-time metadata; the space-time metadata comprises text space-time metadata and image moment metadata; performing semantic alignment processing by combining a large model through a user query statement and a culture knowledge graph to obtain joint representation of cross-modal semantic alignment; and performing multi-modal retrieval-generation joint optimization based on the joint representation to obtain cultural knowledge description and associated multi-modal content. According to the method, through multi-modal data fusion, dynamic knowledge completion and fuzzy time reasoning, culture knowledge description can be accurately generated, associated multi-modal content can be retrieved, and the accuracy and the intelligent level of culture knowledge service are improved.

The invention relates to a cross-modal knowledge reasoning method based on a multi-modal large model. In a cross-modal knowledge reasoning process, an existing model is usually limited by single-modal information extraction and shallow feature fusion, so that deep semantic association among data such as texts, images and videos is difficult to fully capture. In order to solve the problem, the invention provides a model for fusing multi-modal information such as texts, images, videos, documents and the like, and processing of multi-modal data is converted into unified feature extraction, interaction and deep reasoning tasks by fully utilizing a supervision fine tuning strategy, a self-adaptive attention mechanism and a cross-language processing technology. The model adopts a modular design, integrates multi-source data complementary analysis, spatial-temporal feature modeling and emotional semantic analysis, and realizes multi-modal collaborative interaction, dynamic scene understanding, long video key event analysis and man-machine co-emotional response. Through sufficient training, the multi-modal large model shows excellent logical reasoning ability and emotion understanding ability in a complex cognitive task, and a brand new solution is provided for efficient extraction, deep semantic analysis and intelligent response of cross-modal information.

The invention discloses a multi-modal bill processing method based on dynamic knowledge enhancement. The multi-modal bill processing method comprises the following steps: S1, constructing a dynamic knowledge base containing an aging weight; s2, synchronously processing text, image and format features of the bill by adopting a multi-modal feature fusion network to generate a composite feature vector; s3, semantic-level, format-level and timeliness three-level fusion retrieval is carried out based on the composite feature vector, and a three-level fusion retrieval engine comprises dynamic weighted sorting with timeliness attenuation, a difference degree triggered artificial review mechanism and a policy sensitive slope adjustment algorithm; s4, setting a multi-expert cooperative verification system, wherein the multi-expert cooperative verification system comprises cooperative work of a rule engine, a large language model and a logical reasoning module; s5, implementing a dynamic knowledge updating mechanism, and automatically triggering incremental learning of the knowledge base when policy change or format update is detected; and S6, outputting structured data, and synchronously generating an auditing traceability chain containing a decision path. According to the method, the key field identification accuracy can be improved, and auditing traceability and non-perceptual increment updating in the whole process are realized.

The invention relates to the technical field of document retrieval and information processing in the data processing technology, in particular to a document retrieval method based on multistage indexing and feature clustering, which comprises the following steps: performing high-dimensional space mapping on multi-modal features such as texts and images through a quantum embedding layer to generate cross-modal joint feature representation; a first-level index of a multi-level index architecture is dynamically initialized based on a meta-clustering algorithm, and semantic blocks of a second-level index are divided in combination with a multi-head self-attention mechanism. And an optimal transmission matrix is generated by using a Sinkhorn algorithm to align cross-node feature distribution. The multi-target mixed retrieval strategy is fused with vector retrieval, keyword retrieval and graph retrieval results, and weight distribution is dynamically adjusted. Through collaborative optimization of quantum calculation, federated learning and causal reasoning, a closed-loop technical architecture from feature analysis to dynamic index construction is formed, the problems of insufficient cross-modal fusion, static clustering deviation and semantic association deficiency are solved, and the precision, efficiency and dynamic adaptability of heterogeneous document retrieval are improved.

A text detail map-based method for supervising end-to-end text detection and recognition, pertaining to the field of text processing. The method comprises the following steps: given an input image containing text in any shape, processing the input image by means of two separate processing branches; designing a text attention head (TAH), and designing a feature pyramid enhancement fusion module (FPEFM); the FPEFM performing feature self-enhancement at different sizes, fusing text image local features and global text position information extracted by a TAH module, and fusing features extracted by the TAH from feature maps of different sizes; stacking a plurality of FPEFMs to continuously enhance the feature representation capability of a model and the depth of the model; and sampling the feature maps to a unified size to obtain a final enhanced feature map.

The invention relates to the technical field of knowledge maps, in particular to a knowledge map method and system for multi-source heterogeneous data fusion. The method comprises the following steps: cleaning multi-source data through an auto-encoder, dynamically weighting and standardizing after low-rank decomposition and PCA denoising, aligning GNN entities and authenticating standard approval serial numbers; combining CNN / GNN to extract texts, images and sensor multi-modal features, fusing time sequence information with BiLSTM, and performing weighted aggregation; a BERT-BiLSTM-SelfAttention-CRF is adopted to identify an entity, a GCN inference relationship is adopted, a TransE is embedded into an entity relationship to a low-dimensional space, and the entity relationship is stored to Neo4j to support real-time query; dynamically updating the atlas by incremental learning; and visually displaying the constructed knowledge graph. According to the method, a complete closed loop from data cleaning, multi-modal fusion and graph construction to dynamic optimization is formed, and comprehensiveness, accuracy and expandability of the knowledge graph in a multi-source heterogeneous scene are ensured.

The invention discloses an advertisement copywriting generation method and device based on multi-modal fusion, equipment and a medium, and relates to the technical field of advertisement marketing. The method comprises the steps of obtaining multi-modal data of a target video, the multi-modal data comprising visual data, auditory data and related metadata of the video, and extracting associated data from a local knowledge base; and preprocessing the multi-modal data and the local knowledge base data, and converting the multi-modal data and the local knowledge base data into feature forms which can be used for analysis. According to the method, semantic calibration is carried out on multi-modal input by means of a local knowledge base, it is ensured that generated content strictly follows domain knowledge constraints, the problem of deviation caused by the fact that a traditional model depends on implicit knowledge is solved, multi-modal information ambiguity is eliminated, collaborative semantic generation of texts, images and structured data is achieved, content dimensions are enriched, and the method has the advantages of being high in practicability and easy to popularize. And an efficient solution is provided for the landing of the intelligent generation technology in the vertical field.

The invention discloses an LLM-driven complex report OCR error self-correction method and system, and the method comprises the following steps: S1, obtaining complex report image data, executing OCR processing, and constructing an original field data set; s2, extracting context information, identifying semantic contradiction fields, and generating a to-be-corrected field set; s3, the pointer generation network generates a plurality of field correction candidates to form a candidate field set; s4, constructing a dobby machine model, selecting an optimal field correction result, and forming a correction field output set; s5, executing format analysis, and extracting a chart title field, a legend field and a data region text; s6, generating a corrected field result of a chart title field according to a chart structure semantic consistency mechanism; and S7, performing field restoration and format reconstruction, and outputting structured report data. According to the method, intelligent error correction and structured reconstruction of fields in a complex report are realized by fusing a large language model, a pointer generation network and a multi-arm machine mechanism.

The invention discloses a multi-agent dynamic arrangement method based on multi-modal analysis and adaptive retrieval, and relates to the technical field of artificial intelligence and information retrieval. Comprising the steps of S1, converting a text, an image, structured data and voice content input by a user into a unified multi-mode semantic representation, S2, converting the unified multi-mode semantic representation into a specific execution process, and S3, automatically scheduling a reasoning agent, a knowledge obtaining agent and an execution agent according to DAG nodes, task elements and available resources, and obtaining the task elements and the execution agent according to the reasoning agent, the knowledge obtaining agent and the execution agent. S4, after task process construction and agent arrangement are completed, dynamic retrieval, evidence convergence and strategy optimization are carried out on information requirements related to a user task, so that a reasoning agent obtains complete knowledge support with consistent context, and S5, knowledge evidence is combined with a task process, so that the task process is completed. The method comprises the following steps: step S6, implementing problem solving, strategy generation and task closed-loop execution through a reasoning agent, step S6, performing actual operation on a target task by an execution agent according to an executable instruction sequence output by the reasoning agent, and outputting a result, and step S7, performing result verification according to an output result returned by the execution agent, and the correctness, integrity and consistency of an output result are examined through rule verification, model evaluation and evidence alignment.

The invention discloses a long-tail image recognition method based on multi-modal semantic generation and image-text fusion. The method comprises the following steps: extracting structured semantic description from a tail image; carrying out semantic rewriting and enhancement based on a multi-modal visual language model, and generating an image semantic extension description; the image semantic extension description is optimized based on semantic duplicate judgment and a style alignment mechanism, and an optimized text description set is obtained; inputting the optimized text description set into a text graph model, generating a tail class image sample, performing semantic and visual quality screening, and constructing to obtain an enhanced image set for training; constructing a training data set based on the original long-tail data set and the enhanced image set, and training an image-text fusion classification model; and inputting a to-be-identified image into the trained image-text fusion classification model, and outputting classification results of all categories. According to the method, the discrimination capability in a long-tail distribution scene is enhanced, and the method has a stronger generalization characteristic.

An AI-powered content generation system that creates consistent, coherent, and engaging multi-modal content by integrating multiple specialized AI components. The system analyzes user input, identifies key elements, and maintains continuity throughout the generation process. It incorporates a feedback loop to learn and adapt based on user preferences, enabling personalized content experiences. The modular architecture allows for seamless integration of AI components focusing on text, images, audio, and interactive elements. The system ensures consistency across modalities and over extended periods, while managing rights, licenses, and royalties using blockchain technology. This advanced platform revolutionizes content creation, consumption, and management in the digital age.

This application relates to the field of home kitchens, and discloses a social public opinion risk intelligent prediction system, including: a data collection module, which is used to collect text, image and video data from multi-source heterogeneous data sources in real time, including social media, news websites, forums, blogs and self-media; a data preprocessing module, which is used to clean, normalize, segment and tag the collected data to form a structured data set; an advanced natural language processing module, which is used to use the pre-trained Transformer model and graph neural network to perform deep semantic analysis on the pre-processed text data, and extract entity, relationship and context semantic features. By integrating advanced natural language processing, deep learning and multimodal data fusion technology, the system can monitor and accurately predict social public opinion in real time, quickly capture emergencies and public opinion changes, and deeply analyze complex semantics and emotional tendencies.

The invention discloses a multi-modal data fused large model Agent intelligent decision-making method and system, belongs to the technical field of artificial intelligence, multi-modal data processing, deep learning, reinforcement learning and intelligent decision-making, and aims to solve the technical problem of how to improve the performance and adaptability of intelligent decision-making in processing complex tasks and dynamic environments. According to the technical scheme, the method comprises the steps of multi-modal data fusion, wherein text, image and audio data from different modals are integrated, and unified feature representation is generated through feature extraction and feature fusion technologies; intelligent decision-making: decision-making reasoning is carried out based on the fused feature representation, and a final decision-making result is generated by adopting a deep learning model and a reinforcement learning algorithm; adaptive learning: monitoring data changes and decision-making effects in real time, and dynamically adjusting deep learning model parameters and strategies; and feedback optimization: further optimizing the performance of the deep learning model by collecting the feedback information of the decision result.

The invention discloses a multi-modal natural language understanding and generating system and method. The method comprises the following steps: constructing a cross-modal pre-training module, training a multi-modal encoder, and establishing a cross-modal association mapping space; mixing prompt fine tuning is carried out, and a complete blank filling template is constructed; according to the intention reasoning network, extracting multi-round dialogue intention representation of the user, and retrieving an external knowledge base for fine-grained reasoning; constructing a unified semantic representation framework, embedding the text, the image and the voice into a unified space, and generating a query vector of multi-modal intention perception; and the knowledge query module based on key value memory generates entity-level multi-modal replies and optimizes the semantic comprehension and generation capability of the dialogue model. According to the method, the multi-modal information understanding and generating capacity is improved, deep association and understanding of image and text information are achieved, downstream task adaptability is enhanced, task completion accuracy and efficiency are improved, unified semantic representation of the multi-modal information is achieved, and support is provided for information retrieval and utilization.

The invention provides a multi-modal heterogeneous model retrieval enhancement method and system, and the method comprises the steps: building a knowledge and application example double-corpus based on user multi-modal query, and designing a joint retrieval mechanism to obtain a result set; mapping and scheduling to obtain feature representation through special processing channels for texts, images and audios and a Spiking neural network with a segmented trapezoidal topological structure; constructing a three-stage cascade architecture of a basic model, an advanced model and human experts, and obtaining a decision path and answer candidate set in combination with a recursive and discarding decision mechanism; a Hamiltonian graph network is used for representing a multi-modal relation, and a gradient-free descent method is used for rapidly training and optimizing model parameters; an enhanced retrieval result is obtained through cross-modal semantic alignment and dynamic retrieval window adjustment; and high-quality response is obtained through context-aware sorting and retrieval enhanced reasoning. According to the method, the multi-modal information retrieval processing efficiency and the heterogeneous model reasoning response quality are improved.

The invention discloses a traffic large model construction and decision-making method and device based on multi-modal two-way map reasoning, and the method comprises the steps: constructing a multi-modal data set of a text, an image and a track, generating fusion features through spatial-temporal clustering and cross-modal Transform coding, carrying out the two-way map reasoning in combination with a traffic knowledge map, and carrying out the decision-making of the traffic large model. The method comprises the following steps: generating an embedded representation through a forward graph neural network, reversely mapping a decision scheme generated by a language model to a graph to verify consistency, outputting knowledge to enhance embedding, fusing multi-modal features and knowledge embedding by adopting an LoRA multi-task joint fine tuning technology, adapting to traffic field tasks, deploying a real-time inference engine, and carrying out real-time inference on the traffic field. And processing the dynamic data flow through an aging perception attention mechanism, and outputting traffic event identification, path planning and scene question and answer results in parallel. Compared with the prior art, the method has the advantages that the problems of insufficient multi-source heterogeneous data fusion, low knowledge utilization efficiency and poor real-time decision consistency can be solved, and the semantic understanding and decision accuracy of the traffic large model is effectively improved.

The invention discloses a multi-modal data fusion method and system based on a large-model intelligent agent, belongs to the technical field of artificial intelligence, big data processing and intelligent agents, and aims to solve the technical problem of how to effectively integrate data of different modalities such as texts, images and voices by using the large-model intelligent agent. According to the technical scheme, the method comprises the steps that data collection and preprocessing are conducted, specifically, various modal data of texts, images and voice are collected through a web crawler, an API interface, a camera and microphone equipment, the collected data are preprocessed, the preprocessed multi-modal data are obtained, and the data quality is ensured; feature extraction and mapping: extracting corresponding modal features from the preprocessed multi-modal data through CNN and Transform models, mapping the different modal features to the same space, and combining the aligned features to form comprehensive feature representation; carrying out multi-modal fusion processing; and performing intelligent decision and feedback.

The invention relates to an innovative embedded reconstructed text-image alignment style migration method, which comprises the following steps of: constructing a basic framework based on a diffusion model, and adding noise through a forward process and denoising to learn image style characteristics. And extracting style embedding from the artwork data set by utilizing perceptron attention and a feedforward network. Text embedding is generated through a text encoder, and guidance is provided for style migration. A cross attention mechanism is adopted to fuse style embedding and text embedding, multi-modal embedding is generated, and interaction of text and image information is optimized. And fusing multi-modal embedding and image embedding through linear interpolation, splicing with text embedding to form complete prompt embedding, and integrating the prompt embedding into a diffusion model to generate a style image matched with text description. According to the method, the quality and diversity of artistic image style migration are remarkably improved, the method has wide application potential in the field of artistic creation and design, rich and diversified stylized visual achievements can be provided, and the requirements of multiple creation scenes are met.

The invention discloses a multi-modal fusion rumor detection method and system based on a dynamic graph convolutional neural network. According to the method, a dynamic feature graph of a language propagation path is constructed, and potential features in the language propagation process are extracted and analyzed by utilizing time sequence changes and key node relations between nodes in a propagation graph. A neural network is adopted to extract and enhance image data, text semantic features are extracted in combination with a text feature modeling network, text feature vectorization expression is achieved based on a BERT model, and rich semantic information is obtained. And a gating mechanism is introduced to dynamically adjust fusion weights of different modal features, and an information fusion strategy is optimized. A collaborative attention mechanism is further adopted for deep fusion, interactive learning of text, image and propagation path features is enhanced, and the relevance of cross-modal and time series data is improved. And finally, inputting the fused feature vectors into a classifier for accurate classification, thereby realizing accurate detection of the social media rumors. According to the method, the multi-modal features are effectively integrated, and the false information identification efficiency is remarkably improved.

The invention discloses a cloud platform auditing method based on multi-modal data processing, and relates to the technical field of cloud platform auditing. Independent features of text, image, audio, video and log data are extracted, each piece of modal information can be subjected to refined analysis in a corresponding feature space, and in a cross-modal feature fusion stage, a multi-modal feature fusion model is obtained. The correlation between different modal data is calculated by using a cross-modal consistency comparison network CMCN, and feature alignment is optimized through an attention mechanism, so that multi-modal information can be mutually verified, and misjudgment and missed judgment caused by independent auditing are reduced; in the auditing decision-making link, an auditing model based on deep learning is constructed, and end-to-end illegal content judgment is realized in combination with anomaly detection and a self-adaptive risk control threshold value, so that the intelligent level of an auditing system is improved; in addition, the suspicious areas of the image and the video are visually marked, and an auditing conclusion is generated by utilizing a natural language to generate an NLG model, so that the auditing result is more transparent.

The invention relates to the technical field of cloud archive management, and discloses an archive management system based on a cloud archive library, which comprises a data acquisition module, a data preprocessing module, a cross-format semantic retrieval module, an intelligent rule generation module, a distributed authority control module and a dynamic resource optimization module. According to the method, through construction of the multi-modal semantic vector space, unified retrieval of heterogeneous archive data such as texts, images and audios is realized, and the problem of low retrieval efficiency caused by format differences in the prior art is solved; an archive life cycle rule is dynamically generated in combination with a reinforcement learning algorithm, so that the accuracy and suitability of resource management are effectively improved; authority control is realized based on a block chain technology, and the security and consistency of data access are enhanced; through load prediction and resource scheduling optimization, the system energy consumption is obviously reduced, and the response capability of a high-concurrency scene is improved. The file security and the data consistency are enhanced, and the problem of low authority control efficiency in a multi-terminal access environment in the prior art is solved.

The invention discloses a multi-modal fused intelligent question answering and knowledge retrieval method and system, and the method comprises the steps: building a multi-modal data index model oriented to a heterogeneous knowledge source, carrying out the feature mapping of text, image, table, chart, audio and video contents through a unified semantic embedding space, and generating a cross-modal index set; after a query request is received, performing semantic matching and structure matching on the cross-modal index set by using a multi-channel retriever to obtain candidate evidence fragments; and based on an evidence granularity decomposition strategy, performing minimum evidence unit division on text statements, table units, chart data points and multimedia frame contents in the candidate evidence fragments, and establishing a semantic consistency graph among the units. According to the method, high-credibility traceable generation of question and answer results is realized through multi-modal fusion and space-time consistency constraint, and the retrieval precision and interpretation transparency in a complex knowledge scene are remarkably improved.

The invention belongs to the technical field of artificial intelligence, and discloses a government information consultation system based on a large language model, which comprises a user interaction module, a large language model core engine, a knowledge base integration module, a multi-level authority management module, a feedback optimization mechanism and a risk control module. The comprehensive intelligent government affair service system is constructed through the six core modules, remarkable advantages are shown in government affair service digital transformation, the system innovatively adopts multi-mode interactive design, multiple input and output modes of voice, text and images are supported, an intelligent authority management mechanism is matched, and the intelligent authority management mechanism is matched with the intelligent authority management mechanism. According to the technical scheme, the accessibility and convenience of government affair services are greatly improved, precise services for different user groups are achieved, it is guaranteed that sensitive data are safe and controllable while wide spreading of government affair information is guaranteed, and a large language model of a system core is subjected to professional government affair scene optimization training and is combined with a dynamically-updated knowledge graph technology.

The invention discloses a false information detection method based on cross-modal confrontation and progressive training, and belongs to the technical field of artificial intelligence and information security. Mainly aiming at a false information detection task in a social media image-text contradiction form, a progressive adversarial training framework with multi-modal consistency constraint is provided. The core content of the method comprises: multi-modal data acquisition and processing: acquiring aligned social media multi-modal data (text, image / video); multi-modal adversarial sample generation: generating an adversarial sample based on text semantic disturbance and a visual contradiction scene; performing cross-modal progressive training, and optimizing model robustness by combining cross-modal cross attention fusion and a progressive three-stage dynamic training strategy; and generating an interpretability analysis result, and outputting an interpretability thermodynamic diagram to position cross-modal logic conflicts. According to the method, the accuracy and robustness of false information detection of the model in a multi-modal scene can be improved.

The invention discloses an expert question and answer technical method, system and equipment based on local geological knowledge graph semantic reasoning, and the method comprises the following steps: collecting and integrating multi-source heterogeneous geological data, including texts, images and remote sensing, and constructing a unified geological knowledge base covering multi-modal data; on the basis of a pre-training language model, semantic analysis is performed on a geological text, and'entity-relationship-entity 'structured knowledge is automatically generated by utilizing a triple extraction module. The method has field breadth and multidisciplinary fusion, is different from a knowledge graph technology focusing on single fields of mineralogy, geophysics and the like in the prior art, innovatively constructs a large-scale comprehensive knowledge graph system covering the whole geological disciplinary, supports cross-field knowledge extraction and reasoning, and is high in practicability. And complex interdisciplinary geological problems can be handled.

The invention discloses a convergent media content intelligent label generation and target classification method, and belongs to the technical field of artificial intelligence and convergent media. In order to solve the problems of low labor efficiency, poor rule label flexibility and the like of traditional fusion media content label generation, the deep learning and multi-modal fusion technology is innovatively adopted. In a data processing layer, through deep integration of NLP and CV technologies, feature extraction of multi-source data such as texts, pictures, videos, audios and the like is realized. In a label generation layer, an intelligent label engine is constructed based on a CNN and RNN hybrid model, and a dynamic label library is established in cooperation with a rule matching strategy. And in the classification optimization level, the K-means clustering and hierarchical classification methods are combined to realize adaptive classification and optimization of the labels. Through continuous data training and a user feedback mechanism, the label generation accuracy, the classification efficiency and the system adaptability are remarkably improved, and an innovative solution is provided for intelligent management of the convergence media content.

The invention relates to the technical field of data analysis, can be applied to business scenes of medical health, financial science and technology, cultural research and the like, and discloses a multi-modal abstract generation and output method, which comprises the following steps of: acquiring target domain data and cleaning to obtain structured data; extracting entities and association relationships among the entities from the structured data, and constructing a target domain knowledge graph; generating a text abstract containing entity associated information based on the target domain knowledge graph; matching the image data and the audio data corresponding to the text abstract, and integrating the text abstract, the image data and the audio data to generate a multi-modal abstract; and constructing a user portrait according to the user behavior data, and outputting the multi-modal abstract based on the user portrait. According to the method, extraction and analysis of entities and association relationships thereof are optimized through knowledge graph construction; through multi-modal fusion, associated display of texts, images and voices is realized; and through user portrait optimization output, the efficiency and adaptability of knowledge summarization are improved.

The invention discloses a multi-modal data driven general report generation method and system based on a large model, and belongs to the technical field of intelligent report generation. Firstly, texts, images and sensor data related to a report theme are obtained and subjected to standardized preprocessing; analyzing the report generation instruction, and matching and querying a task modal mapping library matching modal configuration scheme according to a task demand; quantitatively evaluating the data quality of each modal, dynamically calculating the final decision weight of each modal in combination with the basic weight, and distributing the final decision weight to a corresponding processing path to form dominant, supplementary and reference data; inputting the dominant data and the supplementary data into a multi-modal model for analysis to obtain a preliminary conclusion with confidence score, performing consistency judgment, if no conflict exists, performing fusion to form a comprehensive conclusion, and if the conflict exists, combining a quality evaluation result and an arbitration rule to complete conflict judgment; and finally, inputting the comprehensive conclusion and the reference data into a large language model to generate a report text, and outputting a complete report after typesetting and proofreading.