2578 results about "Question answering" patented technology

The invention discloses a traffic engineering large model intelligent question answering system and method based on a knowledge graph, and the method comprises the steps: extracting a structured degree feature, a semantic ambiguity feature and a context association feature through receiving and analyzing a natural language query statement inputted by a user, generating a retrieval intention vector, and carrying out the retrieval of the retrieval intention vector; and dynamically selecting a retrieval path according to the intention classification model. And according to the retrieval path, constructing a structured query statement or a semantic vector, and respectively retrieving in the knowledge graph and the vector database to obtain a first retrieval result and a second retrieval result. Further performing bidirectional verification through entity consistency, semantic similarity and relation connectivity indexes, screening a candidate result set, and constructing a reasoning chain; if the inference chain is broken, a large model inference gap complementation mechanism is adopted to generate relay nodes, a complete inference chain is formed, and inference type answer output is generated based on the complete chain. According to the method, the retrieval accuracy and reasoning continuity of the question-answering system are improved.

The invention provides an intelligent real-time interactive question-answering system based on a virtual digital human, and belongs to the technical field of voice signal processing and voice recognition, and the system comprises a data acquisition module which receives a voice or text interaction request input by a user, collects the expression dynamic parameter sequence and limb movement sequence data of the user in real time, and transmits the data to a user interaction module; obtaining a standardized voice feature vector and structured text data; the cross-modal fusion module is used for constructing an interactive feature matrix; the behavior decision module outputs a decision instruction set; the knowledge retrieval module is used for generating an answer text with emotional adaptability and voice features; and the voice generation module is used for generating a mouth shape animation key frame, a micro expression parameter sequence and a limb action track of the virtual digital human, generating a voice response in combination with the answer text and the voice characteristics, and pushing the voice response to the user terminal. According to the method, the interaction experience and adaptability of the virtual digital human are remarkably improved.

The invention relates to the technical field of knowledge graph construction, and discloses a knowledge graph construction method and system based on a large language model technology. The method comprises the following steps: receiving a multi-source heterogeneous data stream, and completing semantic space mapping and cross-modal feature fusion to generate a unified semantic representation vector set; constructing an initial knowledge graph skeleton; performing incremental optimization on the skeleton, and performing entity relationship disambiguation and conflict detection; and iteratively updating the knowledge representation, and outputting a target knowledge graph meeting semantic consistency. The system comprises a data receiving module, a semantic fusion module, a skeleton construction module, an optimization module and a knowledge updating module. According to the method, multi-source heterogeneous data is effectively processed, the accuracy, the dynamic updating capability and the semantic consistency of the knowledge graph are improved, and the method has wide application prospects in the fields of intelligent question answering, information retrieval and the like.

The invention relates to an intelligent question-answering system optimization method and device based on a knowledge graph, and the method comprises the steps: obtaining original knowledge data of a target knowledge domain, and constructing a knowledge graph structure model; extracting term information of entity nodes in the knowledge graph structure model, and constructing an entity term set; receiving a natural language question input by a user, executing a semantic understanding operation based on the standardized expression set to obtain a structured question semantic representation, and matching the question semantic representation with the case training set to obtain context semantic features; constructing a cue word template, and executing a query instruction generation operation to obtain a target query statement of the graph database; submitting the target query statement to a graph database to execute data retrieval operation, and obtaining query result data corresponding to the question semantic representation; and performing personalized rendering processing on the query result data based on the user portrait information to generate final question and answer return content. The method has the effect of improving the query accuracy.

The invention relates to the technical field of computer systems for natural language processing or semantic processing, and discloses an enterprise-level schedule planning and knowledge base-oriented intelligent collaborative question-answering system and an enterprise-level schedule planning and knowledge base-oriented intelligent collaborative question-answering method. The system comprises a vectorization processing module, an RAG knowledge base module and the like. The system converts natural language input of a user and enterprise knowledge data into semantic vectors, and retrieves related enterprise knowledge fragments from a vector database based on semantic similarity. The large language model core module analyzes the user intention, generates a preliminary answer and identifies whether a schedule type operation request is included or not; and if the schedule operation request exists, the system splits the request through the multi-agent cooperation module and distributes the request to the corresponding agent to obtain a task processing result. And finally, semantic consistency fusion is carried out on the preliminary answer and a task processing result through a context fusion module, a comprehensive answer is generated by a large language model and is returned to a user side, and unified intelligent response of enterprise knowledge and schedule service is realized.

The invention discloses an adaptive question-answering system based on knowledge distillation and multi-modal dynamic fusion, and the system comprises a knowledge distillation module which is used for migrating knowledge of a teacher model pre-trained on corpora in the communication field to a lightweight student model, achieving model compression through optimizing a distillation loss function, and obtaining a multi-modal dynamic fusion model; the loss function comprises a soft label output by the teacher model and a KL divergence constraint output by the student model; the multi-modal knowledge fusion module comprises a feature extraction unit, a self-adaptive weighting unit and an attention fusion unit; the self-adaptive inference engine comprises a semantic analysis unit; according to the cross-modal reasoning method and system, semantic alignment of equipment parameters, protocol texts and topological graphs is achieved through the multi-modal dynamic fusion technology, and the cross-modal reasoning accuracy is improved; compared with an original model, the lightweight student model has the advantage that the reasoning speed is increased in a protocol analysis task.

Provided in the embodiments of the present disclosure are a question answering processing method and system, a device and a storage medium. The method comprises: on the basis of a user question, a first prompt template containing first tool set information, and an LLM, a first agent acquires, successively generated by the LLM, a plurality of pieces of solving task information that are required to be executed by invoking a second agent and summary task information that is required to be executed by the first agent, and successively transmits the plurality of pieces of solving task information to the second agent; on the basis of received current solving task information, a second prompt template containing second tool set information, and the LLM, the second agent acquires task execution information that is generated by the LLM and corresponds to the current solving task information, and accordingly acquires from an application server a task execution result of the current solving task information; and the first agent summarizes the task execution results of the plurality of pieces of solving task information, so as to determine reply information corresponding to the user question. By means of the cooperation of different agents, user questions are gradually and accurately solved.

The invention discloses a question answering method for graph-driven fusion retrieval in the field of computer networks, and relates to technologies such as intention recognition, multi-channel fusion retrieval, hierarchical semantic matching and answer traceable generation. Aiming at the problems of professional term matching, incomplete knowledge, single retrieval and low answer credibility of an existing question-answering system, a dual-channel fusion retrieval and hierarchical semantic matching mechanism based on a knowledge graph is provided, a retrieval strategy is dynamically controlled through intention recognition, accurate matching of structured knowledge is realized by adopting phrase preliminary screening and triple fine matching, and the accuracy of the structured knowledge is improved. In combination with graph subgraph dynamic construction and Prompt generation, a large language model is guided to output high-quality answers, traceable knowledge sources are attached, and the accuracy and credibility of the answers are improved. The method is widely applicable to computer network professional question and answer scenes such as network protocols, configuration management and troubleshooting.

The invention discloses a self-adaptive learning question answering system and method based on multi-modal interaction, and particularly relates to the technical field of self-adaptive learning question answering. By constructing a modal recognition and preprocessing module, standardization and structuralization of multi-modal input of texts, voices, images and the like are realized; establishing a cross-round modal memory map through time sequence coding and map modeling; dynamically updating a user portrait in combination with user interaction history and current modal characteristics; in the question and answer generation process, current input, a user portrait and a historical graph are fused, intermediate semantic representation is generated through a context enhancement module, and the intermediate semantic representation is combined with a knowledge base to generate answers; meanwhile, a modal confidence degree dynamic evaluation mechanism is introduced, and weights are distributed according to the input quality, the user adaptation degree and the context correlation; and finally, incremental optimization is carried out on the graph structure, the user portrait and the question and answer strategy through a user feedback driving system, and multi-round, multi-mode and self-adaptive intelligent question and answer interaction is realized.

The invention relates to an intelligent operation and maintenance question-answering system for cable manufacturing equipment, and belongs to the technical field of computer systems based on specific calculation models. The system comprises an edge data acquisition module, a predictive map construction module, a semantic perception module, a question-oriented reasoning module and a question and answer generation module. According to the system, multi-dimensional real-time data in the operation process of equipment is collected and structurally processed, a process knowledge graph is constructed in combination with industry knowledge, and the causal relationship and reasoning parameters in the graph are dynamically updated according to the data trend. A semantic perception module is used for recognizing the problem intention of a user, a semantic weight vector is formed to guide the reasoning process, and a problem-oriented reasoning module is made to execute joint reasoning on the basis of combining real-time data and a knowledge graph and generate an explanatory conclusion. Finally, operation and maintenance suggestions with high readability are output through a question and answer generation module, and the targets of equipment fault intelligent diagnosis, process optimization and man-machine efficient interaction are achieved.

The invention discloses a double-engine government affair question and answer method based on large model fine tuning and RAG retrieval, belongs to the field of government affair digitization and natural language processing, and combines large model language understanding generation ability, retrieval enhancement generation technology and a structured reasoning mode. The defects of a traditional government affair question and answer method in the aspects of dynamic policy response, complex semantic understanding and compliance control are overcome. Government affair field knowledge is adapted through large-model fine adjustment, and high-precision and timeliness answering of government affair consultation is realized in combination with vector retrieval and a dynamic updating mechanism. The core innovation of the method lies in deep fusion of a double-engine architecture and dynamic knowledge management, the accuracy and response efficiency of government affair questions and answers are improved on the premise of ensuring policy compliance, and the method is suitable for intelligent upgrading of scenes such as government affair service halls and online consultation platforms.

Disclosed is an industrial question answering model training method based on reinforcement learning and knowledge base matching, comprising the following steps: S1, collecting professional knowledge questions and answers in an industrial field to construct an industrial knowledge base, training a reward model, carrying out, for industrial knowledge questions and answers, matching comparison on outputs of an industrial question answering model and content of the industrial knowledge base, and obtaining reward values on the basis of similarities; S2, sorting the reward values, and using a sorting loss function to train and update parameters of a reward model network; and S3, carrying out industrial question answering model training, incorporating a penalty term for the reward values, and using a reinforcement learning algorithm to train the industrial question answering model multiple times to obtain an optimal strategy. According to the industrial question answering model training method based on reinforcement learning and knowledge base matching of the present invention, the reinforcement learning algorithm is used, and iterative training is carried out multiple times, thereby helping the industrial question answering model to learn and understand industrial professional knowledge and improving the question answering accuracy of the industrial question answering model.

The invention discloses an electric power field knowledge question-answering system construction method based on a large language model, and relates to the field of electric power field knowledge question-answering, and the method comprises the steps: judging the data type of electric power field knowledge, and carrying out the processing of the electric power field knowledge according to the judgment result through matching with a processing technology, and generating an entity relation triple; constructing a power field knowledge graph; optimizing the power field knowledge graph based on the attention network, outputting an answer causal path of the fault problem by using the optimized power field knowledge graph, and marking a confidence score of the answer causal path; and inputting the solution causal path and the confidence score into a language model to obtain a fault question answering result, and optimizing the question answering result according to the consistency of the fault question answering result and the power field knowledge graph. According to the method, on the premise that the fault diagnosis logic is rigorous and the result is traceable, knowledge in large-scale unstructured literatures in the power industry is activated, so that accurate question and answer services can be provided for operation and maintenance personnel in real time.

The invention discloses a reverse question guide question answering implementation method and system, and belongs to the technical field of artificial intelligence and natural language processing. Context-aware intention dynamic correction is realized through a three-level intention classification system, and cross-modal knowledge matching is realized by adopting a distributed semantic index technology; based on the reinforcement learning strategy, optimizing a cooperative work mechanism of the dialogue strategy and the knowledge base; comprising the steps of intention recognition: analyzing a session of a user by using an intention recognition model, and constructing a three-level intention classification system based on deep semantic understanding, including main class recognition, fine-grained analysis and context perception; question rewriting: constructing a dynamic rewriting engine to rewrite the user question; recalling and cleaning multi-source item knowledge; generating a reverse question; locking items and acquiring item data; generating questions and answers. According to the method, the robustness, the real-time performance and the scene adaptation capability of a professional question answering system can be improved, and the government affair service question answering accuracy, the intention recognition precision and the cross-region recommendation adoption rate are improved.

The invention relates to a low-altitude intelligent question and answer construction method and system based on dynamic parameters. The method comprises the following steps: collecting low-altitude domain data, cleaning the low-altitude domain data, generating a semantic vector index, and constructing a low-altitude domain knowledge base based on the semantic vector index; receiving a natural language query of a user, analyzing a query intention, extracting keywords in the natural language query, and matching a corresponding candidate word quantity based on query types of the natural language query of the user, the query types at least comprising high-frequency phrase query and low-frequency long-tail query; and respectively carrying out fusion semantic retrieval and keyword retrieval, carrying out secondary sorting on the candidate results based on a preset resorter, preferentially sorting the candidate results related to the query intention, and outputting the corresponding candidate results. By adopting the method, a combined domain retrieval enhancement generation mechanism is provided, so that the professionality and accuracy of answers are improved; and the retrieved knowledge base content is re-screened to increase the hit probability of the knowledge base.

The invention discloses an inquiry model training method and device based on a long thinking chain, and relates to the field of large models, semantic information is extracted through strategy network analysis of a model, and an initial step decision is generated in combination with context information in a historical memory library; sending the initial step decision into a reasoning path generator, and reasoning to generate a primary diagnosis disease source and an intermediate diagnosis step; sending the primary diagnosis source and the intermediate diagnosis step into a verification module, performing pathological logic verification according to a case diagnosis report and a medical knowledge base, and feeding back a verification result; the reasoning path generator updates the historical memory bank based on the feedback result, the preliminary diagnosis disease source and the intermediate diagnosis steps; the strategy network continues reasoning based on user feedback input and the updated context information in the historical memory bank, and finally an inquiry result is output. According to the scheme, technical means such as reinforcement learning, self-adaptive backtracking and memory enhancement are introduced into a long thinking chain reasoning framework, so that a large language model realizes multi-aspect comprehensive improvement in medical question and answer and auxiliary diagnosis scenes.

The invention relates to a government affair file information extraction and question answering method and device and a medium, and the method comprises the steps: carrying out the entity extraction of a government affair file through employing a BERT-CRF joint model, and obtaining a structured entity set; performing relation extraction on the structured entity set to generate a semantic relation set between the entities; constructing a knowledge graph according to the structured entity set and the semantic relationship set, storing entity nodes into a graph database, and storing an embedded vector of an entity text into a vector database; when a query request of a user is received, relation query of the graph database and semantic retrieval of the vector database are carried out, sub-graph structures and semantic matching vectors related to query are extracted, and a mixed retrieval result is obtained; and inputting the mixed retrieval result into a large language model, and generating a question and answer response text conforming to a preset format by applying a dynamic prompt template. According to the method, the document processing efficiency and accuracy are effectively improved, and a solid technical support is provided for intelligent management of government affair documents.

The invention provides a traditional Chinese medicine knowledge question-answering system based on a fine-tuning large model and dual retrieval enhancement. The traditional Chinese medicine knowledge question-answering system comprises a large model fine-tuning module, a dual retrieval enhancement module, a prompt template module and an answer generation module. The large model fine tuning module constructs a high-quality corpus by using traditional Chinese medicine ancient books, clinical cases and the like, performs incremental pre-training and supervised fine tuning on a ChatGLM3-6B pre-training model, and adopts technologies such as low-rank adaptation (LoRA) and direct preference optimization (DPO) to improve the adaptability of the model to traditional Chinese medicine professional knowledge. The dual retrieval module enables the model to map user questions to related contents in traditional Chinese medicine classical literatures, guidelines and modern literatures through text retrieval based on a LangChain framework on one hand, and provides structured background knowledge through map retrieval on the other hand. The system effectively makes up for the knowledge blind area of a large-scale general model in the field of traditional Chinese medicine, realizes the improvement of the accuracy, continuity and speciality of question and answer results, and has wide application prospects and relatively high innovativeness.

The invention relates to the technical field of electric power engineering, and discloses an electric power engineering multi-modal RAG system based on a knowledge graph and multi-Agent collaboration, and the system comprises a multi-modal dynamic knowledge base construction module which is configured to carry out the structural processing, multi-dimensional knowledge organization and dynamic optimization of electric power engineering multi-modal data; the self-adaptive retrieval strategy engine module is configured to construct a weight decision network based on deep reinforcement learning and execute multi-channel parallel retrieval and result fusion; the iterative self-reflection reasoning module is configured to generate a reasoning path in combination with the retrieval result and verify evidence validity from multiple dimensions; the MCP tool intelligent calling module is configured to integrate multiple types of standardized MCP tools; and the multi-Agent collaborative framework is configured to provide multiple types of Agents which are specific in function and have a cross-module interaction capability. According to the method, the question and answer accuracy, the reasoning depth and the result interpretability in the complex multi-modal scene of the electric power engineering can be remarkably improved.

The invention relates to the technical field of intelligent retrieval, in particular to an enterprise RAG intelligent question answering system based on vector database reordering. The system specifically comprises: a document recall module, which retrieves a vector database to obtain candidate document blocks containing business metadata; the comprehensive scoring module is used for calculating a semantic correlation score by adopting a later-stage interaction architecture based on bidirectional token importance weighting, performing path semantic matching and context sensing rule evaluation according to a preset metadata ontology graph to obtain a service attribute score, and analyzing the evidence sub-graph to obtain a fact path score; fusing the semantic correlation score, the service attribute score and the fact path score to generate a comprehensive correlation score; and the sorting output module performs optimization resorting based on a preset punishment mechanism and the comprehensive correlation score to generate an optimized context set, and calls a generation model to output answers based on the optimized context set. According to the method, semantic accuracy, business compliance and fact reliability can be considered, and more trustworthy high-quality enterprise-level answers can be generated.

The invention discloses a multi-source knowledge-enhanced big language model question and answer method, device and equipment and a medium, and the method comprises the steps: inputting a user question into a big language model, and obtaining an initial answer and a cooperation strategy deduced by the big language model for the user question; after taking the initial answer as a current candidate answer, extracting a target knowledge source type in a collaborative strategy and generating a corresponding calling expression; according to the calling expressions, target knowledge sources matched with the target knowledge source types are called respectively, and an enhanced retrieval document set is obtained; re-inputting the user question and the enhanced retrieval document set into the large language model to obtain a new answer and a collaboration strategy; and taking the new answer as the current candidate answer, repeating the operation of extracting the target knowledge source type, and determining a target answer from all the candidate answers as a feedback result when an iteration ending condition is met. According to the technical scheme, the question and answer accuracy can be optimized through iterative retrieval and knowledge enhancement.

The invention provides a minority language professional question and answer method based on mixed retrieval and retrieval enhancement generation, which comprises the following steps: S1, constructing a multi-language knowledge constructing a vector index database and a term knowledge graph by using a multi-language model according to a related minority language document; s2, multi-layer mixed retrieval: cross-language document recall is realized through a multi-layer mixed retrieval module, and the multi-layer mixed retrieval module is composed of keyword retrieval, semantic retrieval and vector retrieval; s3, answer generation: performing answer generation through an adaptive multi-language model by using a retrieval enhancement generation module, and introducing rule constraint decoding and a dynamic attention mechanism in the generation stage to improve the professionality and accuracy of the answer; and S4, self-adaptive optimization and knowledge updating: through a user feedback reinforcement learning module, optimizing the model based on user error correction data and supervising updating of the knowledge base. According to the method, professional questions and answers of the minority language can be realized based on a small amount of professional data of the minority language, the recall rate of the document of the minority language is improved, and the generation quality is improved.

The invention provides a medical multi-hop text question and answer retrieval enhancement method based on heterogeneous knowledge. The method comprises the following steps: firstly, constructing a uniform heterogeneous graph structure based on a medical knowledge graph of a medical document, and establishing a semantic bridge through an entity-document mapping relationship; performing semantic decomposition on a complex medical problem input by a user by utilizing the large model, and iteratively generating a series of mutually independent atomic queries; searching a reasoning path in the entity sub-graph of the heterogeneous graph, and calculating a path score by fusing the weighted combination of the entity association text similarity, the entity matching degree and the path edge weight; training a retriever by adopting a marginal sorting loss function, and optimizing a retrieval effect through positive and negative sample comparative learning; and finally, calling a large model to convert the reasoning path with the highest score into a text, and extracting a document fragment corresponding to a path node. According to the method, the problems that an existing retrieval enhancement technology is insufficient in complex problem processing capacity, poor in reasoning interpretability and the like are effectively solved, and high-accuracy medical questions and answers are achieved.

The invention discloses a multi-modal fused intelligent question answering and knowledge retrieval method and system, and the method comprises the steps: building a multi-modal data index model oriented to a heterogeneous knowledge source, carrying out the feature mapping of text, image, table, chart, audio and video contents through a unified semantic embedding space, and generating a cross-modal index set; after a query request is received, performing semantic matching and structure matching on the cross-modal index set by using a multi-channel retriever to obtain candidate evidence fragments; and based on an evidence granularity decomposition strategy, performing minimum evidence unit division on text statements, table units, chart data points and multimedia frame contents in the candidate evidence fragments, and establishing a semantic consistency graph among the units. According to the method, high-credibility traceable generation of question and answer results is realized through multi-modal fusion and space-time consistency constraint, and the retrieval precision and interpretation transparency in a complex knowledge scene are remarkably improved.

The invention discloses an agent question-answering method, device, equipment and product based on knowledge graph recall, and relates to the technical field of large models, agents, artificial intelligence and knowledge graphs. The agent question-answering method comprises the steps that a target question input by a user in an intelligent interaction page is obtained, keywords in the target question are extracted, and the keywords are extracted; a target knowledge graph node matched with the keyword is determined in a target knowledge graph, a target sub-graph is determined at least based on the target knowledge graph node, and the target knowledge graph comprises a plurality of knowledge graph nodes; and generating an answer corresponding to the target question at least based on the target question and the target sub-graph through the question and answer large model. By introducing the knowledge graph, in the process of generating the answer by the question and answer large model, deep reasoning can be carried out based on the input question along the path in the knowledge graph by using the multi-level and structured information of the knowledge graph, so that the processing capability of the question and answer large model on the complex question is improved, and the accuracy and comprehensiveness of outputting the answer are effectively improved.

The present application relates to the technical field of artificial intelligence, and in particular to a question answering method based on a large model, and an electronic device. The electronic device comprises a communication interface, a memory, and a processor, the memory is configured to store a computer instruction, and the processor is configured to execute a computer program to enable the electronic device to: determine, on the basis of each pre-stored text block, a target text block matched with a question to be answered; obtain, if non-text data including a picture and / or a table is pre-stored for the target text block in advance, stored summary text of the non-text data; and input the question to be answered, the target text block, and the summary text into a first large model on the basis of a preset format to obtain answer information. Since the summary text is text that summarizes all content in the non-text data, the first large model does not need to process the picture and / or the table and still can ensure that the content recorded in the picture and / or the table is taken into consideration during generation of the answer information, thereby improving the model-based question answering accuracy.

The invention relates to the field of knowledge graph completion, provides a knowledge graph completion method based on multi-modal visual angle perception and a deep neural network, and aims to solve the problems of weak multi-modal information expression ability, rough fusion mode and insufficient structural reasoning ability in the prior art. According to the method, structure information, text description and visual image information of an entity in a knowledge graph are obtained, structure, text and image modal input is constructed respectively, and a graph neural network, a pre-training language model and a visual encoder are adopted for feature coding; weighted fusion and semantic enhancement of multi-modal features are realized through a visual angle fusion mechanism and hierarchical attention processing; cross-modal contrast learning is introduced to improve modal consistency; and carrying out triple reasoning by using a uniform Transform encoder, and verifying a completion result by scores. According to the method, multi-modal semantics are effectively integrated, the entity representation capability and the triple prediction accuracy are improved, the model robustness is enhanced, and the method is suitable for application scenes such as intelligent question answering and recommendation systems and has remarkable practical value and popularization prospects.

The invention relates to the technical field of artificial intelligence, and discloses an AI intelligent auxiliary question answering system for student practice. According to the system, various interactive data such as texts, voices and handwritten formula images of students are collected through a multi-modal data acquisition module, semantic feature vectors of all modals are generated through a multi-dimensional feature analysis module, and then the semantic feature vectors are mapped to a unified semantic space through a cross-modal fusion module. The double-layer graph construction module constructs a domain knowledge graph and a learning behavior graph, and the dynamic reasoning and recommendation module generates a personalized problem solving path recommendation and knowledge point completion strategy based on fusion semantic representation and a graph library. In addition, the system also has the functions of interactive behavior log anomaly detection, path backtracking, knowledge forgetting curve prediction and the like. The system can comprehensively understand questions of students, provides personalized question answering service, and effectively improves the learning efficiency and knowledge mastering degree of the students.

The invention discloses a long video multi-modal understanding and question-answering method and system based on large model and retrieval enhancement generation. The method comprises the following steps: 1) a multi-modal feature extraction module; 2) a multi-modal synchronization and alignment mechanism; 3) constructing a structured memory pool; 4) querying a drive generation mechanism; 5) incremental updating and memory compression strategy; and 6) unifying the multi-modal representation space. The invention provides a long video multi-mode understanding method fusing a large language model and retrieval enhancement generation, and aims to break through the limitation of a traditional method in the aspects of single-mode processing and semantic fragmentation. According to the method, video image features are extracted through a visual model (such as YOLO and ViT), voice transcription and environment voice description are obtained in combination with an audio model (such as Whisper and Qwen-Audio), and unified coding of vision, voice and audio in a long video is achieved. Then, a structured memory pool is constructed through semantic consistency segmentation and timestamp alignment technologies to store time slice data of different modalities.

The invention relates to a multi-mode large language model fine tuning method, system and device and a medium, and belongs to the technical field of artificial intelligence and computer vision crossing. The fine tuning method comprises the steps that an original business scene image is acquired and preprocessed, and a preprocessed image is obtained; performing bounding box coordinate labeling and semantic label definition on the entity target in the preprocessed image through a labeling tool, and outputting a structured labeling file; based on the preprocessed image and the structured annotation file, constructing a training sample set comprising multiple rounds of image-text dialogues; loading the pre-trained multi-modal large language model, configuring low-rank matrix decomposition parameters, and generating a fine tuning instruction set; and inputting the training sample set into a pre-trained multi-modal large language model, carrying out joint training operation based on the fine tuning instruction set, and outputting the fine-tuned multi-modal large language model. According to the method, the identification accuracy, the interaction capability and the system availability of the visual question-answering system in an actual application scene are improved.