592 results about "Text retrieval" patented technology

The invention provides a cross-modal remote sensing image-text retrieval method based on multistage semantic collaborative matching, which comprises the following steps of: extracting a region of interest by using a semantic segmentation algorithm through an image preprocessing module, segmenting a remote sensing image into a plurality of regions and generating image blocks; respectively extracting global features, regional features and pixel-level features of the image, and carrying out fine-grained coding on key regions such as fine-grained ground feature edges and the like; the text multi-level coding module is used for carrying out three-level feature coding of documents, sentences and words on texts based on a pre-training language model to ensure multi-level understanding of the texts; in the multi-level matching and fusion module, the similarity between the remote sensing image and the text description is calculated through a cross attention mechanism, weighted fusion is carried out on features of all levels, and finally a retrieval score is output. The method not only improves the accuracy and robustness of image-text retrieval, but also can be widely applied to the fields of remote sensing monitoring, environment change recognition, geographic information systems and the like.

The invention discloses a knowledge graph-driven water conservancy intelligent question and answer interaction platform and method, and relates to the related field of data processing, and the platform comprises an entity disambiguation module which is used for constructing a water conservancy knowledge graph to carry out standardized modeling, generating structured knowledge nodes to carry out entity disambiguation on water conservancy objects, and generating entity identifiers; the full-text retrieval module is used for performing semantic analysis on a natural language query text of the user according to the entity identifier, identifying a query intention type, performing full-text retrieval and generating a multi-modal answer; and the iterative interaction module is used for performing dynamic questioning based on the multi-modal answers, generating user semantic feedback information, optimizing a query path according to the user semantic feedback information, and realizing iterative interaction of water conservancy knowledge questions and answers. The technical problem that existing water conservancy intelligent question and answer interaction is insufficient in question and answer accuracy and richness is solved, and the technical effect of improving the question and answer accuracy and richness is achieved.

The invention provides a traditional Chinese medicine knowledge question-answering system based on a fine-tuning large model and dual retrieval enhancement. The traditional Chinese medicine knowledge question-answering system comprises a large model fine-tuning module, a dual retrieval enhancement module, a prompt template module and an answer generation module. The large model fine tuning module constructs a high-quality corpus by using traditional Chinese medicine ancient books, clinical cases and the like, performs incremental pre-training and supervised fine tuning on a ChatGLM3-6B pre-training model, and adopts technologies such as low-rank adaptation (LoRA) and direct preference optimization (DPO) to improve the adaptability of the model to traditional Chinese medicine professional knowledge. The dual retrieval module enables the model to map user questions to related contents in traditional Chinese medicine classical literatures, guidelines and modern literatures through text retrieval based on a LangChain framework on one hand, and provides structured background knowledge through map retrieval on the other hand. The system effectively makes up for the knowledge blind area of a large-scale general model in the field of traditional Chinese medicine, realizes the improvement of the accuracy, continuity and speciality of question and answer results, and has wide application prospects and relatively high innovativeness.

According to the knowledge text retrieval method and device, the storage medium and the computer equipment, when knowledge text retrieval is carried out, the retrieval statement and the target knowledge document corresponding to the retrieval statement are obtained firstly, the document structure tree of the target knowledge document is determined, the document structure tree is of a tree structure constructed by utilizing the directory structure of the target knowledge document, and the retrieval statement is stored in the document structure tree; according to the method, a target knowledge document is taken as a component of the target knowledge document, an index structure between each index node in the component of the target knowledge document and the target knowledge document is recorded, and leaf nodes in the index nodes are connected through pointers to form a linked list structure. The corresponding index node can be dynamically retrieved based on the association relationship between the index structure and the linked list structure of the tree structure, and the context content corresponding to the retrieval statement in the target knowledge document is intercepted by adopting different modes based on the type of the index node. And omission of unrelated contents of other chapters and key contents of the same chapter is avoided, so that the retrieval precision and quality are improved.

The invention discloses an RAG performance improvement method based on process optimization. The method comprises the following steps: S1, preprocessing text data; s2, performing hierarchical text segmentation, obtaining a document, storing the text by utilizing a database after the document is segmented, converting the text into vectors, and storing the vectors into a vector knowledge base; s3, querying and rewriting to obtain a rewritten question; s4, combining parent document search with mixed retrieval, and retrieving related contexts for the rewritten questions; s5, performing rearrangement and denoising to obtain a final context; and S6, generating an answer, and sending the final context into a large language model for understanding questions and answers. According to the method, the links of text preprocessing, text partitioning, text embedding, query optimization, retrieval, retrieval post-processing, generation and the like are sequentially optimized according to the data flowing direction, the semantic retrieval precision is improved, and the performance of RAG in the aspects of text retrieval precision, answer generation quality, user interaction and the like can be remarkably improved; and the flexibility and the practicability of the system are enhanced, so that the system is more excellent in practical application.

The invention discloses an image report generation method, system and device based on multi-modal learning optimization, and the method comprises an image retrieval module which is used for carrying out the image feature extraction of an input medical image, and obtaining similar medical images and reports; the simulation generation module is used for guiding an image report generation model to generate an initial draft report by utilizing the medical image and the report; the personalized diagnosis module is used for carrying out personalized adjustment on the report generation model according to the historical report preference of the doctor or the historical medical data of the patient; the reinforcement learning optimization module is used for dynamically adjusting a text generation mode based on a reinforcement learning report optimization strategy and optimizing an image report generation model in combination with doctor scoring, automatic evaluation and clinical feedback; and the iterative refining module is used for performing text retrieval by using the initial draft report as a query, acquiring text fragments which are semantically similar to the initial draft report, and performing neural symbol reasoning optimization to generate a final report.

The invention provides a multimodal reasoning method based on a Monte Carlo tree and dynamic retrieval. The method is applied to an artificial intelligence assistant-assisted teacher paper judgment scene. Specifically, the framework comprises a unified retrieval module, wherein the retrieval module comprises a mixed modal retrieval corpus and a multi-modal retrieval module; the multi-modal retrieval module comprises two retrieval modules, namely a text retrieval module and a cross-modal retrieval module, key problem solving insights are selected by adopting knowledge concept filtering, and progressive multi-modal reasoning annotations of multi-modal reasoning are automatically obtained by utilizing Monte Carlo tree search and an active retrieval mechanism for data annotation; and gradually aligning a process reward model through course process reward modeling in two stages by utilizing the annotation data so as to realize fine-grained verification of multi-modal large model reasoning. An MCTS algorithm and an active retrieval strategy are combined to improve multi-modal reasoning. The framework can automatically obtain high-quality step-by-step reasoning annotations, so that a process reward model is aligned step by step, and finally reliable automatic multi-modal reasoning verification is realized.

The invention discloses a text retrieval enhancement generation method and device, a medium and equipment, and relates to the technical field of retrieval enhancement generation. Comprising the following steps: establishing a database through vectorized corpora, and screening and labeling key information to generate a golden section vector library; extracting an adversarial sample from an external database by using a training problem, and fusing the adversarial sample with related corpora in a golden section vector library to obtain a training set; and adding a noise classification layer to improve the language large model, and training the improved language large model by using the fusion training set. When a user question is processed, relevant corpora are recalled, noise is recognized through the classification layer, context representation is generated through the attention fusion layer, and the output layer dynamically adjusts a decoding strategy and generates an answer. According to the method, a large language model added with a noise classification task is trained through the fusion training set, so that the model can identify and distinguish different types of noise, the adaptability of the model to different types of noise is enhanced, the model is helped to learn how to distinguish related and unrelated information, and thus the method is more robust when an actual problem is processed.

The invention provides a retrieval method and device based on document segmentation and a document retrieval system. The method comprises the following steps: acquiring a to-be-segmented document; based on an NLP algorithm, calculating the semantic similarity between the partial texts of the to-be-segmented document to obtain a first semantic relevancy; according to the first semantic relevancy of all the partial texts, the document to be segmented is segmented, a plurality of semantic text blocks are obtained, and each semantic text block comprises at least one partial text; under the condition that a query request is received, calculating semantic similarity between a query text corresponding to the query request and each semantic text block based on an NLP algorithm to obtain a plurality of second semantic relevancy, and determining the semantic text block with the highest second semantic relevancy of the query text corresponding to the query request as a target semantic text block, and displaying the target semantic text block in a display interface. According to the scheme, the problem that in the prior art, the accuracy rate is low during text retrieval is solved.

The invention relates to an enhanced LLM-RAG multi-hop question and answer method based on logic tree reasoning, and belongs to the technical field of new-generation information, and the method comprises the following steps: inputting a multi-hop question and answer question into a computer system; the computer system calls a pre-training large language model LLM, the multi-hop question-answer question is decomposed into a hierarchical logic tree in a recursive mode, and each node of the logic tree comprises a sub-question and a corresponding hypothesis answer; performing image retrieval from a structured knowledge source Wikidata and performing text retrieval from an unstructured knowledge source Wikipedia on the basis of each node sub-question and the hypothesis answer to obtain corresponding evidence; traversing the logic tree, verifying the consistency between the hypothetical answer of each node and the evidence through LLM, if the contradiction exists, reconstructing the corresponding sub-tree, and dynamically correcting the reasoning path; and integrating the verified logic tree node information, and outputting an accurate answer to the multi-hop question and answer question.

The invention discloses a multi-mode fusion product document and source code association retrieval method based on a knowledge graph, and relates to the technical field of software engineering and artificial intelligence. The method comprises the steps that source codes are preprocessed, and code structure information and business semantics are mapped in combination with a predefined business term dictionary; performing controlled induction on a code file and a product document by utilizing a large model, extracting business terms, logic intentions and a subject relationship, fusing with original codes, and establishing a vector retrieval index system; further analyzing a code structure by using an abstract syntax tree, and extracting an entity and a calling relationship; semantic enhancement and relation normalization are performed in combination with the large model, entities and relations are stored in a graph database, and a knowledge graph is formed; and performing parallel processing on user query based on a full-text retrieval index, a vector retrieval index system and a knowledge graph, and finally generating a product concept. According to the method, the retrieval speed, the semantic depth and the logical reasoning ability can be considered at the same time, and the retrieval accuracy is improved.

The invention relates to the technical field of data processing, and discloses a multi-modal data alignment method and system based on an iterative Riemannian manifold, a terminal and a medium, and the method comprises the steps: obtaining a multi-modal data sample set for cross-modal image text retrieval, and creating a multi-modal data alignment network training model; performing iterative Riemannian manifold training on the multi-modal data alignment network model according to the multi-modal data sample set to obtain a multi-modal data alignment model; and obtaining to-be-aligned multi-modal data, inputting the to-be-aligned multi-modal data into the multi-modal data alignment model, and outputting a multi-modal data alignment result. According to the method, different modal data are embedded into the same Riemannian manifold, and the geometrical characteristics of the Riemannian manifold are kept by utilizing the Riemannian curvature regularization, so that the embedded representation can better reflect the internal structure of the data, cross-modal alignment is carried out based on the optimal transmission of the manifold geodesic distance, more accurate and deeper semantic correspondence is realized, and the accuracy and the reliability of the data are improved. Therefore, the multi-modal data alignment effect is obviously improved.

The invention provides an intelligent text retrieval method and system, a storage medium and a program product, and relates to the field of intelligent text retrieval, the method comprises the following steps: segmenting each original document into a plurality of semantic segments according to semantic similarity; extracting a text feature of each semantic fragment to generate a text fragment vector; selecting a related fragment from each text fragment vector, wherein the first similarity between the related fragment and the retrieval vector is greater than a preset first threshold value; calculating a second similarity between each related fragment and the adjacent fragment; combining the adjacent segments with the second similarity greater than a preset second threshold value with the corresponding related segments to obtain stitching segments; based on a retrieval result corresponding to a historical retrieval request of a user, determining a retrieval granularity which is firstly displayed; and according to the retrieval granularity, adjusting the display content of the stitching fragment, and returning a text retrieval result. By implementing the method, the semantic integrity and the result relevancy of the text retrieval result can be improved.

The invention discloses a text retrieval method and device, equipment, a storage medium and a product. The method comprises the steps of obtaining a question input by a user and at least one candidate text; for each candidate text, determining semantic similarity and keyword similarity between the question and the candidate text; sentence features of the questions and document features of the candidate texts are extracted respectively, and the weight of semantic similarity and the weight of keyword similarity are determined according to the sentence features and the document features; based on the weight of the semantic similarity and the weight of the keyword similarity, according to the semantic similarity and the keyword similarity of the question and each candidate text, calculating the comprehensive similarity of the question and each candidate text; at least one candidate text with the maximum comprehensive similarity is screened out, and an answer candidate set is obtained; and inputting the answer candidate set and the question into a preset large language model, and outputting a retrieval result. According to the method, the accuracy and efficiency of long text retrieval can be improved.

The invention discloses a water conservancy design archive retrieval system and method based on a local lightweight large model, and the method comprises the steps: S1, constructing a Python automatic preprocessing assembly line, extracting texts for PDF and Word multi-format archives, correcting metadata, and outputting standardized data; s2, constructing a full-text retrieval and semantic retrieval dual-mode cross-document retrieval service by relying on a Weavi ate local vector database and a lightweight text embedding model; s3, analyzing a user query intention through a local large model, synchronously triggering metadata accurate retrieval and content semantic retrieval, and generating a structured result; and S4, integrating the core module into a local area network Web platform, adopting Docker containerization deployment, and combining an RBAC permission model and JWT authentication to guarantee security. The system comprises a preprocessing module, a cross-document retrieval module, an intelligent agent module and a background management module, and collaboration is achieved through a standardized API. According to the method, the problem of archive fragmentation is solved, multi-mode retrieval breaks through keyword limitation, an intelligent agent reduces manual intervention, a localized architecture prevents secret-related leakage, background management adapts to an existing I T environment, and full-process intelligent archive service is provided for water conservancy design.

The invention relates to a retrieval method and device based on static word embedding, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of performing word segmentation on an original text to obtain a first word segmentation result, training a static word embedding model by utilizing the first word segmentation result to obtain word vectors, and generating a synonym word library; the method comprises the following steps: establishing a full-text inverted index by utilizing an original text, and expanding query words by utilizing a synonym library in a retrieval stage; encoding the original text into a semantic vector by using a semantic generation model, and constructing a vector index based on the semantic vector; based on a to-be-queried text in a user query request, performing retrieval by using the full-text inverted index to obtain a first candidate document, and performing retrieval by using the vector index to obtain a second candidate document; performing fusion processing on the first candidate document and the second candidate document to obtain a target candidate document; and inputting the target candidate document into the text generation model to obtain a retrieval result. By adopting the method, the accuracy of text retrieval can be improved.

The invention discloses a file retrieval method, device and equipment based on multi-modal AI and a storage medium, relates to the technical field of data retrieval, and aims to solve the problem that traditional file retrieval is low in efficiency and precision. The method comprises the following steps: performing content analysis on a multi-modal file including a picture file, a video file and a document file of text and / or visual information, and extracting a text semantic feature, a picture visual feature and a video time sequence feature; mapping the text semantic feature, the picture visual feature and the video time sequence feature to a cross-modal semantic space used for representing a high-dimensional vector associated with different modal features, and generating a cross-modal association vector; according to the modal type of the retrieval information, a corresponding retrieval module in a mixed retrieval engine is called to conduct retrieval in a cross-modal semantic space, an initial retrieval result is obtained, and the mixed retrieval engine comprises a text retrieval module, a visual retrieval module and a cross-modal fusion module; and performing dynamic weight distribution sorting on the initial retrieval result, and outputting a target matching result.

The invention discloses a remote sensing cross-modal retrieval method and system based on large model fine tuning. The method comprises the following steps: S1, inputting an obtained remote sensing image and / or description text into a trained remote sensing image text retrieval network for processing to obtain image features and / or text features after cross-modal fusion; s2, acquiring a text and / or an image matched with the remote sensing image and / or the description text based on the image features and / or the text features, and outputting the text and / or the image; the step S1 comprises: extracting initial image features and / or initial text features from the remote sensing image and / or the description text through an image text encoder; performing cross-modal fusion processing on the initial image features and / or the initial text features through a cross-modal asymmetric adapter to obtain image features and / or text features after cross-modal fusion; and carrying out optimization processing on the image features and / or the text features through a dual-task consistency loss function.

The invention discloses a retrieval enhancement question and answer generation method and system based on a large language model in combination with a multi-agent cooperation mechanism. The method comprises the steps of S1, text preprocessing and semantic difference enhancement and amplification; s2, checking and complementing user questions; s3, evaluating problem complexity and selecting a generator; s4, text retrieval and answer generation; the system comprises an information enhancement processing module, a query understanding optimization module, a question complexity intelligent evaluation module and a collaborative answer generation module, and is used for realizing the method. According to the method, a multi-agent collaborative RAG framework of an information enhancement agent, an interaction analysis agent and a problem complexity evaluation agent is introduced, and semantic difference enhancement and amplification, query integrity verification and complementation and a dynamic generation strategy based on problem complexity are performed after hierarchical document partitioning are combined; and the performance of the system in the aspects of semantic distinguishing capability, query understanding precision, generation efficiency and reliability is comprehensively improved.

The invention relates to the technical field of natural language processing, in particular to a knowledge base-based self-adaptive hybrid retrieval enhanced question and answer method and device.The method comprises the following steps of: extracting entities in an initial text to form a triple; converting the structured information of the triple into text retrieval information and vector retrieval information, and constructing a basic data source; presetting a query question level, judging the complexity of a to-be-queried question, and if the to-be-queried question is a first question, directly obtaining an answer by utilizing semantic matching; if the to-be-queried question is a second question, extracting a keyword of the to-be-queried question, and obtaining structured data of entities and relationships in the basic data source based on the keyword to obtain a retrieval result; and reordering based on a retrieval result to generate a question answer. The retrieval path is adaptively selected according to the depth of the problem in the retrieval process, so that the efficiency of simple query is improved, and deeper graph structure support is provided in complex query.

The invention discloses a large model retrieval enhancement generation method and system, and the method comprises the steps: obtaining the comprehensive document data of a power grid field, carrying out the partitioning processing of the comprehensive document data, and obtaining the partitioned text data; performing deep analysis on the block text data to generate a thinking chain enhanced text; encoding the thinking chain enhanced text based on a hierarchical weighted encoding method to generate a thinking chain enhanced semantic vector; inputting the block text data into a pre-trained large language model to obtain a generated task perception representation vector; splicing the thinking chain enhanced semantic vector and the generated task perception representation vector to obtain a fusion vector, and constructing a vector database in the power grid field based on the fusion vector; responding to the text retrieval signal, obtaining question text data of the target user, analyzing the question text data to obtain a question fusion vector, comparing the question fusion vector with a vector database, and obtaining retrieval data of the question text data based on a comparison result.

The invention provides a full-text retrieval method and system fusing various types of documents, and relates to the technical field of information retrieval, and the method comprises the following steps: obtaining document representation through document content extraction and structure recognition, generating a cross-modal semantic vector by using word embedding and nonlinear transformation, constructing a hierarchical index and a cross-document association graph, and obtaining a full-text retrieval result; the basic correlation score is calculated after the query request is received, and the comprehensive score of the candidate content segments is calculated based on the association graph to determine the optimal retrieval result, so that unified representation and retrieval of heterogeneous documents are realized, the cross-document retrieval precision and relevance are improved, and the processing capability of a retrieval system on complex queries is enhanced.

The invention relates to a hierarchical retrieval method and device based on multi-modal questions and answers, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of receiving multi-modal input data, wherein the multi-modal input data comprises image data and corresponding to-be-answered question data; performing encoding processing on the document data and the multi-modal input data in the preset knowledge base to obtain a feature vector of the document data in the preset knowledge base and an image feature vector of the multi-modal input data; according to feature vectors of document data in a preset knowledge base, performing at least one time of text retrieval in the preset knowledge base for the image feature vectors to obtain a text retrieval result; and integrating the text retrieval results into an input sequence, inputting the input sequence into the multi-mode question and answer model, and generating a corresponding answer text of the to-be-answered question data. By adopting the method, the retrieval effect and the knowledge integration efficiency can be improved, and the performance and the flexibility of the multi-modal question-answering system are enhanced.

The invention relates to a network security intelligent analysis method and device based on big data, equipment and a medium, and the method comprises the steps: collecting log data and flow data through a log collector and a flow collector, carrying out the normalization processing of the log data, filtering repeated logs, standardizing the format, carrying out the full-protocol auditing of the flow data, and carrying out the real-time auditing of the flow data. Restoring and encrypting the traffic log; storing the processed log data and flow data in a distributed manner, and constructing a full-text retrieval function; performing real-time streaming data analysis and offline analysis on the distributed storage data, associating the data with threat intelligence data, and detecting potential attack events; and detecting the restored network flow data, identifying malicious files and generating alarm information so as to realize intelligent detection, real-time analysis and automatic response of a closed loop of network security threats, thereby improving the accuracy and efficiency of network security protection.

A system for extracting a number of data elements from one or more data sources. The system may separate the text from the tables in a document, such that only the table data may be sent to the large language model (LLM), when the LLM only needs to review the table data. The system may include converting a PDF to text, and separating the tables form the document text using markdown language from converting the PDF. The system may form table chunks and text chunks, index the chunks using a vector embedding and store a chunk identifier, document identifier, and or a page identifier with the chunk to provide result traceability. The system may, in response to a prompt, retrieve and send the targeted table chunks or text chunks to the LLM to extract the data elements. The system populates an ontological data store based on the LLM response.

The embodiment of the invention provides a cross-modal data search method and device based on Surreal DB. The method comprises the steps that multi-modal data input by a user at a data query interface is received, and the multi-modal data comprises text, picture, audio, video, webpage and document data; the method comprises the following steps: converting audio, video, webpage and document data into texts and pictures, performing keyword extraction on the texts to obtain text cues, and performing vectorization processing on the texts and the pictures to obtain query vectors; performing full-text retrieval on the text cue word based on a Surreal DB database to obtain a text search result, and performing vector retrieval on the query vector to obtain a vector search result; reordering the text search result and the vector search result by using a ranking-based search result fusion algorithm, and outputting a preliminary search result; and performing associated query on the preliminary retrieval result to obtain associated data.

The invention relates to the technical field of remote sensing image analysis and cross-modal retrieval. The invention discloses a remote sensing image text retrieval method based on a remote sensing multi-modal basic model, which applies the large-scale pre-training capability of a CLIP large model to semantic alignment of a remote sensing image and a text by finely adjusting the CLIP large model. By introducing the visual saliency calculation module and the visual block fine-grained selection integration module, the problems of multi-scale targets and redundant information in the remote sensing image are effectively solved, fine-grained semantic alignment between the image and the text is realized, and the retrieval accuracy is improved. Particularly, under the condition that the image contains a plurality of salient targets and redundant regions, the cross-modal semantic alignment fine-grained filtering method provided by the invention can accurately identify key information blocks in the image and perform fine matching with text description.

The invention discloses an image-text cross-modal retrieval method, which comprises the following steps of: constructing an image-text data pair set, and respectively carrying out feature extraction on data of an image modal and a text modal; extracting hierarchical features of the image and the text through a multi-level attention distillation network, and performing inter-modal fine-grained alignment by adopting a cross attention mechanism and knowledge distillation to obtain fusion features; a dynamic batch clustering strategy is adopted to optimize feature alignment, a positive intra-class distance and a positive and negative inter-class distance are calculated, a distance correlation coefficient is adopted to calculate feature similarity, a joint loss function is constructed, and the loss function is minimized, intra-class distance correlation of similar samples is minimized, and inter-class distance correlation of heterogeneous samples is maximized; and mapping the fusion features to a public feature space, calculating the similarity between the fusion features and target modal features, and outputting image-text cross-modal and single-modal retrieval results. The technical problem that the retrieval precision and efficiency are limited due to the fact that modal heterogeneity and semantic heterogeneity are difficult to guarantee in an existing method is solved.

The invention provides a cross-modal joint contrast learning method and device and electronic equipment, and the method comprises the steps: constructing a sample data set which covers a plurality of task types, such as a text retrieval image, an image retrieval text, a text retrieval text, an image retrieval image, an image-text joint retrieval image and an image-text joint retrieval text; and generating prompt words for identifying task types for each task sample, splicing the prompt words with sample data to form task input, and determining modal types of a retrieval object and a retrieval target. A retrieval object and a retrieval target are respectively input into encoders of corresponding modes to extract features, the features are mapped to the same semantic space through a unified projection layer to obtain an embedded vector, and the parameters of the encoders and the projection layer are optimized by utilizing a contrast learning loss function based on the similarity of the retrieval object and the retrieval target, so that multi-task unified training is realized. Various cross-modal retrieval tasks can be supported in a unified semantic space at the same time, and the overall retrieval effect is improved on the premise of ensuring multi-task performance balance.

The invention discloses a global knowledge extraction method and system based on a large model and an RAG technology, and belongs to the technical field of large model data processing. According to the global knowledge extraction method and system based on the large model and the RAG technology, local sensitive hashing is adopted for duplicate removal, similar texts are combined through dynamic thresholds, and redundant data interference is reduced; a text segmentation strategy based on separator confidence ensures that segmented text blocks maintain semantic integrity and are adaptive to large model input length limitation, similarity retrieval and full-text retrieval are fused, results are reordered in combination with an RRF algorithm, semantic relevance and accuracy are considered, the multi-scene knowledge construction requirement is met, and the multi-scene knowledge construction efficiency is improved. High-correlation entity pairs are screened through mutual information, context knowledge is retrieved from a vector database in combination with an RAG system, semantic reasoning is conducted through a large language model, hidden logic relations among entities are generated, and jumping from data association to knowledge generation is achieved.