99 results about "Document segmentation" patented technology

The invention provides an enhanced document generation and retrieval method based on a knowledge graph, and relates to the technical field of knowledge graphs, and the method comprises the following steps: receiving heterogeneous document input, and carrying out dynamic segmentation on the heterogeneous document; constructing a Graph-RAG model, wherein the Graph-RAG model comprises a knowledge graph-document index extraction module and a design document generation module; constructing and updating a knowledge graph based on document segmentation through a knowledge graph-document index extraction module; extracting feature vectors of the document segments by using a feature extraction model, establishing a mapping relationship between the document segments and entities in the knowledge graph through a matching strategy, and constructing a retrieval index; and receiving a query sequence of a user, retrieving the related entities and document segments through the retrieval index, and generating a target design document in combination with a preset template. According to the method, accurate retrieval and high-quality document generation based on semantic understanding are realized, and the technical problems of insufficient semantic understanding, low knowledge association degree, poor generation quality and the like in a traditional document processing method can be solved.

The invention belongs to the field of artificial intelligence and financial science and technology, and relates to a document segmentation method, which comprises the following steps: carrying out style feature recognition and title text recognition on a to-be-segmented document to obtain a title style result and a title text result, and then carrying out recognition verification on the title style result and the title text result through a multi-modal title recognition large model to obtain a segmented document. And obtaining a final title recognition result, constructing tree hierarchical representation based on the title recognition result to obtain a document chapter structure, and segmenting the to-be-segmented document according to the document chapter structure. The invention further provides a document segmentation device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-segmented document can be stored in a block chain. According to the method and the device, the document is segmented in a chapter form, the semantic integrity of segmentation is ensured, and key information is effectively extracted and structured.

The invention provides a retrieval method and device based on document segmentation and a document retrieval system. The method comprises the following steps: acquiring a to-be-segmented document; based on an NLP algorithm, calculating the semantic similarity between the partial texts of the to-be-segmented document to obtain a first semantic relevancy; according to the first semantic relevancy of all the partial texts, the document to be segmented is segmented, a plurality of semantic text blocks are obtained, and each semantic text block comprises at least one partial text; under the condition that a query request is received, calculating semantic similarity between a query text corresponding to the query request and each semantic text block based on an NLP algorithm to obtain a plurality of second semantic relevancy, and determining the semantic text block with the highest second semantic relevancy of the query text corresponding to the query request as a target semantic text block, and displaying the target semantic text block in a display interface. According to the scheme, the problem that in the prior art, the accuracy rate is low during text retrieval is solved.

The invention provides a document segmentation method and device and electronic equipment, relates to the technical field of artificial intelligence such as natural language processing, large language models, intelligent search and knowledge maps, and can be applied to scenes such as knowledge base construction, intelligent question and answer systems and document management systems. According to the specific implementation scheme, document analysis is conducted on a to-be-processed document, an analysis result of the to-be-processed document is obtained, and the analysis result comprises layout information and hierarchy information of a title; a title tree is constructed based on the hierarchical information of the titles, and the titles associated with nodes in the title tree are multi-level composite titles; and based on the layout information and the title tree, splitting and recombining the document region of the to-be-processed document according to a given maximum slice length to obtain document slices of the to-be-processed document. According to the method and the device, the document slicing precision and flexibility can be remarkably improved, and the document readability and the user experience are enhanced.

The invention provides a document segmentation method based on a multi-modal large model, and the method comprises the following steps: S1, carrying out the preprocessing of a document, and extracting the original features of each modal in a to-be-segmented document; s2, multi-modal feature encoding: encoding the original features of each modal by using an encoder to generate feature representations which can be identified and processed by the model; s3, modal fusion: fusing the multi-modal features according to the weight of each modal to obtain a fused document feature representation; s4, document segmentation: segmenting the document feature representation by using a segmentation model, and outputting a segmentation boundary and a category label of the document; and S5, performing post-processing and optimization, evaluating the accuracy of a segmentation result, and adjusting the segmentation result and model parameters according to an evaluation result. According to the method, multi-modal features such as texts, images, tables and formats are fused, the weight of each modal is dynamically distributed in combination with an adaptive weighted fusion mechanism, relevance and importance differences among different modals are effectively captured, and the segmentation accuracy of complex documents is improved.

The embodiment of the invention provides a document-based intelligent question and answer method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: performing document analysis on a target electronic document to obtain document attributes and document contents; based on the document attribute and the document content, document segmentation is conducted on the target electronic document, document segmentation units and unit position information of the document segmentation units are obtained, and the document segmentation units comprise segmentation unit identifiers; based on the document attribute, performing network construction on the target electronic document to obtain a document structure tree network; on the basis of the unit position information, the document segmentation units are arranged in sequence, and a document linear list is obtained; performing document query on the document structure tree network or the document linear list based on the document problem information and the segmentation unit identifier to obtain document query information; and based on the document query information, performing question reply on the document question information. According to the embodiment of the invention, the intelligent question and answer accuracy based on the document can be improved.

The invention discloses an AI (artificial intelligence) agent construction system based on hybrid retrieval and father-child segmentation, which comprises the following steps of: dividing a subclass knowledge base according to domain knowledge, performing father-child segmentation processing, and constructing a hierarchical semantic network; vectorization embedding and deep semantic reconstruction are carried out on the user question text; retrieving the reconstructed problem by adopting a mixed retrieval algorithm combining sparse retrieval and dense retrieval, and forming a high-score sub-segment set according to a comprehensive score obtained by dynamic weight distribution; mapping the sub-segments to the parent segment through a hierarchical backtracking algorithm, aggregating brother nodes to form an extended candidate set, and generating an associated sub-segment set after duplicate removal and re-retrieval; and finally inputting a large language model to generate a complete answer. According to the method, the problems of context segmentation, low retrieval accuracy and complicated knowledge base maintenance of traditional document segments are solved, the answer coverage and accuracy of an intelligent question-answering system are remarkably improved, and the method is suitable for knowledge question-answering scenes in the complicated technical fields such as intelligent network connection automobiles and the like.

The invention relates to the field of text processing, in particular to a bidding document information extraction method. Comprising the following steps: segmenting a bidding and tendering file into pages, and identifying the pages to obtain corresponding texts; generating complementary text description for images and tables in the page and adding the complementary text description to the tail of a text corresponding to the page to form an enhanced text block sequence; matching a label from the text block sequence according to a pre-constructed hierarchical label system, and generating a corresponding cue word template according to the label and a pre-constructed cue word template library; inputting the cue word template, the enhanced text block sequence and the context text abstract as a combination into a large language model to obtain a structured extraction result with a hierarchical relationship; and matching the extracted entity content with a local dictionary, carrying out aggregation arrangement on a result after the matching is passed, and outputting a structured data file. On the premise that the model does not need to be retrained, the illusion risk of the generated content is reduced.

The invention discloses a document segmentation method for large model retrieval enhancement generation, and relates to the technical field of document segmentation, the method comprises the following steps: taking any secondary title and text content under the secondary title as a to-be-analyzed object, and counting the number of characters in each to-be-analyzed object, when the number of the characters exceeds the preset number of the characters, paragraph splitting is carried out on the text content; and carrying out segmentation confidence calculation on each split part, and carrying out merging or segmentation processing on the adjacent parts based on the segmentation confidence corresponding to the adjacent parts. According to the method, the segmentation confidence coefficient of the adjacent parts is calculated, the document can be accurately and partially segmented, the segmentation effect is effectively guaranteed, and in addition, based on the segmentation confidence coefficient, the semantic integrity after segmentation and before and after segmentation can be effectively guaranteed.

The invention discloses a retrieval enhancement generation method based on a medical map, and belongs to the field of artificial intelligence and medical information processing. The method comprises the following three main steps: (1) constructing a medical map: constructing a multi-level medical knowledge map through semantic document segmentation, element extraction, hierarchical linking, relation linking, label generation and map combination; (2) map retrieval: retrieving related information in the constructed map according to user query by using a U-retrieve method; and (3) text generation: accurate and comprehensive answers are generated based on retrieval results. According to the method, by integrating large-scale medical literatures, textbooks and clinical data, the answer accuracy and interpretability of the large language model in the medical field are remarkably improved. Detailed answers containing evidence sources can be generated, and the illusion problem is effectively avoided. A powerful tool is provided for the fields of medical decision support, medical education, patient consultation and the like, and wide application of artificial intelligence in the field of medical health is expected to be promoted.

The invention provides a document question-answering system and method based on a large model, and relates to the technical field of artificial intelligence, and the system comprises a document segmentation module which is used for segmenting a document uploaded by a user according to a paragraph structure to obtain a plurality of paragraph nodes; and the answer generation module is connected with the document segmentation module and is used for performing similarity matching on a query question proposed by a user and each paragraph node and sending the paragraph node with the highest similarity into a local large language model for processing so as to display answer information associated with the query question to the user. The method has the beneficial effects that the document is segmented according to the paragraph structure, so that the related content is effectively prevented from being split into different paragraph nodes, the context continuity is ensured, and the accuracy of a search result is improved; and the answer information is generated by adopting a local large language model, so that data leakage and privacy risks are avoided, and the data security is enhanced while the user privacy is effectively protected.

The invention discloses a document review method based on multi-agent collaboration and retrieval enhancement generation. The method comprises the following steps: firstly, analyzing a document review rule input by a user into a rule semantic intermediate representation through a natural language processing technology, and calling a retrieval enhancement generation (RAG) module to expand related knowledge to form a structured rule library; secondly, multi-modal analysis and chapter segmentation are carried out on a to-be-examined document, and elements such as texts, tables, pictures and formulas are expressed in a unified mode; tasks such as rule analysis, document segmentation, knowledge retrieval, matching comparison and report generation are completed through a multi-agent cooperation mechanism; finally, when ambiguity exists in rule and document matching, an RAG module is introduced to retrieve supplementary evidences from an external knowledge base, explanatory comparison is conducted in combination with the generative model, and the accuracy and authority of judgment are improved.

The invention relates to the technical field of intelligent questioning and answering, in particular to an intelligent questioning and answering system implementation method and system.The intelligent questioning and answering system implementation method comprises the following steps of document collection and preprocessing, document dicing, document layout analysis and table layout analysis; the method has the beneficial effects that a semantic association network and a multi-modal index system are formed through offline document collection, preprocessing, slicing, layout analysis, data extraction and knowledge graph construction; in the online part, the capabilities of Embedding vectorization, multi-index joint retrieval, tensor reordering, AI database integration and large language model generation are combined, accurate semantic understanding and rapid knowledge matching of user questions are realized, high-quality answers are generated, and interactive feedback optimization is supported.

The invention provides a document processing method and system based on a multi-modal large model. According to the method, firstly, a target type document is converted into a target picture based on a multi-modal large model according to a preset document segmentation rule, then structured text information is extracted from the target picture based on a preset cue word and by utilizing an advanced optical character recognition technology PaddleOCR technology, the multi-modal large model is guided to generate a text abstract for each target picture, and the text abstract is extracted from the target picture. And the integrated graphic and text information and the condensed picture content are converted into a Markdown document comprising a picture link of the target picture and text summary information. Through the steps of format recognition, image-text separation, content condensation and the like, a target type document is converted into a format which is easy to manage and retrieve, rapid retrieval of information is achieved by means of RAG retrieval enhancement, and documents and information related to user query can be rapidly found to serve as candidate answers. Compared with a traditional OCR technology, the method has the advantages that the missing rate of key information is remarkably reduced, and the accuracy of document processing is improved.

The invention discloses an optimized RAG-based urban rail transit emergency field question and answer method and system, and relates to the technical field of artificial intelligence, and the method comprises the steps: firstly rewriting a user question into a new question conforming to an urban rail transit style, and then analyzing different types of documents in a knowledge base; then, the document is segmented, vector embedding is carried out, and meanwhile, a new problem is converted into a vector; then, by calculating the similarity between a problem vector and a document block vector, the most relevant document block is selected to form a recall set; then, a cross encoder is used for reordering the recalled documents, and the most relevant documents are selected to form a rearranged document set; and finally, inputting the new question and the rearranged document set into the fine-tuned large language model to generate a final answer. According to the method, through a large language model of question rewriting, accurate document segmentation and vectorization, multi-level retrieval and rearrangement and field fine adjustment, the accuracy and correlation of question answering in the urban rail transit emergency field are remarkably improved, and more professional and reliable intelligent question answering service is achieved.

The invention discloses an automatic compliance examination method and system based on graph retrieval enhancement. The method comprises the following steps of: obtaining a billing specification and a historical inquiry report as input documents; the input document is subjected to dual-mode document segmentation based on a large language model, logic blocks and physical blocks are generated, and each logic block is a continuous page unit and is attached with a content abstract generated by the model; constructing a precedent graph, extracting key legal elements in the historical inquiry report through multi-stage recursion, generating semantic network nodes carrying traceability identifiers, and establishing cross-document semantic association; and constructing a state graph, automatically identifying chapters and logic and semantic relationships among the chapters based on a document hierarchical structure, and generating a machine-readable structured index and the like. According to the method, the long text semantic understanding depth, the cross-section consistency and the legal reasoning accuracy are remarkably improved, the manual review cost is reduced, and the method is suitable for a listing compliance review scene under a registration system.

The invention relates to the technical field of natural language processing (NLP), in particular to an intelligent self-adaptive document segmentation method oriented to an RAG system. The method comprises the steps that S1, a deep learning model is adopted to dynamically adjust the size and the step length of a window according to document content density, structure information and context semantics; s2, in combination with a window representation result, calculating the context semantic similarity of the segmentation blocks by adopting a language model, and automatically adjusting the size of an overlapping region based on the context semantic similarity; and S3, based on a context segmentation block representation result, associating the segmentation block with the context by introducing a BERT model, automatically adjusting an overlapping part and a window size, and optimizing a segmentation effect. The invention aims to provide the intelligent self-adaptive document segmentation method oriented to the RAG system so as to improve the document segmentation efficiency and quality and optimize the recall rate and the generation quality of the RAG system.

The invention discloses a multi-level document segmentation method adopting an adaptive dynamic partitioning algorithm, and relates to the technical field of document segmentation, the method comprises the following steps: determining a target document type and target chapter information of a to-be-segmented document; when the target document type is a standard document, calculating a target information density corresponding to the to-be-segmented document based on the target chapter information; and based on the target information density, determining a target chapter overlapping degree corresponding to the to-be-segmented document, and segmenting each target chapter according to the target chapter overlapping degree to complete segmentation processing of the to-be-segmented document. According to the method and device, it is ensured that the chapter content of the segmented document is complete and logically coherent, the problems of information loss or chapter content segmentation and the like caused by blind segmentation are solved, the document segmentation quality and efficiency are effectively improved, and the segmented document better meets the actual use requirement.

The invention provides a GLM-4-based unmarked course knowledge point graph construction method and system. The method comprises the steps of generating a first hierarchical structure of a knowledge point graph; according to the first hierarchical structure, performing document segmentation operation on the chapters and all levels of subordinate titles to obtain chapter independent units and hierarchical title independent units; performing data cleaning operation to obtain a preprocessed chapter independent unit and a preprocessed title independent unit; performing knowledge point acquisition operation, calling an application programming interface (API) of a GLM-4 model, giving a prompt instruction, and extracting knowledge points conforming to the outline from the preprocessing chapter independent unit and the preprocessing title independent unit by utilizing the GLM-4 model according to the prompt instruction; using a GLM-4 model to carry out noise removal operation to obtain de-noised data; comparing through a cosine similarity algorithm; and constructing a knowledge graph. The technical problem that knowledge is difficult to understand due to the fact that the knowledge point graph is too refined in granularity is solved.

The invention provides a document segmentation method and device based on a large language model, equipment and a storage medium, and relates to the technical field of text processing. The method comprises the steps of inputting a to-be-segmented target document into a pre-trained large language model, and executing the following operations through the large language model: performing text layout analysis on the target document, and identifying titles and all paragraphs of each level in the target document; for each paragraph, inserting an associated title related to the paragraph in all titles into an initial position of the paragraph to obtain a corresponding target paragraph; and sorting all the target paragraphs based on the semantic similarity among all the target paragraphs, and determining a segmentation result of the target document based on all the sorted target paragraphs. By the adoption of the technical scheme, when document segmentation is carried out, semantic loss in the document segmentation process can be effectively reduced, and therefore the document segmentation effect is improved.

The invention provides an emergency plan auditing method and system, electronic equipment and a storage medium. An emergency plan is analyzed to obtain plan content; performing text cleaning and formatting on the chapter content of each chapter, the paragraph content of each paragraph, the entry content of each entry and a plurality of titles which are obtained by performing document segmentation on the emergency plan according to the plan content and dividing according to the logic hierarchy of the emergency plan; obtaining a target chapter content, a target paragraph content, a target entry content and a target title; extracting a plurality of keywords related to the emergency field from each target chapter content, each target paragraph content and each target entry content, and endowing each chapter, each paragraph and each entry with a corresponding semantic category; and adding field information for each chapter, each paragraph and each entry, and constructing a nested structure for each chapter, each paragraph and each entry added with the field information according to the logic level of the emergency plan to obtain a corresponding tree data model.

The invention discloses a document segmentation method for self-adaptive slices generated by large model retrieval enhancement, and relates to the technical field of large model retrieval enhancement generation, the method comprises the following steps: obtaining a to-be-segmented document, and segmenting the to-be-segmented document according to a title type to obtain at least one group of original blocks; according to the information density and the theme change degree corresponding to any original block, the optimal segmentation number of the original block is calculated; and performing segmentation processing on the original blocks according to the optimal segmentation number. According to the method, firstly, the document is segmented according to the hierarchical titles, then the information density and the theme change degree under the hierarchical titles are calculated, and the optimal segmentation size under the hierarchical titles is automatically calculated by taking the hierarchical titles as units, so that self-adaptive segmentation of the document is guided, and the effects of subsequent retrieval and task generation are improved.

The invention discloses a paragraph segmentation method and device based on content awareness, and relates to the field of document processing. In the method, an original file in an original format is converted into a target file in a target format, and a text and a table are identified and separated from the target file; traversing the text to identify paragraphs and outline levels corresponding to the paragraphs, and splicing paragraph contents according to the outline levels to form structured text data; traversing the table, extracting header data and data of each cell, and constructing key value pairs for the header data and the data of each cell to form table data; identifying a plain text from the text, segmenting the plain text into lexical units, and processing the lexical units by using a preset rule to form array data; and combining and outputting the text data, the table data and the array data. By implementing the technical scheme provided by the invention, the bid inviting word file is automatically and accurately segmented into logically independent paragraphs.

The invention belongs to the technical field of large model training, and particularly relates to a power technology standard knowledge question and answer large model training data construction method and device. The method comprises the following steps: acquiring a power technology standard document; preprocessing the power technology standard document to obtain a preprocessed power technology standard document; inputting the preprocessed power technology standard document into a GNN network for feature extraction to obtain structured information; dynamically optimized document segmentation is carried out on the power technology standard document based on the structured information, and segmented document fragments are obtained; and performing multi-task and multi-mode combined processing on the segmented document segments to obtain training data of the power technology standard knowledge question-answer large model. According to the method, automatic splitting of the electric power technology standard document can be efficiently and accurately carried out, the performance of an electric power technology standard retrieval question-answering system is greatly improved, the labor cost is reduced, the information acquisition speed is increased, and remarkable social and economic benefits are achieved.

The invention is suitable for the technical field of literature data processing, and provides a medical literature classification system based on intelligent annotation, which comprises a database and a classification module in communication connection with the database, the classification module comprises a retrieval unit, an extraction unit, a training unit, an integration unit, a labeling unit, a management unit and a classification unit which are in communication connection in sequence; the training unit is used for training the classification model by taking the annotation data as training data to obtain a medical literature classification model; according to the system, data extraction is carried out on retrieved literatures, data mining analysis is carried out on obtained key information, and training based on a professional classification system and labeling rules is carried out on the literatures, so that the problem that classification of existing medical literatures is not refined and professional enough can be solved, regular word screening is carried out on literature word segmentation, and the classification efficiency of the medical literatures is improved. And the problem of relatively large classification data in the prior art can be solved, and an efficient classification effect is obtained.

The invention provides an image and data analysis processing method for a super-large file and electronic equipment and a storage medium thereof, and the method comprises the steps: segmenting the super-large file into a plurality of sub-files through an intelligent segmentation algorithm, building an index, and constructing a distributed storage network based on edge computing and cloud computing; according to the method, distributed storage and parallel processing of sub-files are achieved, GPU accelerated image processing and data analysis algorithms are utilized, the deep learning technology is combined, efficient processing and intelligent analysis are conducted on the sub-files, finally, processing results of storage nodes are summarized through the data fusion technology, the processing flow is optimized, and the overall efficiency and quality are improved. According to the technical scheme, the problems of resource bottleneck, low processing speed and low analysis precision in ultra-large file processing are effectively solved, large-scale data processing requirements are met, and low delay, high availability and safety of data processing are ensured.

The present invention proposes an enhanced document generation and retrieval method based on a knowledge graph, which relates to the field of knowledge graph technology and includes: receiving heterogeneous document input and dynamically segmenting the heterogeneous documents; constructing a Graph-RAG model, including a knowledge graph-document index extraction module and a design document generation module; constructing and updating a knowledge graph based on document segmentation via the knowledge graph-document index extraction module; extracting feature vectors of document segments using a feature extraction model, establishing a mapping relationship between document segments and entities in the knowledge graph through a matching strategy, and constructing a retrieval index; receiving a user's query sequence, retrieving relevant entities and document segments through the retrieval index, and generating a target design document based on a preset template. The present invention achieves precise retrieval and high-quality document generation based on semantic understanding, and can address technical issues such as insufficient semantic understanding, low knowledge relevance, and poor generation quality that exist in traditional document processing methods.

The invention relates to the technical field of natural language processing, and provides a document segmentation method based on context marking and model cascading, which comprises the following steps: in response to a document segmentation request, loading a to-be-processed document and initializing segmentation parameters; calling a large language model to analyze a current to-be-processed text segment, identifying a logic demarcation point, and generating wedge information containing a segmentation point mark and a context; positioning absolute positions of segmentation points in the text segment according to the wedge information, and segmenting the text segment into a plurality of text sub-segments; repeating the execution until a preset recursion termination condition is reached; after recursion is completed, segmenting results of all layers are aggregated, a hierarchical document structure is constructed, and the segmenting results are output. Through a wedge mechanism, only tiny positioning marks are output, the token cost is reduced, and the overall cost is optimized in combination with a model cascading strategy. And the generated text block is highly aligned with the semantic boundary of the document, so that the context fragmentation problem is effectively solved, the context relevance is enhanced, and the model illusion is inhibited.

The invention discloses a question and answer method and device, electronic equipment and a computer storage medium, and the method comprises the steps: carrying out the intention recognition of collected original question information of a user, and obtaining an intention recognition result; and performing vector matching in a preset knowledge base according to the intention recognition result, and searching in the preset knowledge base to obtain knowledge points corresponding to the original question information of the user and pictures matched with the knowledge points. And outputting knowledge points corresponding to the original question information of the user and pictures matched with the knowledge points. Through the multi-modal document segmentation method, the picture can be bound with the previous paragraph or the following paragraph after being recognized in the segmentation process, and the picture can be taken out together as a reference picture when the output text relates to the previous or following content, so that knowledge loss is avoided.

Disclosed are a method, a device, and / or a system of audio enhancement of video through video file segmentation, event extraction, and contextual data structuring for efficient matching, generation, and / or alignment of audio to a depicted event. In one embodiment, a system includes a memory storing computer readable instructions that when executed initiate a video object in a database representing a video file and store a video segmentation reference drawn from the video object to a segmentation object, which may represent a shot or scene in the video. The system may parse the video file to extract an event including an event range, an event description, and an event ontology, and may generate encoding vector(s) therefrom. The system may initiate an event object, then link the event object to the video object through the segmentation object, to enable efficient import of context for audio matching and / or audio generation for the event.