75 results about "Document segmentation" patented technology

The invention provides a retrieval method and device based on document segmentation and a document retrieval system. The method comprises the following steps: acquiring a to-be-segmented document; based on an NLP algorithm, calculating the semantic similarity between the partial texts of the to-be-segmented document to obtain a first semantic relevancy; according to the first semantic relevancy of all the partial texts, the document to be segmented is segmented, a plurality of semantic text blocks are obtained, and each semantic text block comprises at least one partial text; under the condition that a query request is received, calculating semantic similarity between a query text corresponding to the query request and each semantic text block based on an NLP algorithm to obtain a plurality of second semantic relevancy, and determining the semantic text block with the highest second semantic relevancy of the query text corresponding to the query request as a target semantic text block, and displaying the target semantic text block in a display interface. According to the scheme, the problem that in the prior art, the accuracy rate is low during text retrieval is solved.

The invention provides a document segmentation method based on a multi-modal large model, and the method comprises the following steps: S1, carrying out the preprocessing of a document, and extracting the original features of each modal in a to-be-segmented document; s2, multi-modal feature encoding: encoding the original features of each modal by using an encoder to generate feature representations which can be identified and processed by the model; s3, modal fusion: fusing the multi-modal features according to the weight of each modal to obtain a fused document feature representation; s4, document segmentation: segmenting the document feature representation by using a segmentation model, and outputting a segmentation boundary and a category label of the document; and S5, performing post-processing and optimization, evaluating the accuracy of a segmentation result, and adjusting the segmentation result and model parameters according to an evaluation result. According to the method, multi-modal features such as texts, images, tables and formats are fused, the weight of each modal is dynamically distributed in combination with an adaptive weighted fusion mechanism, relevance and importance differences among different modals are effectively captured, and the segmentation accuracy of complex documents is improved.

The invention discloses an AI (artificial intelligence) agent construction system based on hybrid retrieval and father-child segmentation, which comprises the following steps of: dividing a subclass knowledge base according to domain knowledge, performing father-child segmentation processing, and constructing a hierarchical semantic network; vectorization embedding and deep semantic reconstruction are carried out on the user question text; retrieving the reconstructed problem by adopting a mixed retrieval algorithm combining sparse retrieval and dense retrieval, and forming a high-score sub-segment set according to a comprehensive score obtained by dynamic weight distribution; mapping the sub-segments to the parent segment through a hierarchical backtracking algorithm, aggregating brother nodes to form an extended candidate set, and generating an associated sub-segment set after duplicate removal and re-retrieval; and finally inputting a large language model to generate a complete answer. According to the method, the problems of context segmentation, low retrieval accuracy and complicated knowledge base maintenance of traditional document segments are solved, the answer coverage and accuracy of an intelligent question-answering system are remarkably improved, and the method is suitable for knowledge question-answering scenes in the complicated technical fields such as intelligent network connection automobiles and the like.

The invention relates to the field of text processing, in particular to a bidding document information extraction method. Comprising the following steps: segmenting a bidding and tendering file into pages, and identifying the pages to obtain corresponding texts; generating complementary text description for images and tables in the page and adding the complementary text description to the tail of a text corresponding to the page to form an enhanced text block sequence; matching a label from the text block sequence according to a pre-constructed hierarchical label system, and generating a corresponding cue word template according to the label and a pre-constructed cue word template library; inputting the cue word template, the enhanced text block sequence and the context text abstract as a combination into a large language model to obtain a structured extraction result with a hierarchical relationship; and matching the extracted entity content with a local dictionary, carrying out aggregation arrangement on a result after the matching is passed, and outputting a structured data file. On the premise that the model does not need to be retrained, the illusion risk of the generated content is reduced.

The invention discloses a document review method based on multi-agent collaboration and retrieval enhancement generation. The method comprises the following steps: firstly, analyzing a document review rule input by a user into a rule semantic intermediate representation through a natural language processing technology, and calling a retrieval enhancement generation (RAG) module to expand related knowledge to form a structured rule library; secondly, multi-modal analysis and chapter segmentation are carried out on a to-be-examined document, and elements such as texts, tables, pictures and formulas are expressed in a unified mode; tasks such as rule analysis, document segmentation, knowledge retrieval, matching comparison and report generation are completed through a multi-agent cooperation mechanism; finally, when ambiguity exists in rule and document matching, an RAG module is introduced to retrieve supplementary evidences from an external knowledge base, explanatory comparison is conducted in combination with the generative model, and the accuracy and authority of judgment are improved.

The invention relates to the technical field of intelligent questioning and answering, in particular to an intelligent questioning and answering system implementation method and system.The intelligent questioning and answering system implementation method comprises the following steps of document collection and preprocessing, document dicing, document layout analysis and table layout analysis; the method has the beneficial effects that a semantic association network and a multi-modal index system are formed through offline document collection, preprocessing, slicing, layout analysis, data extraction and knowledge graph construction; in the online part, the capabilities of Embedding vectorization, multi-index joint retrieval, tensor reordering, AI database integration and large language model generation are combined, accurate semantic understanding and rapid knowledge matching of user questions are realized, high-quality answers are generated, and interactive feedback optimization is supported.

The invention provides a document processing method and system based on a multi-modal large model. According to the method, firstly, a target type document is converted into a target picture based on a multi-modal large model according to a preset document segmentation rule, then structured text information is extracted from the target picture based on a preset cue word and by utilizing an advanced optical character recognition technology PaddleOCR technology, the multi-modal large model is guided to generate a text abstract for each target picture, and the text abstract is extracted from the target picture. And the integrated graphic and text information and the condensed picture content are converted into a Markdown document comprising a picture link of the target picture and text summary information. Through the steps of format recognition, image-text separation, content condensation and the like, a target type document is converted into a format which is easy to manage and retrieve, rapid retrieval of information is achieved by means of RAG retrieval enhancement, and documents and information related to user query can be rapidly found to serve as candidate answers. Compared with a traditional OCR technology, the method has the advantages that the missing rate of key information is remarkably reduced, and the accuracy of document processing is improved.

The invention discloses an automatic compliance examination method and system based on graph retrieval enhancement. The method comprises the following steps of: obtaining a billing specification and a historical inquiry report as input documents; the input document is subjected to dual-mode document segmentation based on a large language model, logic blocks and physical blocks are generated, and each logic block is a continuous page unit and is attached with a content abstract generated by the model; constructing a precedent graph, extracting key legal elements in the historical inquiry report through multi-stage recursion, generating semantic network nodes carrying traceability identifiers, and establishing cross-document semantic association; and constructing a state graph, automatically identifying chapters and logic and semantic relationships among the chapters based on a document hierarchical structure, and generating a machine-readable structured index and the like. According to the method, the long text semantic understanding depth, the cross-section consistency and the legal reasoning accuracy are remarkably improved, the manual review cost is reduced, and the method is suitable for a listing compliance review scene under a registration system.

The invention relates to the technical field of natural language processing (NLP), in particular to an intelligent self-adaptive document segmentation method oriented to an RAG system. The method comprises the steps that S1, a deep learning model is adopted to dynamically adjust the size and the step length of a window according to document content density, structure information and context semantics; s2, in combination with a window representation result, calculating the context semantic similarity of the segmentation blocks by adopting a language model, and automatically adjusting the size of an overlapping region based on the context semantic similarity; and S3, based on a context segmentation block representation result, associating the segmentation block with the context by introducing a BERT model, automatically adjusting an overlapping part and a window size, and optimizing a segmentation effect. The invention aims to provide the intelligent self-adaptive document segmentation method oriented to the RAG system so as to improve the document segmentation efficiency and quality and optimize the recall rate and the generation quality of the RAG system.

The invention discloses a multi-level document segmentation method adopting an adaptive dynamic partitioning algorithm, and relates to the technical field of document segmentation, the method comprises the following steps: determining a target document type and target chapter information of a to-be-segmented document; when the target document type is a standard document, calculating a target information density corresponding to the to-be-segmented document based on the target chapter information; and based on the target information density, determining a target chapter overlapping degree corresponding to the to-be-segmented document, and segmenting each target chapter according to the target chapter overlapping degree to complete segmentation processing of the to-be-segmented document. According to the method and device, it is ensured that the chapter content of the segmented document is complete and logically coherent, the problems of information loss or chapter content segmentation and the like caused by blind segmentation are solved, the document segmentation quality and efficiency are effectively improved, and the segmented document better meets the actual use requirement.

The invention provides a document segmentation method and device based on a large language model, equipment and a storage medium, and relates to the technical field of text processing. The method comprises the steps of inputting a to-be-segmented target document into a pre-trained large language model, and executing the following operations through the large language model: performing text layout analysis on the target document, and identifying titles and all paragraphs of each level in the target document; for each paragraph, inserting an associated title related to the paragraph in all titles into an initial position of the paragraph to obtain a corresponding target paragraph; and sorting all the target paragraphs based on the semantic similarity among all the target paragraphs, and determining a segmentation result of the target document based on all the sorted target paragraphs. By the adoption of the technical scheme, when document segmentation is carried out, semantic loss in the document segmentation process can be effectively reduced, and therefore the document segmentation effect is improved.

The invention belongs to the technical field of large model training, and particularly relates to a power technology standard knowledge question and answer large model training data construction method and device. The method comprises the following steps: acquiring a power technology standard document; preprocessing the power technology standard document to obtain a preprocessed power technology standard document; inputting the preprocessed power technology standard document into a GNN network for feature extraction to obtain structured information; dynamically optimized document segmentation is carried out on the power technology standard document based on the structured information, and segmented document fragments are obtained; and performing multi-task and multi-mode combined processing on the segmented document segments to obtain training data of the power technology standard knowledge question-answer large model. According to the method, automatic splitting of the electric power technology standard document can be efficiently and accurately carried out, the performance of an electric power technology standard retrieval question-answering system is greatly improved, the labor cost is reduced, the information acquisition speed is increased, and remarkable social and economic benefits are achieved.

The present invention proposes an enhanced document generation and retrieval method based on a knowledge graph, which relates to the field of knowledge graph technology and includes: receiving heterogeneous document input and dynamically segmenting the heterogeneous documents; constructing a Graph-RAG model, including a knowledge graph-document index extraction module and a design document generation module; constructing and updating a knowledge graph based on document segmentation via the knowledge graph-document index extraction module; extracting feature vectors of document segments using a feature extraction model, establishing a mapping relationship between document segments and entities in the knowledge graph through a matching strategy, and constructing a retrieval index; receiving a user's query sequence, retrieving relevant entities and document segments through the retrieval index, and generating a target design document based on a preset template. The present invention achieves precise retrieval and high-quality document generation based on semantic understanding, and can address technical issues such as insufficient semantic understanding, low knowledge relevance, and poor generation quality that exist in traditional document processing methods.

The invention relates to the technical field of natural language processing, and provides a document segmentation method based on context marking and model cascading, which comprises the following steps: in response to a document segmentation request, loading a to-be-processed document and initializing segmentation parameters; calling a large language model to analyze a current to-be-processed text segment, identifying a logic demarcation point, and generating wedge information containing a segmentation point mark and a context; positioning absolute positions of segmentation points in the text segment according to the wedge information, and segmenting the text segment into a plurality of text sub-segments; repeating the execution until a preset recursion termination condition is reached; after recursion is completed, segmenting results of all layers are aggregated, a hierarchical document structure is constructed, and the segmenting results are output. Through a wedge mechanism, only tiny positioning marks are output, the token cost is reduced, and the overall cost is optimized in combination with a model cascading strategy. And the generated text block is highly aligned with the semantic boundary of the document, so that the context fragmentation problem is effectively solved, the context relevance is enhanced, and the model illusion is inhibited.

The invention discloses a question and answer method and device, electronic equipment and a computer storage medium, and the method comprises the steps: carrying out the intention recognition of collected original question information of a user, and obtaining an intention recognition result; and performing vector matching in a preset knowledge base according to the intention recognition result, and searching in the preset knowledge base to obtain knowledge points corresponding to the original question information of the user and pictures matched with the knowledge points. And outputting knowledge points corresponding to the original question information of the user and pictures matched with the knowledge points. Through the multi-modal document segmentation method, the picture can be bound with the previous paragraph or the following paragraph after being recognized in the segmentation process, and the picture can be taken out together as a reference picture when the output text relates to the previous or following content, so that knowledge loss is avoided.

Disclosed are a method, a device, and / or a system of audio enhancement of video through video file segmentation, event extraction, and contextual data structuring for efficient matching, generation, and / or alignment of audio to a depicted event. In one embodiment, a system includes a memory storing computer readable instructions that when executed initiate a video object in a database representing a video file and store a video segmentation reference drawn from the video object to a segmentation object, which may represent a shot or scene in the video. The system may parse the video file to extract an event including an event range, an event description, and an event ontology, and may generate encoding vector(s) therefrom. The system may initiate an event object, then link the event object to the video object through the segmentation object, to enable efficient import of context for audio matching and / or audio generation for the event.

The invention provides a method for extracting data in literatures, which combines a literature segmentation tool with a large language model for use, establishes an end-to-end material literature data extraction workflow capable of being quickly operated and modified, solves the problems of confusion and mismatching when different parameters are associated to the same material, and improves the data extraction efficiency. Character extraction and chart extraction are independently carried out step by step and then are matched and merged, so that accuracy reduction and video memory requirements generated when a large language model processes a long sentence sequence are effectively reduced, time for model modification and data pre-processing annotation by a user is greatly shortened when various literatures are extracted, and user experience is improved. And the document extraction and analysis efficiency is improved.

The invention discloses a retrieval enhancement generation system and method based on network public opinion analysis opinions in the technical field of artificial intelligence. The system comprises a data module used for obtaining public opinion event information; the analysis model module comprises a document segmentation unit, a retrieval unit, a generation module, a memory enhancement unit and a multi-hop query unit; the generation module comprises a first generation unit and a second generation unit, and the first generation unit is used for generating an analysis answer to the event based on a progressive confidence compensation generation method; the memory enhancement unit is used for effectively distinguishing different events and realizing integration and optimization of multiple rounds of queries; a multi-hop query unit is used to enhance the context through hierarchical queries and stepwise focusing. According to the invention, through a memory enhancement mechanism and a progressive confidence compensation optimization strategy, and by introducing a multi-hop query strategy step-by-step focusing problem, information redundancy is significantly reduced and information tracking capability is enhanced, so that context consistency is maintained and improved in an information processing process.

The invention belongs to the field of information retrieval, particularly relates to a full-text retrieval method and system, computer equipment and a computer readable storage medium, and aims to solve the problem of improving the full-text retrieval accuracy. The method comprises the following steps: segmenting a document entering a corpus; calculating paragraph weights of words contained in each segmented document in the segmented document; calculating the document weight of the word in the document according to the paragraph weight of the word; calculating the query weight of the query word, wherein the calculation method of the query weight is the same as the calculation method of the paragraph weight; determining a corresponding target word in a corpus according to the query word; respectively calculating one or more query relevancy according to the query weight and the document weight of the target word; and taking the document corresponding to the document weight corresponding to the maximum n query relevancy as a query result. Semantic features are introduced into weight calculation, and document segmentation processing is combined, so that the full-text retrieval accuracy is effectively improved.

This invention provides a rapid document element extraction system based on a pre-trained large model, belonging to the field of computer software application technology. The system includes: a parameter domain adaptation module for textualizing documents and constructing an industry-standard corpus based on the textualization results, then adjusting a pre-defined language model using the industry-standard corpus; a dynamic document segmentation module for semantically segmenting industry-standard documents to obtain several text blocks; an entity alignment module for extracting entities and relations from the text blocks and performing entity alignment using uniform manifold approximation and projection methods; and a relational reasoning and knowledge graph completion module for completing a preliminary knowledge graph and storing the completion results. This invention eliminates the need for pre-defined rule templates or data annotation, directly improving element extraction efficiency.

The invention relates to the technical field of artificial intelligence recognition, and particularly provides an iterative multi-granularity document segmentation method. The method comprises the following steps: constructing a training corpus, performing segmentation of segments, words and sentences with different granularities on the training corpus, and forming the training corpus by an unsegmented document and a segmented document; training a deep learning model of a GPT structure through the training corpus to obtain a trained segmentation model; the input document is segmented according to the trained segmentation model, and the segmentation result is output, the problem that multi-granularity segmentation cannot be unified is solved, and the segmentation semantics and the segmentation result of the whole document are improved.

Utilizing a secure collaborative document segmentation, allocation, and editing system and method, users can securely edit individual sections of documents where different sections are allocated to different users or groups of multiple users. The electronic document is segregated into one or more sections. The one or more sections are allocated to one or more users. Each of the one or more segregated sections is shared with the one or more users by providing an access link to each of the one or more users to access their allocated one or more segregated section. Each edited section is received from the corresponding one or more users and compiled to automatically generate a final electronic document.

A computer-implemented method for automatically splitting and classifying an input document into one or more sub-documents using a machine learning system is described. The machine learning system includes a visual segmentation neural network, an optical character recognition subsystem, a title classifier, a document classifier, and a grouper subsystem. The method includes receiving visual input representing a plurality of pages of the input document, using the visual segmentation neural network to classify each page of the input document into each of a plurality of templates, determining, for each page of the input document, the final document type to which the page belongs, and using the grouper subsystem to group the plurality of pages of the input document into one or more sub-documents based on (i) each template of each page and (ii) each final document type to which each page belongs.

This application discloses a method, apparatus, and device for constructing a hierarchical domain ontology and knowledge graph. The method includes: acquiring multi-source input data and performing document segmentation processing on the multi-source input data to obtain parsable document fragments. The multi-source input data includes at least one of device service relationships, operating procedures, and system topology. Based on AI, metadata is extracted from the document fragments to construct a metadata hierarchy tree corresponding to multiple semantic levels. Through a semantic alignment engine and adapter module, the introduced open-source external ontology is semantically mapped and aligned with the multiple semantic levels. The aligned metadata and external ontology knowledge are organized, classified and layered according to six semantic levels to form a structured hierarchy tree, and a hierarchical domain ontology and corresponding RDF knowledge graph are generated based on the structured hierarchy tree to improve the support of the knowledge graph for complex mechanism modeling and reasoning, as well as its usability and maintainability.

The invention discloses an intelligent tendering and bidding question-answering method based on document segmentation, and relates to the technical field of document segmentation, and the method comprises the steps: carrying out the layout analysis of a tendering and bidding document, so as to extract a structural unit set, construct a structural hierarchical relation, and carrying out the sentence processing of each structural unit; constructing a semantic window for the structural unit and generating a mixed semantic window vector, and performing primary segmentation on the structural unit by calculating a semantic gradient to generate a semantic continuous block set; second-level segmentation is carried out on the basis of the length constraint of the token, a text block set is generated, cross-level semantic coding is carried out on text blocks, and block-level vector codes are generated; outputting a local abstract for each text block according to a cross-level coding result, constructing a local abstract set, performing compression to generate a global abstract, and constructing a bidirectional mapping system of clause numbers and the text blocks; and efficient and traceable bidding and tendering document analysis and intelligent question and answer are provided for the user.

The invention provides a method and device for matting and displaying a magnetic file on a whiteboard, and the method comprises the steps: respectively collecting whiteboard data and magnetic file data, and correspondingly training a whiteboard detection model and a magnetic file segmentation model; performing whiteboard positioning and segmentation on an original image containing a magnetic file based on the whiteboard detection model to obtain a whiteboard area image; performing magnetic suction file segmentation on the white board area image based on the magnetic suction file segmentation model to obtain a target magnetic suction file image; and establishing association between the target magnetic file image and the corrected and amplified image through a mapping table, and realizing real-time matting display of the magnetic file. When a speaker demonstrates and explains the magnetic files on the whiteboard, only the effect presented by the magnetic files is achieved on a picture.

This application discloses a document segmentation method, apparatus, computer device, and storage medium. In response to a segmentation command, a document to be segmented is acquired; the document is parsed to obtain multiple text units; visual features related to the document layout are determined based on the text units; a segmentation score is calculated based on the visual features; and segmentation is performed based on the segmentation score. In this application, the visual layout information of the document is referenced from a visual layout perspective, avoiding the text extraction quality defects of treating the document as a plain text stream without relying on OCR. Instead, segmentation is performed by combining the visual geometric layout characteristics of the document when the user browses the text, conforming to the browsing patterns of users reading documents, accurately identifying document structural boundaries, and improving the accuracy of text segmentation.

A method for extracting document emotion reasons based on a graph neural network with edge features and document segmentation, belonging to the field of natural language processing. To solve the problem of improving the efficiency of emotion reason classification, the key points are as follows: obtaining emotion node factors and reason node factors according to the node vectors; pairing the emotion node factors and reason node factors pairwise to obtain edge feature vectors; constructing a fully connected graph through the node feature vector matrix and the edge feature vector matrix; updating the node features according to the fully connected graph, pairing the updated node features pairwise, splicing the paired node pairs with the edge features, and updating the spliced result to obtain the updated edge features; passing the updated edge features through a linear classifier to obtain the emotion reason relationship classification result, and using the emotion reason relationship classification result as the emotion reason relationship prediction result label. The effect is to improve the accuracy and efficiency of emotion reason classification.

A multi-page document positioning and printing system, apparatus, medium, and product are disclosed, relating to the field of printing systems. In this system, a data mapping module establishes a mapping relationship between first and second encoded information and stores it in a database; a document segmentation module splits the target document into independent units by page; a content recognition module performs intelligent text recognition on each single-page document and extracts key encoded information; a remote triggering module enables the collection of first encoded information via a mobile terminal and remote triggering of background processing, allowing operators to work flexibly in various areas of the warehouse without having to travel to fixed workstations; and a printing execution module accurately locates the single-page document containing the corresponding encoded information based on the established mapping relationship and drives printing. The entire system not only improves the operational efficiency of warehouse packing and shipping processes and reduces the error rate of searching and matching, but also meets the high requirements of cross-border e-commerce warehousing and logistics for rapid response and precise operation.

The embodiment of the invention provides a sample set construction method of power grid data and application of the sample set construction method in power grid management, and belongs to the technical field of data processing. The sample set construction method comprises the following steps: acquiring fault data scheduled by a distribution network in a power grid as sample data and importing the sample data into a data set; labeling the sample data imported into the data set; performing classification management on the labeled sample data; performing duplicate removal processing on different types of sample data in the data set; performing document segmentation on the sample data in the data set after the duplicate removal processing; and performing knowledge feature extraction on sample data in the data set after document segmentation, and storing the sample data in a database to obtain a sample set. The sample set construction method can collect and classify power grid data.