Retrieval-augmented generation method based on PDF text structure

Abstract

Provided is a retrieval-augmented generation method based on a PDF text structure. The method comprises: S1, performing extraction processing on structural elements of a PDF document, the structural elements of the PDF document including headings, paragraphs, tables, forms and images in the document, so as to obtain structured headings, paragraphs, tables, forms and images; S2, performing segmentation processing and indexing processing on each structural element extracted in step S1, and storing obtained text embedding vectors and metadata in a vector database; and S3, receiving a query inputted by a user and performing retrieval on the basis of the vector database, and outputting a retrieval result.

Description

A Search Enhancement Generation Method Based on PDF Text Structure Technical Field

[0001] This invention relates to the field of search enhancement generation technology, and more specifically, to a search enhancement generation method based on PDF text structure. Background Technology

[0002] Recent advancements in Large Language Models (LLMs) such as ChatGPT and ChatGLM have injected new vitality into practical artificial intelligence technologies and laid a solid technical foundation for the development of general artificial intelligence. However, due to the timeliness of training data and limitations in parameterized knowledge, these models suffer from knowledge gaps and knowledge illusions in some specialized fields, hindering their widespread application and implementation across industries. In contrast, the emergence of Retrieval Augmented Generation (RAG) technology, with its advantages of no training required, low cost, and high efficiency, effectively addresses the knowledge gap problem of large language models in specific domains, thus attracting widespread research and attention.

[0003] Domain-Specific Aqueous Question Answering (RAG) technology primarily leverages the powerful contextual learning and text understanding capabilities of LLM (Low-Level Learning) systems, combining this with relevant information retrieved from domain-specific knowledge bases to generate answers relevant to that domain. This technology addresses both the limitations of LLM in specific domain knowledge and the low efficiency of using such knowledge bases. The rapid development of RAG technology has greatly promoted the application of LLM across various fields, becoming a mainstream solution for AI applications such as domain-specific question answering systems. Simultaneously, the industry has also launched excellent technical frameworks such as LangChain and Llamaindex, paving the way for the practical application and widespread adoption of RAG technology.

[0004] In practical applications, although technical frameworks such as Langchain have embedded various RAG (Research and Animation) technology strategies and solutions, these technologies mainly focus on optimizing each stage of RAG technology. Furthermore, while these frameworks support a wide range of knowledge base types, they haven't designed comprehensive technical solutions for specific formats like PDF. PDF files, as crucial digital information carriers, are widely used in multiple fields. Currently, the development of RAG technology strategies for PDF documents in these frameworks is relatively superficial, failing to fully consider the impact of PDF's unique document structure on RAG performance. Therefore, based on the structural characteristics of PDF documents, this invention develops a complete RAG technology strategy and process, which is crucial for improving the performance of PDF-based RAG applications. Summary of the Invention

[0005] This invention provides a retrieval enhancement generation method based on PDF text structure to solve the technical problems existing in the prior art.

[0006] To achieve the above objectives, the present invention provides a retrieval enhancement generation method based on PDF text structure, which includes:

[0007] S1: Extract the structural elements of the PDF document. The structural elements of the PDF document include the title, paragraphs, tables, forms and images in the document, and obtain the structured title, paragraphs, tables, forms and images.

[0008] S2: Perform segmentation and indexing on each structural element extracted in step S1, and store the resulting text embedding vector and metadata into the vector database.

[0009] S3: Receives user-input queries and performs searches based on a vector database, then outputs the search results.

[0010] In one embodiment of the present invention, the heading includes a primary heading and secondary headings. The primary heading is the highest-level heading in the document, and a document includes one or more primary headings. Secondary headings are the headings at the next level below the primary headings, and a primary heading includes one or more secondary headings.

[0011] For each first-level heading in a document, extract the following data: Id, Text, Doc-id, and Doc-name, representing: the ID of the first-level heading, the text content of the first-level heading, the ID of the document to which the first-level heading belongs, and the document name, respectively.

[0012] Extract the following data for each level heading in the document: Id, Level, Parent-head, Text, Order, and Title, which represent: the ID of the subheading, the level of the subheading, the ID of the parent heading of the subheading, the text content of the subheading, the order of the subheading in the entire document, and the first heading to which the subheading belongs.

[0013] In one embodiment of the invention, the subheading is further divided into multiple levels.

[0014] In one embodiment of the present invention, a paragraph is a single paragraph or multiple consecutive paragraphs. When extracting paragraphs from a document, multiple consecutive paragraphs are extracted separately or one or more consecutive paragraphs are merged and then extracted. The following data is extracted from each paragraph of the document: Id, Belonged-To-Head, Order, and content, which have the following meanings: the paragraph's ID, the ID of the title to which the paragraph belongs, the order of paragraphs in the entire document, and the list of sentences in the paragraph.

[0015] In one embodiment of the present invention, the content in the form is extracted and stored in the form of key-value pairs. The following data is extracted for each form in the document: Id, Belonged-To-Head, Form-Title, and Form-Content, which have the following meanings: the form ID, the ID of the title to which the form belongs, the form title, and the form content, respectively.

[0016] In one embodiment of the present invention, the rows and columns of the table are of one or more levels. The content in the table is simplified into a redundant two-dimensional table form according to the principle of accurate information description and then extracted. The following data is extracted for each table in the document: Table-Id, Belonged-To-Head, Table-Title, Table-Header, Column-Headers, Row-Headers, and Table-Cells, which have the following meanings: table ID, Belonged-To-Head ID, table title, table header, [["First-level column header"], ["Second-level column header"], ["i-th level column header"]], [["First-level row header"], ["Second-level row header"], ["j-th level row header"]], [["First row data"], ["Second row data"], ["n-th row data"]], where i is the level of column header, j is the level of row header, and n is the total number of rows in the table.

[0017] In one embodiment of the present invention, the following data is extracted for each image in the document: Id, Belonged-To-Head, Image-Encode, Image-Caption, Is_Chart, and Image-Content, which respectively mean: the image ID, the ID of the title to which the image belongs, the base64 encoded data of the image, the title of the image, whether it is an infographic image, and the text content of the image. An infographic image refers to an image containing a statistical chart.

[0018] In one embodiment of the present invention, the processing flow for processing paragraphs to obtain embedding vectors in step S2 is as follows:

[0019] S21: Segment paragraphs according to sentence granularity;

[0020] S22: The segmented paragraphs are processed using a text embedding model to obtain text embedding vectors, text to be embedded, and text to be retrieved. The text embedding vectors are then stored in a vector database.

[0021] S23: Perform document tree path parsing on the paragraph to obtain the document tree path of the paragraph;

[0022] S24: Store the document tree path of the paragraph, the text to be retrieved, and the title ID to which the paragraph belongs as metadata of the text embedding vector in the vector database;

[0023] S25: Add a type field to the text embedding vector to indicate that the structural element corresponding to the text embedding vector is a paragraph.

[0024] In one embodiment of the present invention, in step S22, when the paragraph length exceeds the limit of the text embedding model, the following steps are performed:

[0025] S221: While ensuring sentence integrity, perform maximum length sliding segmentation on paragraphs to obtain multiple sentence vectors and multiple paragraph vectors;

[0026] S222: For each sentence, calculate the similarity between the sentence and the vectors of each paragraph and determine whether the maximum similarity is greater than the preset threshold T. If it is, discard the sentence.

[0027] S223: Organize the remaining sentence vectors and paragraph vectors into text embedding vectors. The texts corresponding to the text embedding vectors are the text to be embedded and the text to be retrieved.

[0028] In one embodiment of the present invention, step S23 is performed according to the following steps:

[0029] S231: Get the ID of the paragraph's heading;

[0030] S232: Retrieve the corresponding title text and the parent title ID of the title based on the ID of the title;

[0031] S233: Then, based on the parent title ID, obtain the corresponding parent title text and the parent title ID of that parent title;

[0032] S234: Recursively repeat step S233 until the top-level title and its corresponding document name are obtained. Store the obtained title and its corresponding document name in a preset format to obtain the document tree path of the paragraph.

[0033] In one embodiment of the present invention, the text to be embedded and the text to be retrieved are the same text.

[0034] In one embodiment of the present invention, the text to be embedded is the text corresponding to a single sentence, and the text to be retrieved is the text including a single sentence and a preset retrieval window length.

[0035] In one embodiment of the present invention, the processing flow for processing the table and form to obtain the text embedding vector in step S2 is as follows:

[0036] S21′: Use table semantic templates and form semantic templates to convert the original table and original form into table semantic text and form semantic text respectively, and obtain the text to be embedded for the table and form;

[0037] S22′: Describe the original table in Markdown format to obtain the text to be searched corresponding to the original table; describe the original form in key-value JSON format to obtain the text to be searched corresponding to the original form.

[0038] S23′: Store the text to be embedded, the text to be retrieved, and the ID of the table as the table's metadata in the vector database; store the text to be embedded, the text to be retrieved, and the ID of the form as the form's metadata in the vector database.

[0039] S24′: Vectorize the text to be embedded in the table and form, and store the text embedding vectors corresponding to the table and form in the vector database.

[0040] S25′: Add a type field to the text embedding vectors corresponding to tables and forms, indicating that the structural element corresponding to the text embedding vector is a table or a form.

[0041] In one embodiment of the present invention, the processing flow for obtaining the text embedding vector by processing the image in step S2 is as follows:

[0042] S21: Determine whether an image contains text.

[0043] If yes, then further determine whether the image is a scientific infographic. If yes, then call the infographic extraction model to convert the scientific infographic in the image into a Markdown table. Then use the table semantic template to convert the Markdown table into image semantic text and store it as the text to be retrieved in the vector database. Vectorize the image semantic text to obtain the text embedding vector corresponding to the image and store it in the vector database. Store the image storage location, image title, image semantic text, and Markdown table as image metadata in the vector database. If no, then extract the text in the image and store it as the text to be retrieved in the vector database. Segment the text in the image according to sentences, vectorize it according to sentence granularity to obtain the corresponding text embedding vector and store it in the vector database. Store the image storage location, image title, and text in the image as metadata of the text embedding vector in the vector database.

[0044] If no, determine whether image caption generation is needed. If yes, call the image caption generation model to generate image caption text and store it as the text to be retrieved in the vector database. Then, segment the image caption text into sentences and vectorize it at the sentence level to obtain the corresponding text embedding vector and store it in the vector database. Store the image storage location, image title, and image caption text as metadata of the text embedding vector in the vector database. If no, do not perform any processing.

[0045] S22‵: Store the image title as the text to be retrieved in the vector database, vectorize the image title and embed the generated text into the vector database;

[0046] S23‵: Add a type field to the image, indicating that the structuring element corresponding to the generated text embedding vector is an image.

[0047] In one embodiment of the present invention, the execution process of step S3 is as follows:

[0048] S31: Vectorize the user-input query to obtain a query vector;

[0049] S32: Use the query vector to search the vector database and obtain the K text embedding vectors that are most similar to the query vector and their corresponding metadata;

[0050] S33: Iterate through the metadata of the K text embedding vectors in sequence, and store the metadata of the text embedding vectors into the paragraph text list, table list, form list and image list in the following order according to the type of the text embedding vector:

[0051] (1) If the text embedding vector is of type paragraph, then:

[0052] Obtain the document tree path of the paragraph and the text to be retrieved, and organize them in dictionary form;

[0053] Store the document tree path of the paragraphs organized in dictionary form and the text to be retrieved into the paragraph text list;

[0054] (2) If the text embedding vector is of type table, then:

[0055] Obtain the semantic text of the table, the text of the table to be retrieved, and the table ID to which it belongs, and organize them in dictionary form;

[0056] Store the semantic text of the table, the table text to be searched, and the table ID to which they belong in a table list, organized in dictionary form;

[0057] (3) If the text embedding vector is of type form, then:

[0058] Retrieve the text to be retrieved from the form, the ID of the form to which it belongs, and the semantic text of the form, and organize them in dictionary form;

[0059] Store the text to be retrieved, the ID of the form to which it belongs, and the semantic text of the form in a form list organized in dictionary form;

[0060] (4) If the text embedding vector is of type image, then:

[0061] Obtain image semantic text, text within the image, and image caption text, and organize them in dictionary form;

[0062] Store the image semantic text, text in the image, and image caption text, organized in dictionary form, into the image list;

[0063] S34: Traverse the list of paragraph texts, extract the text to be retrieved and store it in the candidate context list, and return the document tree path as reference information;

[0064] S35: Iterate through the list of tables and get the length M of the list of tables.

[0065] To determine if text embedding vectors exceeding M / 2 correspond to the IDs of the same table:

[0066] If so, the text to be retrieved corresponding to that table is stored in the candidate context list, and the table ID is returned as the reference information. For the remaining elements, the semantic text of the table corresponding to each element is stored in the candidate context list, and the table ID is returned as the reference information.

[0067] If not, store the table semantic text corresponding to each text embedding vector into the candidate context list, and return the table ID as reference information;

[0068] S36: Traverse the form list, extract the text to be retrieved from each text embedding vector, store it in the candidate context list, and return the form ID as reference information;

[0069] S37: Traverse the image list and extract all the image semantic text, text in the image, and image caption text corresponding to each text embedding vector, and store them in the candidate context list;

[0070] S38: Fill in the candidate context list sequentially according to the prompt template of the large language model, and send it into the large language model together with the user query or question to obtain the search results.

[0071] This invention provides a retrieval enhancement generation method based on PDF text structure. Combining the characteristics and processing requirements of different elements of the PDF document structure, it proposes a complete RAG strategy, covering the main stages of the RAG process, including segmentation, indexing, and retrieval. This provides an effective and universal technical solution for domain-specific RAG applications based on PDF document knowledge bases. Built upon the structural features of PDF documents, this invention, compared to solutions in open-source technical frameworks, more comprehensively considers the characteristics of different structural elements (such as tables, headings, and images), resulting in better performance and greater adaptability to complex application scenarios such as reasoning and global question answering. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 is a schematic diagram of the overall process of a retrieval enhancement generation method based on PDF text structure according to an embodiment of the present invention;

[0074] Figure 2 shows the application code used to extract the title in an embodiment of the present invention;

[0075] Figure 3 shows the application code used to extract paragraphs according to an embodiment of the present invention;

[0076] Figure 4 shows the application code used when extracting a form according to an embodiment of the present invention;

[0077] Figure 5 shows the application code used when extracting a table according to an embodiment of the present invention;

[0078] Figure 6 shows the application code used when extracting images according to an embodiment of the present invention;

[0079] Figure 7 is a schematic diagram of the specific process of segmenting and indexing structural elements according to an embodiment of the present invention;

[0080] Figure 8 is a schematic diagram of the search results obtained according to an embodiment of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] Figure 1 is a schematic diagram of the overall process of a retrieval enhancement generation method based on PDF text structure according to an embodiment of the present invention. As shown in Figure 1, the retrieval enhancement generation method based on PDF text structure provided by the present invention includes:

[0083] S1: Extract the structural elements of the PDF document. The structural elements of the PDF document include the title, paragraphs, tables, forms and images in the document, and obtain the structured title, paragraphs, tables, forms and images.

[0084] S2: Perform segmentation and indexing on each structural element extracted in step S1, and store the resulting text embedding vector and metadata into the vector database.

[0085] S3: Receives user-input queries and performs searches based on a vector database, then outputs the search results.

[0086] The following describes the specific process of extracting various structural elements from a PDF document:

[0087] Titles are typically concise in textual semantics and lack rich morpheme objects in form, describing the text content or theme under the title with relatively brief phrases. In one embodiment of the present invention, a title includes a primary title and secondary titles. The primary title is the highest-level title in the document, and a document includes one or more primary titles. Secondary titles are the titles at the next level below the primary titles, and a primary title includes one or more secondary titles. Generally, the highest-level title in a document is the Title. A document can have multiple Titles, analogous to the titles of different chapters in a paper. Their main characteristics are centered alignment, relatively clear semantics and themes of text content under the same Title, and relatively independent semantic themes between different Titles. Different levels of titles can be understood as different headers. A Title can store multiple levels of headers, similar to different summaries under a chapter in a paper. Their main characteristic is that they are usually left-aligned in layout. Figure 2 shows the application code used to extract titles in one embodiment of the present invention.

[0088] As shown in Figure 2, the following data is extracted for each first-level heading (Title) of the document: Id, Text, Doc-id, and Doc-name, representing: the ID of the first-level heading, the text content of the first-level heading, the ID of the document to which the first-level heading belongs, and the document name, respectively.

[0089] As shown in Figure 2, the following data is extracted from each level heading (Header) of the document: Id, Level, Parent-head, Text, Order, and Title, which mean: the ID of the subheading, the level of the subheading, the ID of the parent heading of the subheading, the text content of the subheading, the order of the subheading in the entire document, and the first heading to which the subheading belongs.

[0090] In other embodiments of the present invention, the subheadings are further divided into multiple levels, and the extraction method of the subheadings at multiple levels is as shown in Figure 2 and so on. The present invention will not elaborate on this further.

[0091] Paragraphs constitute the main body of a document's content. They typically belong to a heading, but are not subheadings of that heading. Their main characteristics are relatively rich text, relatively complete and rich semantics, and a full range of morphemes. A heading can have multiple paragraphs. The boundaries between paragraphs under different headings are relatively clear, while the boundaries between different paragraphs under the same heading can be blurred. For paragraphs with blurred boundaries, they can be merged into a single paragraph during extraction. This merging usually does not affect the understanding of the text's semantics and has minimal impact on the document structure.

[0092] In one embodiment of the present invention, a paragraph is a single paragraph or multiple consecutive paragraphs. When extracting paragraphs from a document, multiple consecutive paragraphs are extracted separately or one or more consecutive paragraphs are merged and then extracted. As shown in Figure 3, the application code used to extract paragraphs in one embodiment of the present invention extracts the following data for each paragraph of the document: Id, Belonged-To-Head, Order, and content, which have the following meanings: the paragraph's ID, the ID of the title to which the paragraph belongs, the order of paragraphs in the entire document, and the list of sentences in the paragraph.

[0093] Forms can typically be simplified to a key-value format, meaning that each field in a form usually corresponds to a unique blank space to be filled in. Forms can generally be converted into a "field: content" format, and forms are usually categorized under a heading. Form content is described in list form, with each element using a key-value dictionary to describe a specific field; values ​​with multiple values ​​are described using a list.

[0094] In one embodiment of the present invention, the content in the form is extracted and stored in the form of key-value pairs. As shown in Figure 4, the application code used to extract the form in one embodiment of the present invention is as follows: the following data is extracted for each form in the document: Id, Belonged-To-Head, Form-Title and Form-Content, which have the following meanings: the form ID, the ID of the title to which the form belongs, the title of the form and the content of the form, respectively.

[0095] Unlike form content, table content typically includes elements such as row headers, column headers, and values ​​for different cells. Real-world tables often have multiple formats. This invention converts all tables into a two-dimensional table description. For multi-level row / column headers and cross-row headers, they are simplified into redundant two-dimensional tables according to the principle of accurate information description, ensuring that each cell's table value has a corresponding row and column header. Tables are usually also categorized under a specific heading.

[0096] In one embodiment of the present invention, the rows and columns of the table are of one or more levels. The content in the table is simplified into a redundant two-dimensional table form according to the principle of accurate information description and then extracted. As shown in Figure 5, the application code used to extract the table in one embodiment of the present invention is as follows: the following data is extracted for each table in the document: Table-Id, Belonged-To-Head, Table-Title, Table-Header, Column-Headers, Row-Headers, and Table-Cells, which have the following meanings: table ID, the ID of the title to which the table belongs, the table title, the table header, [["first-level column header"], ["second-level column header"], ["i-th level column header"]], [["first-level row header"], ["second-level row header"], ["j-th level row header"]], [["first row data"], ["second row data"], ["n-th row data"]], where i is the level of column header, j is the level of row header, and n is the total number of rows in the table. Column-Headers represent the column headers of the table. Multiple levels of column headers are allowed, and a single column header can span multiple columns. However, the length of all column headers must be consistent in the extracted results. For column headers spanning multiple columns and rows, they should be split into multiple cells, with each cell containing consistent content. Row-Headers represent the row headers of the table. Multiple levels of row headers are allowed, and a single row header can span multiple rows. However, the length of all row headers must be consistent in the extracted results. For column headers spanning multiple rows and columns, they should be split into multiple cells, with each cell containing consistent content. Each element in the Column-Headers list is a list of column headers at a specific level, and the length of the header list must be consistent. Each element in the Row-Headers list is a list of row headers at a specific level, and the length of the header list must be consistent. The Table-Cells list is a list of the actual table data, with each element corresponding to a row of data. The length of the data list must be consistent.

[0097] Images can be categorized based on whether they contain text and the type of text they contain: textless images, non-infographic images, and infographic images. Infographic images refer to images that include statistical charts such as line charts and histograms, and these images are usually also associated with a title.

[0098] Figure 6 shows the application code used to extract images in an embodiment of the present invention. The following data is extracted for each image in the document: Id, Belonged-To-Head, Image-Encode, Image-Caption, Is_Chart, and Image-Content, which have the following meanings: image ID, ID of the title to which the image belongs, base64 encoded data of the image, title of the image, whether it is an infographic image, and text content of the image. An infographic image refers to an image containing a statistical chart.

[0099] In addition to the common structural elements mentioned above, PDF documents also contain other structural elements. These other structural elements typically contain less information and do not significantly affect the understanding of the document content. Other structural elements, such as headers and footers, can be configured in the PDF parsing settings to determine which structural elements are classified as "other structural elements." This invention ignores all other structural elements by default.

[0100] Figure 7 is a schematic diagram of the specific process of segmenting and indexing structural elements according to an embodiment of the present invention. As shown in Figure 7, the specific process of segmenting and indexing each structural element extracted in step S1 is described below:

[0101] In one embodiment of the present invention, the processing flow for processing paragraphs to obtain embedding vectors in step S2 is as follows:

[0102] S21: Segment paragraphs according to sentence granularity;

[0103] S22: The segmented paragraphs are processed using a text embedding model to obtain text embedding vectors, text to be embedded, and text to be retrieved. The text embedding vectors are then stored in a vector database.

[0104] S23: Perform document tree path parsing on the paragraph to obtain the document tree path of the paragraph;

[0105] S24: Store the document tree path of the paragraph, the text to be retrieved, and the title ID to which the paragraph belongs as metadata of the text embedding vector in the vector database;

[0106] In other embodiments, only the text embedding vector can be stored in the vector database, while the metadata can be stored in other databases. In this case, it is necessary to establish a mapping relationship between the vector database and other databases to facilitate later retrieval and use.

[0107] S25: Add a type field to the text embedding vector to indicate that the structural element corresponding to the text embedding vector is a paragraph.

[0108] In one embodiment of the present invention, in step S22, when the paragraph length exceeds the limit of the text embedding model, the following steps are performed:

[0109] S221: While ensuring sentence integrity, perform maximum length sliding segmentation on paragraphs to obtain multiple sentence vectors and multiple paragraph vectors;

[0110] For example, a paragraph contains six sentences: [sentence1, sentence2, sentence3, sentence4, sentence5, sentence6]. Each of these six sentences is vectorized to obtain six sentence vectors. The lengths of the six sentences are [100, 234, 200, 180, 30, 200]. Since the downstream text embedding model has a length limit of 512, the paragraph is then subjected to maximum length sliding segmentation.

[0111] Sentence1 and sentence2 are vectorized to form a single paragraph vector 1.

[0112] Sentence2 and sentence3 are vectorized to form a paragraph vector 2.

[0113] Sentence3, sentence4, and sentence5 are vectorized to form a single paragraph vector 3.

[0114] Sentence4, sentence5, and sentence6 are vectorized to form a single paragraph vector 4.

[0115] This results in two types of embedding vectors: sentence vectors and paragraph vectors. In this example, there are 6 sentence vectors, corresponding to the 6 sentences sentence1 to sentence6, and 4 paragraph vectors, namely paragraph vector 1 to paragraph vector 4.

[0116] S222: For each sentence, calculate the similarity between the sentence and the vectors of each paragraph and determine whether the maximum similarity is greater than the preset threshold T. If it is, discard the sentence.

[0117] In this example, assuming the preset threshold T is 0.9, the similarity of Sentence1 with paragraph vectors 1 through 4 is calculated sequentially. If the similarity results are 0.95, 0.6, 0.7, and 0.5 respectively, the maximum similarity of 0.95 is compared. Since this value is greater than the preset threshold T (0.95 > 0.9), Sentence1 is discarded, and its corresponding sentence vector is also no longer used. If the maximum similarity of Sentence5 with paragraph vectors 1 through 4 is 0.8 (0.8 < 0.95), then Sentence5 and its corresponding sentence vector are retained.

[0118] S223: Organize the remaining sentence vectors and paragraph vectors into a text embedding vector. The text corresponding to the text embedding vector is the text to be embedded and the text to be retrieved. This invention assumes that the text to be embedded and the text to be retrieved are the same text.

[0119] The PDF document structure itself is divided into levels of document---Title--header---paragraph. The path from leaf node to root node in the document tree is also planned in this way. Therefore, the method to obtain the document tree path of a paragraph is as follows.

[0120] In one embodiment of the present invention, step S23 is performed according to the following steps:

[0121] S231: Get the ID of the paragraph's heading;

[0122] S232: Retrieve the corresponding title text and the parent title ID of the title based on the ID of the title;

[0123] S233: Then, based on the parent title ID, obtain the corresponding parent title text and the parent title ID of that parent title;

[0124] S234: Recursively repeat step S233 until the top-level title and its corresponding document name are obtained. Store the obtained title and its corresponding document name in a preset format to obtain the document tree path of the paragraph.

[0125] In another embodiment of the present invention, the text to be embedded is different from the text to be retrieved. The text to be embedded is the text corresponding to a single sentence, while the text to be retrieved is the text of a preset retrieval window length including the single sentence. For example, when the preset retrieval window length is 1, the text to be retrieved corresponding to a sentence 2 in a paragraph is the sentence before and after that sentence, that is: the text to be embedded is sentence2, a text embedding vector sentence2_vector is obtained based on sentence2, and the text to be retrieved is sentence1+sentence2+sentence3.

[0126] In one embodiment of the present invention, the processing flow for processing the table and form to obtain the text embedding vector in step S2 is as follows:

[0127] S21′: Use table semantic templates and form semantic templates to convert the original table and original form into table semantic text and form semantic text respectively, and obtain the text to be embedded for the table and form;

[0128] In practice, different semantic templates can be used for different characteristics of tabular and form data. For tables, different semantic templates can be used based on the size of the table, as shown in the following example:

[0129] Table Semantic Template 1: For larger tables, such as those with a total character count exceeding a certain number, a semi-structured descriptive text is generated, organized by row or column. This is primarily to reduce redundant text and describe the table's semantics from a framework perspective. A possible semantic template is: "The table title is {{Table-Title}}, its row headers are {{Row-Headers}}, and its column headers are {{Column-Headers}}." Using this method, a table description corresponds to a semantic description.

[0130] Table Semantic Template 2: For smaller tables, such as those with a total character count below a certain limit, a semi-structured descriptive text is generated for each cell of each table. This method may result in more text embeddings. One possible semantic template is: "The table title is {{Table-Title}}, where {{rowheaders1}}'s {{columnheader1}} is {{cell value}}." Using this method, each cell in the Table-Cells corresponds to a descriptive text; rowheaders1 is the row header corresponding to the cell in the Table-Cells; columnheader1 is the column header corresponding to the cell in the Table-Cells.

[0131] For forms, since their semantic information is relatively simple, the following semantic templates can be used:

[0132] Form semantic template: "The field {{field-1}} of the form {{Form-Title}} has a value of {{value-1}}, the field {{field-2}} has a value of {{value-2}}, ...." Here, field-1 and field-2 are the fields in the form, and value-1 and value-2 are the corresponding values. This is just an example; for more fields, you can add more descriptions by referring to the template. A form typically generates a semantic description text.

[0133] In this step, the original table and original form are converted into table semantic text and form semantic text respectively by using table semantic template and form semantic template, so as to obtain the text to be embedded for the table and form.

[0134] S22′: Describe the original table in Markdown format to obtain the text to be searched corresponding to the original table; describe the original form in key-value JSON format to obtain the text to be searched corresponding to the original form.

[0135] S23′: Store the text to be embedded, the text to be retrieved, and the ID of the table as the table's metadata in the vector database; store the text to be embedded, the text to be retrieved, and the ID of the form as the form's metadata in the vector database.

[0136] S24′: Vectorize the text to be embedded in the table and form, and store the text embedding vectors corresponding to the table and form in the vector database.

[0137] S25′: Add a type field to the text embedding vectors corresponding to tables and forms, indicating that the structural element corresponding to the text embedding vector is a table or a form.

[0138] In one embodiment of the present invention, the processing flow for obtaining the text embedding vector by processing the image in step S2 is as follows:

[0139] S21: Determine whether an image contains text.

[0140] If yes, then further determine whether the image is a scientific infographic. If yes, then call the infographic extraction model to convert the scientific infographic in the image into a Markdown table. Then use the table semantic template to convert the Markdown table into image semantic text and store it as the text to be retrieved in the vector database. Vectorize the image semantic text to obtain the text embedding vector corresponding to the image and store it in the vector database. Store the image storage location, image title, image semantic text, and Markdown table as image metadata in the vector database. If no, then extract the text in the image and store it as the text to be retrieved in the vector database. Segment the text in the image according to sentences, vectorize it according to sentence granularity to obtain the corresponding text embedding vector and store it in the vector database. Store the image storage location, image title, and text in the image as metadata of the text embedding vector in the vector database.

[0141] If no, determine whether image caption generation is needed. If yes, call the image caption generation model to generate image caption text and store it as the text to be retrieved in the vector database. Then, segment the image caption text into sentences and vectorize it at the sentence level to obtain the corresponding text embedding vector and store it in the vector database. Store the image storage location, image title, and image caption text as metadata of the text embedding vector in the vector database. If no, do not perform any processing.

[0142] S22‵: Store the image title as the text to be retrieved in the vector database, vectorize the image title and embed the generated text into the vector database;

[0143] S23‵: Add a type field to the image, indicating that the structuring element corresponding to the generated text embedding vector is an image.

[0144] In one embodiment of the present invention, the execution process of step S3 is as follows:

[0145] S31: Vectorize the user-input query to obtain a query vector;

[0146] S32: Use the query vector to search the vector database and obtain the K text embedding vectors that are most similar to the query vector and their corresponding metadata;

[0147] In this step, the retrieval is performed by calculating the similarity between vectors. The retrieval methods include dense vector retrieval or sparse keyword retrieval.

[0148] S33: Iterate through the metadata of the K text embedding vectors in sequence, and store the metadata of the text embedding vectors into the paragraph text list, table list, form list and image list in the following order according to the type of the text embedding vector:

[0149] (1) If the text embedding vector is of type paragraph, then:

[0150] Obtain the document tree path of the paragraph and the text to be retrieved, and organize them in dictionary form;

[0151] Store the document tree path of the paragraphs organized in dictionary form and the text to be retrieved into the paragraph text list;

[0152] (2) If the text embedding vector is of type table, then:

[0153] Obtain the semantic text of the table, the text of the table to be retrieved, and the table ID to which it belongs, and organize them in dictionary form;

[0154] Store the semantic text of the table, the table text to be searched, and the table ID to which they belong in a table list, organized in dictionary form;

[0155] (3) If the text embedding vector is of type form, then:

[0156] Retrieve the text to be retrieved from the form, the ID of the form to which it belongs, and the semantic text of the form, and organize them in dictionary form;

[0157] Store the text to be retrieved, the ID of the form to which it belongs, and the semantic text of the form in a form list organized in dictionary form;

[0158] (4) If the text embedding vector is of type image, then:

[0159] Obtain image semantic text, text within the image, and image caption text, and organize them in dictionary form;

[0160] Store the image semantic text, text in the image, and image caption text, organized in dictionary form, into the image list;

[0161] S34: Traverse the list of paragraph texts, extract the text to be retrieved and store it in the candidate context list, and return the document tree path as reference information;

[0162] S35: Iterate through the list of tables and get the length M of the list of tables.

[0163] To determine if text embedding vectors exceeding M / 2 correspond to the IDs of the same table:

[0164] If so, the text to be retrieved corresponding to that table is stored in the candidate context list, and the table ID is returned as the reference information. For the remaining elements, the semantic text of the table corresponding to each element is stored in the candidate context list, and the table ID is returned as the reference information.

[0165] If not, store the table semantic text corresponding to each text embedding vector into the candidate context list, and return the table ID as reference information;

[0166] S36: Traverse the form list, extract the text to be retrieved from each text embedding vector, store it in the candidate context list, and return the form ID as reference information;

[0167] S37: Traverse the image list and extract all the image semantic text, text in the image, and image caption text corresponding to each text embedding vector, and store them in the candidate context list;

[0168] S38: Fill in the candidate context list sequentially according to the prompt template of the large language model, and send it into the large language model together with the user query or question to obtain the retrieval results. Figure 8 shows a schematic diagram of the retrieval results obtained in an embodiment of the present invention.

[0169] This invention provides a retrieval enhancement generation method based on PDF text structure. Combining the characteristics and processing requirements of different elements of the PDF document structure, it proposes a complete RAG strategy, covering the main stages of the RAG process, including segmentation, indexing, and retrieval. This provides an effective and universal technical solution for domain-specific RAG applications based on PDF document knowledge bases. Built upon the structural features of PDF documents, this invention, compared to solutions in open-source technical frameworks, more comprehensively considers the characteristics of different structural elements (such as tables, headings, and images), resulting in better performance and greater adaptability to complex application scenarios such as reasoning and global question answering.

[0170] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0171] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A retrieval enhancement generation method based on PDF text structure, characterized in that, include: S1: Extract the structural elements of the PDF document. The structural elements of the PDF document include the title, paragraphs, tables, forms and images in the document, and obtain the structured title, paragraphs, tables, forms and images. S2: Perform segmentation and indexing on each structural element extracted in step S1, and store the resulting text embedding vector and metadata into the vector database. S3: Receives user-input queries and performs searches based on a vector database, then outputs the search results.

2. The retrieval enhancement generation method based on PDF text structure according to claim 1, characterized in that, Headings include primary headings and secondary headings. Primary headings are the highest-level headings in a document, and a document may contain one or more primary headings. Secondary headings are the next level of headings below primary headings, and a primary heading may contain one or more secondary headings. For each first-level heading in a document, extract the following data: Id, Text, Doc-id, and Doc-name, representing: the ID of the first-level heading, the text content of the first-level heading, the ID of the document to which the first-level heading belongs, and the document name, respectively. Extract the following data for each level heading in the document: Id, Level, Parent-head, Text, Order, and Title, which represent: the ID of the subheading, the level of the subheading, the ID of the parent heading of the subheading, the text content of the subheading, the order of the subheading in the entire document, and the first heading to which the subheading belongs.

3. The retrieval enhancement generation method based on PDF text structure according to claim 2, characterized in that, Subheadings are further divided into multiple levels.

4. The retrieval enhancement generation method based on PDF text structure according to claim 1, characterized in that, A paragraph can be a single paragraph or multiple consecutive paragraphs. When extracting paragraphs from a document, multiple consecutive paragraphs are extracted separately or one or more consecutive paragraphs are merged before extraction. For each paragraph in the document, the following data is extracted: Id, Belonged-To-Head, Order, and content, which mean: the paragraph's ID, the ID of the heading to which the paragraph belongs, the order of the paragraphs in the entire document, and the list of sentences in the paragraph, respectively.

5. The retrieval enhancement generation method based on PDF text structure according to claim 1, characterized in that, The content in the form is extracted and stored in the form of key-value pairs. For each form in the document, the following data is extracted: Id, Belonged-To-Head, Form-Title, and Form-Content, which mean: the form ID, the ID of the title to which the form belongs, the form title, and the form content, respectively.

6. The retrieval enhancement generation method based on PDF text structure according to claim 1, characterized in that, The tables have one or more levels of rows and columns. The content of the tables is simplified into a redundant two-dimensional table format based on the principle of accurate information description. For each table in the document, the following data is extracted: Table-Id, Belonged-To-Head, Table-Title, Table-Header, Column-Headers, Row-Headers, and Table-Cells, representing: the table ID, the ID of the header to which the table belongs, the table title, the table header, [["First-level column header"], ["Second-level column header"], ["i-th level column header"]], [["First-level row header"], ["Second-level row header"], ["j-th level row header"]], [["First row data"], ["Second row data"], ["n-th row data"]], where i is the column header level, j is the row header level, and n is the total number of rows in the table.

7. The retrieval enhancement generation method based on PDF text structure according to claim 1, characterized in that, Extract the following data for each image in the document: Id, Belonged-To-Head, Image-Encode, Image-Caption, Is_Chart, and Image-Content, which respectively represent: the image ID, the ID of the title to which the image belongs, the base64 encoded data of the image, the title of the image, whether it is an infographic image, and the text content of the image. An infographic image refers to an image that contains statistical charts.

8. The retrieval enhancement generation method based on PDF text structure according to claim 1, characterized in that, In step S2, the processing flow for processing paragraphs to obtain embedding vectors is as follows: S21: Segment paragraphs according to sentence granularity; S22: The segmented paragraphs are processed using a text embedding model to obtain text embedding vectors, text to be embedded, and text to be retrieved. The text embedding vectors are then stored in a vector database. S23: Perform document tree path parsing on the paragraph to obtain the document tree path of the paragraph; S24: Store the document tree path of the paragraph, the text to be retrieved, and the title ID to which the paragraph belongs as metadata of the text embedding vector in the vector database; S25: Add a type field to the text embedding vector to indicate that the structural element corresponding to the text embedding vector is a paragraph.

9. The retrieval enhancement generation method based on PDF text structure according to claim 8, characterized in that, In step S22, when the paragraph length exceeds the limit of the text embedding model, the following steps are performed: S221: While ensuring sentence integrity, perform maximum length sliding segmentation on paragraphs to obtain multiple sentence vectors and multiple paragraph vectors; S222: For each sentence, calculate the similarity between the sentence and the vectors of each paragraph and determine whether the maximum similarity is greater than the preset threshold T. If it is, discard the sentence. S223: Organize the remaining sentence vectors and paragraph vectors into text embedding vectors. The texts corresponding to the text embedding vectors are the text to be embedded and the text to be retrieved.

10. The retrieval enhancement generation method based on PDF text structure according to claim 8, characterized in that, Step S23 is performed according to the following steps: S231: Get the ID of the paragraph's heading; S232: Retrieve the corresponding title text and the parent title ID of the title based on the ID of the title; S233: Then, based on the parent title ID, obtain the corresponding parent title text and the parent title ID of that parent title; S234: Recursively repeat step S233 until the top-level title and its corresponding document name are obtained. Store the obtained title and its corresponding document name in a preset format to obtain the document tree path of the paragraph.

11. The retrieval enhancement generation method based on PDF text structure according to claim 8, characterized in that, The text to be embedded and the text to be retrieved are the same text.

12. The retrieval enhancement generation method based on PDF text structure according to claim 8, characterized in that, The text to be embedded is the text corresponding to a single sentence, and the text to be retrieved is the text of a preset retrieval window length, including the single sentence.

13. The retrieval enhancement generation method based on PDF text structure according to claim 1, characterized in that, In step S2, the processing flow for obtaining text embedding vectors by processing tables and forms is as follows: S21′: Use table semantic templates and form semantic templates to convert the original table and original form into table semantic text and form semantic text respectively, and obtain the text to be embedded for the table and form; S22′: Describe the original table in Markdown format to obtain the text to be searched corresponding to the original table; describe the original form in key-value JSON format to obtain the text to be searched corresponding to the original form. S23′: Store the text to be embedded, the text to be retrieved, and the ID of the table as the table's metadata in the vector database; store the text to be embedded, the text to be retrieved, and the ID of the form as the form's metadata in the vector database. S24′: Vectorize the text to be embedded in the table and form, and store the text embedding vectors corresponding to the table and form in the vector database. S25′: Add a type field to the text embedding vectors corresponding to tables and forms, indicating that the structural element corresponding to the text embedding vector is a table or a form.

14. The retrieval enhancement generation method based on PDF text structure according to claim 1, characterized in that, In step S2, the process of processing the image to obtain the text embedding vector is as follows: S21: Determine whether an image contains text. If yes, then further determine whether the image is a scientific infographic. If yes, call the infographic extraction model to convert the scientific infographic in the image into a Markdown table. Then, use the table semantic template to convert the Markdown table into image semantic text and store it as the text to be retrieved in the vector database. Vectorize the image semantic text to obtain the text embedding vector corresponding to the image and store it in the vector database. Store the image storage location, image title, image semantic text, and Markdown table as image metadata in the vector database. If no, extract the text in the image and store it as the text to be retrieved in the vector database. Segment the text in the image according to sentences, vectorize it according to sentence granularity to obtain the corresponding text embedding vector and store it in the vector database. Store the image storage location, image title, and text in the image as metadata of the text embedding vector in the vector database. If no, determine whether image caption generation is needed. If yes, call the image caption generation model to generate image caption text and store it as the text to be retrieved in the vector database. Then, segment the image caption text into sentences and vectorize it at the sentence level to obtain the corresponding text embedding vector and store it in the vector database. Store the image storage location, image title, and image caption text as metadata of the text embedding vector in the vector database. If no, do not perform any processing. S22‵: Store the image title as the text to be retrieved in the vector database, vectorize the image title and embed the generated text into the vector database; S23‵: Add a type field to the image, indicating that the structuring element corresponding to the generated text embedding vector is an image.

15. The retrieval enhancement generation method based on PDF text structure according to claim 1, characterized in that, The execution process of step S3 is as follows: S31: Vectorize the user-input query to obtain a query vector; S32: Use the query vector to search the vector database and obtain the K text embedding vectors that are most similar to the query vector and their corresponding metadata; S33: Iterate through the metadata of the K text embedding vectors in sequence, and store the metadata of the text embedding vectors into the paragraph text list, table list, form list and image list in the following order according to the type of the text embedding vector: (1) If the text embedding vector is of type paragraph, then: Obtain the document tree path of the paragraph and the text to be retrieved, and organize them in dictionary form; Store the document tree path of the paragraphs organized in dictionary form and the text to be retrieved into the paragraph text list; (2) If the text embedding vector is of type table, then: Obtain the semantic text of the table, the text of the table to be retrieved, and the table ID to which it belongs, and organize them in dictionary form; Store the semantic text of the table, the table text to be searched, and the table ID to which they belong in a table list, organized in dictionary form; (3) If the text embedding vector is of type form, then: Retrieve the text to be retrieved from the form, the ID of the form to which it belongs, and the semantic text of the form, and organize them in dictionary form; Store the text to be retrieved, the ID of the form to which it belongs, and the semantic text of the form in a form list organized in dictionary form; (4) If the text embedding vector is of type image, then: Obtain image semantic text, text within the image, and image caption text, and organize them in dictionary form; Store the image semantic text, text in the image, and image caption text, organized in dictionary form, into the image list; S34: Traverse the list of paragraph texts, extract the text to be retrieved and store it in the candidate context list, and return the document tree path as reference information; S35: Iterate through the list of tables and get the length M of the list of tables. To determine if text embedding vectors exceeding M / 2 correspond to the IDs of the same table: If so, the text to be retrieved corresponding to that table is stored in the candidate context list, and the table ID is returned as the reference information. For the remaining elements, the semantic text of the table corresponding to each element is stored in the candidate context list, and the table ID is returned as the reference information. If not, store the table semantic text corresponding to each text embedding vector into the candidate context list, and return the table ID as reference information; S36: Traverse the form list, extract the text to be retrieved from each text embedding vector, store it in the candidate context list, and return the form ID as the reference information; S37: Traverse the image list and extract all the image semantic text, text in the image, and image caption text corresponding to each text embedding vector, and store them in the candidate context list; S38: Fill in the candidate context list sequentially according to the prompt template of the large language model, and send it into the large language model together with the user query or question to obtain the search results.

Images