The invention discloses a PDF (
Portable Document Format) identification and preprocessing method for
large model application. The method comprises the following steps: S1, file reading and primary
processing: reading and
processing the content of a PDF file; s2, document content replacement: identifying and extracting a text segment by using LLamaIndex, and replacing the text segment with the text content extracted by PDFP luber; s3, overlapping the content between the pages: from the second page, splicing the content
tail of the previous page with the content head of the current page; s4, node manufacturing and
processing; s5, performing preliminary segmentation; s6, vectorization processing: calculating the
cosine similarity of the embedded vector, and converting the
cosine similarity into a
cosine distance; s7, threshold setting and semantic segmentation: setting a
distance threshold, and segmenting the text into a plurality of paragraphs; s8, summarizing and storing text segments; s9, processing the table; and S10, processing the picture. According to the method, the PDF content is efficiently preprocessed by adopting the PDFP blue
library, the OCR and the semantic segmentation technology, and the method has the advantages of high
data accuracy, high processing efficiency and wide application range.