Unstructured Document Extraction with Table Separation and Traceability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing RAG-based systems struggle with accurately extracting data from PDFs due to the intertwining of tabular and document text, leading to semantic meaning loss and increased computational inefficiencies, especially when feeding large language models, which results in incorrect responses and higher energy and operational costs.
Innovation Solution
The system separates tabular information from document text using markdown language tags during the OCR process, allowing the large language model to process both types of data independently, improving the quality of document retrieval and reducing computational effort.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If tabular information and document text are processed together in existing RAG-based systems, then the system can handle complete documents, but data extraction accuracy deteriorates due to semantic meaning loss and computational inefficiencies
Solution Approach 1:
The patent segments the document processing by separating tabular information from narrative text into distinct processing streams. Tables are converted to structured formats (CSV, JSON, or SQL) and processed independently from the text content, allowing each type of information to be handled with appropriate methods and improving both accuracy and efficiency.
Solution Approach 2:
The patent extracts tabular data from the document and removes it from the text processing flow. By pulling out tables and converting them to structured formats separately, the system avoids the semantic confusion that occurs when tables and text are processed together, thereby improving data extraction accuracy while reducing computational overhead.
2Reliability
If all document portions are fed to the large language model, then comprehensive information is available, but computational costs and energy consumption increase
Solution Approach 1:
The patent applies partial action by feeding only the necessary document portions to the large language model. After converting tables to structured formats and processing them separately, only the relevant text segments that require LLM interpretation are submitted, reducing energy consumption while maintaining response quality through targeted processing.
3Loss of information
If tabular and document text are intertwined during OCR processing, then complete document content is captured, but semantic meaning is lost leading to incorrect responses
Solution Approach 1:
The patent segments the OCR output processing by separating tabular structures from narrative text. By identifying and extracting tables from the OCR output and converting them to structured formats independently, the system preserves the semantic meaning of both tables and text without the confusion that arises from intertwining them, while managing complexity through systematic processing steps.
Data Source
AI summary
A system for extracting a number of data elements from one or more unstructured data sources. The system may separate the text from the tables in a document, such that only the table data may be sent to the large language model (LLM), when the LLM only needs to review the table data. The system generates chunks from the document. The system associates unique identifiers with each chunk to provide traceability. The system identifies relevant chunks from the documents and includes the relevant chunks with a request to extract the data elements in a prompt to the LLM. The system also includes a request for the LLM to report the chunks used during extraction of the data elements. The reported chunks are stored with the extracted data for verification, auditing, and error control.