Independent Tabular and Text Retrieval for PDF Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing RAG-based systems struggle with accurately extracting tabular information from PDF documents due to intertwined text and tables, leading to semantic meaning loss and increased computational inefficiencies, especially when feeding large language models, which results in inaccurate responses and higher energy and operational costs.
Innovation Solution
The method involves separating tabular information from document text using markdown language tags during the OCR process, allowing the large language model to process both independently, thereby improving index quality and reducing computational effort.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If tabular information and document text are processed together in existing RAG-based systems, then comprehensive data extraction is achieved, but computational efficiency deteriorates and accuracy decreases due to semantic meaning loss
Solution Approach 1:
The patent segments the document processing into two independent streams: one for tabular information and one for textual content. Tables are extracted and processed separately from the main text body, allowing each to be handled with appropriate methods. This segmentation prevents semantic confusion between tabular data structures and narrative text, thereby improving extraction accuracy while reducing computational overhead by avoiding unnecessary processing of irrelevant content.
Solution Approach 2:
The patent extracts tabular information from the document and removes it from the main text stream before processing. By taking out tables and handling them through a dedicated table processing pipeline, the system avoids the semantic meaning loss that occurs when tables are intertwined with text in traditional RAG systems. This extraction approach maintains computational efficiency by preventing the model from processing redundant or confusing combined content.
2Reliability
If all document portions are fed to the large language model, then complete information availability is achieved, but operational costs and energy consumption increase
Solution Approach 1:
The patent applies partial action by selectively processing only the portions of the document that are most relevant to the query. Instead of feeding the entire document to the LLM, the system identifies and extracts specific tables and text segments that contain the necessary information. This partial processing approach maintains information availability for accurate responses while significantly reducing the computational energy required compared to processing complete documents.
Solution Approach 2:
The patent extracts only the necessary tabular information and relevant text portions required to answer the specific query, rather than processing all document content. By taking out and processing only the essential portions, the system ensures that sufficient information is available for reliable responses while minimizing energy consumption and operational costs associated with processing unnecessary content.
3Loss of information
If tabular information is intertwined with text in PDF documents, then complete document representation is achieved, but semantic meaning is lost and processing complexity increases
Solution Approach 1:
The patent segments the document into distinct components: tabular information and textual content. By separating these elements during the extraction phase, the system preserves the semantic meaning of each component type. Tables maintain their structured data relationships, while text retains its narrative flow. This segmentation eliminates the semantic confusion that arises when tables and text are intertwined, thereby preserving meaning while reducing processing complexity.
Solution Approach 2:
The patent transitions from a two-dimensional mixed layout (where tables and text are intertwined on the same page structure) to a multi-dimensional processing approach. Tables are extracted into their own separate data structure dimension, distinct from the text dimension. This dimensional separation allows the system to preserve the inherent structure and semantics of tabular data while maintaining text processing independence, thereby reducing overall processing complexity.
Data Source
AI summary
A system for extracting a number of data elements from one or more data sources. The system may separate the text from the tables in a document, such that only the table data may be sent to the large language model (LLM), when the LLM only needs to review the table data. The system may include converting a PDF to text, and separating the tables form the document text using markdown language from converting the PDF. The system may form table chunks and text chunks, index the chunks using a vector embedding and store a chunk identifier, document identifier, and or a page identifier with the chunk to provide result traceability. The system may, in response to a prompt, retrieve and send the targeted table chunks or text chunks to the LLM to extract the data elements. The system populates an ontological data store based on the LLM response.