Contract Ingestion Using OCR, Embeddings, and Segmented LLM Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large contract documents is challenging due to their complexity, length, interrelated content, variability in format and structure, and the need to present extracted data in a meaningful way for analysis or integration into other systems.
Innovation Solution
An application that interprets and parses contract documents, using optical character recognition (OCR) for image-based files, segments text into manageable chunks, creates vector embeddings to maintain context, and utilizes large language models (LLMs) to categorize and extract relevant data, presenting it in a user-friendly graphical interface or integrating it into enterprise resource planning systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional methods are used to process large contract documents, then the system can handle basic text extraction, but the processing time and computational resources increase significantly due to the documents' complexity and length
Solution Approach 1:
The patent segments the contract document into multiple manageable sections or pages, processing each segment independently through the LLM. This segmentation allows the system to handle large documents without overwhelming computational resources, improving processing speed while maintaining accuracy in extracting information from complex, lengthy contracts.
2Reliability
If the entire contract document is processed at once, then comprehensive understanding is achieved, but computational resources are wasted and processing efficiency decreases
Solution Approach 1:
The system divides the contract into segments and processes them individually, maintaining comprehensive understanding through contextual analysis of each segment while reducing overall computational resource consumption. This approach ensures accurate extraction of interrelated information without wasting resources on processing the entire document at once.
Solution Approach 2:
The patent extracts only the relevant information and key insights from each contract segment using LLM analysis, rather than processing every detail uniformly. This extraction approach maintains reliability in capturing important interrelated information while significantly reducing computational resource expenditure on unnecessary processing.
3Ease of manufacture
If standard processing methods are used for nonstandard contract pages, then processing simplicity is maintained, but extraction accuracy and handling of varied formats deteriorates
Solution Approach 1:
The patent implements dynamic processing where the system adapts its extraction methodology based on the specific characteristics of each contract page or segment. For nonstandard pages, the LLM analyzes the content structure and applies appropriate extraction techniques, maintaining processing simplicity through automated adaptation while significantly improving data extraction accuracy across varied formats.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Efficiently extracts and presents meaningful data from complex contract documents, ensuring accurate categorization and integration into downstream systems, thereby reducing resource waste and enhancing data utility.
Implementation Method 1
If the document already contains recognizable text, then the application can read and analyze the text directly. For any images containing text, the application can perform image text extraction, also referred to as optical character recognition ('OCR').
Data Source
AI summary
Systems and methods are described for processing contract documents. In an example, an application can identify text in an image-based data file that contains text of a contract. The application can segment text in the document into smaller units. The application can then create vector embeddings for each unit and feed the vector embeddings into a first large language model (“LLM”) that classifies each unit. Based on the classification, the application can retrieve a prompt template for each unit and feed the vector embeddings with their corresponding prompt into a second LLM. The second LLM can extract specific data from each unit based on the prompt template. The output from the second LLM can then be converted into a useable format, such as a web page or passed to another system.