Benchmarking Dataset Generation for Contextual Document Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document management platforms and large language models are ineffective in accurately analyzing and summarizing electronic documents, leading to inaccurate or incorrect results due to the lack of mechanisms to assess the effectiveness of these models.
Innovation Solution
A system for generating benchmarking datasets using labeled and unlabeled electronic documents to evaluate and train large language models, by determining document types, generating generic and content-specific labels, and applying validation rules to ensure accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing large language models are used to process documents, then document analysis can be performed, but the results are inaccurate or incorrect due to lack of effectiveness assessment mechanism
Solution Approach 1:
The patent implements a feedback mechanism by generating benchmarking datasets with ground truth labels to evaluate and assess the effectiveness of large language models in document analysis tasks. This closed-loop system allows for continuous improvement of model accuracy through performance measurement and comparison against known correct answers.
Solution Approach 2:
The patent replaces manual document analysis with automated large language models, then substitutes the lack of assessment mechanism with an automated benchmarking system that uses pre-generated ground truth data to objectively evaluate model performance, replacing subjective or absent evaluation with systematic automated testing.
2Reliability
If document analysis is performed on large documents, then comprehensive analysis can be achieved, but the operation is compute-intensive and difficult
Solution Approach 1:
The patent generates benchmarking datasets using a subset of documents with detailed ground truth labels, rather than manually labeling all documents. This partial action approach creates sufficient evaluation data to assess model performance without the excessive computational cost of comprehensive manual analysis of all documents.
Solution Approach 2:
The patent performs preliminary generation of benchmarking datasets with ground truth labels before evaluating large language models. This pre-computed reference data enables efficient model assessment without requiring intensive computational resources during the evaluation phase, as the ground truth is already established.
3Measurement precision
If benchmarking datasets are generated with multiple label sets, then model evaluation accuracy is improved, but the data processing complexity increases
Solution Approach 1:
The patent segments the label generation process into two distinct components: generic labels that capture overall document characteristics and content-specific labels that capture detailed contextual information. This segmentation allows for systematic processing and comparison of different label types, improving evaluation accuracy while managing complexity through structured organization.
Solution Approach 2:
The patent creates a universal benchmarking framework that can evaluate multiple large language models using the same dataset structure with both generic and content-specific labels. This multi-functional system allows the same benchmarking infrastructure to assess different models across various document analysis tasks, improving evaluation precision without proportionally increasing processing complexity.
Data Source
AI summary
A method, a system, and a computer program product for generating a benchmarking dataset. A type of an electronic document is determined. Using the type of the electronic document, a first request to generate one or more labels for the electronic document is generated. Using a content of the electronic document, a second request to generate one or more labels for the electronic document is generated. The electronic document and the first and second requests are sent to a generative artificial intelligence (AI) model. The generative AI model generates one or more first labels for the electronic document based on the first request and one or more second labels for the electronic document based on the second request. Using one or more first and second labels, one or more labels for the electronic document are generated.