Synthetic Document Generation for AI Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating training data for document-parsing AI systems are time-consuming and inefficient, particularly when dealing with a wide variety of forms and documents, as they often require manual parsing or reliance on pre-made configuration files.
Innovation Solution
A synthetic data generation system that creates training, validation, and testing data by generating simulated documents with specific characteristics, allowing for the efficient creation of diverse data variations and reducing the need for manual intervention or reliance on pre-existing databases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual parsing is used to create training data, then data quality and correctness are improved, but time consumption and labor effort increase significantly
Solution Approach 1:
The patent uses synthetic data generation to create copies of document structures with known ground truth annotations. Instead of manually parsing each document, the system generates synthetic documents with predefined layouts and automatically creates corresponding JSON annotations, eliminating manual parsing while maintaining data quality through controlled generation parameters.
Solution Approach 2:
The synthetic data generation system performs self-annotation by automatically creating structured JSON representations of generated documents. The system serves its own data annotation needs through programmatic generation, eliminating the need for external manual parsing while ensuring consistent and correct formatting across all training samples.
2Loss of time
If pre-made configuration files are used to parse documents, then time consumption is reduced, but parsing accuracy deteriorates when text cannot be extracted from PDFs
Solution Approach 1:
The system generates synthetic document images from structured data representations, creating perfect copies of document layouts with guaranteed extractability. By generating documents in a controlled manner rather than parsing existing PDFs, the system ensures that all text is properly formatted and extractable, eliminating the extraction failures that plague conventional approaches.
Solution Approach 2:
Instead of parsing documents to extract text and create annotations (forward approach), the system inverts the process by generating documents from structured data and automatically creating annotations during generation. This reverse approach ensures that all documents are created with known, correct annotations rather than attempting to extract and interpret text from potentially problematic PDFs.
3Quantity of substance
If pre-existing databases are used for training data, then data availability is improved, but data format quality deteriorates due to undesirable JSON formats
Solution Approach 1:
The system changes the generation parameters of training data by creating synthetic documents with customizable JSON schema configurations. Users can specify desired JSON output formats, data structures, and annotation styles during generation, allowing the system to produce data in exactly the required format rather than accepting pre-existing databases with incompatible or suboptimal structures.
Solution Approach 2:
The synthetic data generation system creates a universal framework that can generate training data in multiple JSON formats and structures by changing generation parameters. Rather than being limited to the formats of pre-existing databases, the system can adapt to various annotation schemas and output requirements, making the training data generation process versatile and format-agnostic.
4Adaptability or versatility
If configuration files are updated to handle new form variations, then adaptability is improved, but device complexity and maintenance effort increase
Solution Approach 1:
The system transitions from static configuration files to dynamic synthetic data generation. Instead of maintaining fixed configuration files that must be manually updated for each new form, the system uses parameterized generation processes that can adapt to new form variations by adjusting generation parameters programmatically, reducing the need for manual configuration updates.
Solution Approach 2:
The system creates synthetic copies of form variations through parameterized generation rather than maintaining separate configuration files for each form type. By generating documents programmatically with different parameters representing various form variations, the system eliminates the need to copy and modify configuration files for each new form, reducing maintenance complexity while maintaining adaptability.
Data Source
AI summary
A method of training a document parsing artificial intelligence (AI) system, the method includes configuring a PYTHON data structure for generating a simulated document for training the document parsing AI system and a JAVA data structure for generating a non-simulated document for training the document parsing AI system. The method includes training the document parsing AI system based on a generated word-processing format file and on a parsed JSON file for the simulated document made via the PYTHON and JAVA data structures. The method includes parsing, a received document, with the trained document parsing AI system to determine one or more characteristics associated with textual data written to the received document, and generating an output of the parsed received document.