Multimodal Technical Document Extraction for Searchable Knowledge Bases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing OCR models fail to extract multimodal content such as tables, drawings, and mathematical expressions from technical documents, leading to inefficiencies in building knowledge bases and requiring extensive human effort.
Innovation Solution
A system and method using machine learning models, including neural networks, to classify, locate, and extract multimodal content like text, tabular columns, and engineering drawings from both searchable and non-searchable documents, generating a knowledge graph for efficient representation and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If OCR models are used to convert non-searchable documents to searchable documents, then text content can be extracted, but multimodal content such as tables, drawings, and mathematical expressions cannot be extracted
Solution Approach 1:
The patent segments the document processing task into multiple specialized components: text recognition module, table structure recognition module, formula recognition module, and drawing recognition module. Each module is optimized for specific content types, allowing comprehensive extraction of multimodal content while maintaining high accuracy for each modality
Solution Approach 2:
The patent creates a universal document processing system that handles multiple content types (text, tables, formulas, drawings) through a single integrated platform. The system uses multiple recognition models that work together to provide multi-functional capability, extracting various modality contents from technical documents in one pass
2Measurement precision
If manual review is performed to deduce useful information from technical documents, then accurate information extraction is achieved, but extensive human effort and time are required
Solution Approach 1:
The patent replaces the manual mechanical review process with automated recognition systems including OCR for text, table structure recognition algorithms, formula recognition models, and drawing analysis systems. This substitution maintains high extraction accuracy while dramatically increasing processing speed and reducing human effort
Solution Approach 2:
The system enables self-service extraction of multimodal content through automated recognition models that independently identify and extract text, tables, formulas, and drawings without human intervention. The knowledge base is automatically populated from technical documents, reducing dependency on manual processing
3Reliability
If technical documents are scanned and stored in non-searchable formats, then document preservation is achieved, but the documents cannot be searched or processed efficiently
Solution Approach 1:
The patent performs preliminary processing of scanned documents by applying multiple recognition models to extract text, tables, formulas, and drawings before the documents need to be searched. This advance processing converts non-searchable images into structured, searchable data while preserving the original scanned document for reference
Solution Approach 2:
The patent introduces an intermediary processing layer between the scanned document and the search function. This layer includes recognition models that translate image content into machine-readable formats, enabling searchability without compromising the original document preservation
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
A system (100), apparatus (105) and method for providing access to multimodal content in a technical document is disclosed herein. The method comprises receiving, by a processing unit (120 receiving, by a processing unit (120), a request for providing access to multimodal content in the technical document from a source (110). The multimodal content in the technical document is further classified into one or more pre-defined categories using one or more machine learning models. Further, a location of the classified multimodal content in the technical document is determined. The classified multimodal content is extracted from the determined location in the technical document using one or more trained functions. The extracted multimodal content is further represented in a searchable format. The representation of the extracted multimodal content is stored in a database (195). Further, access to the representation of the multimodal content in the searchable format stored in the database (195) is provided.