Unstructured Document Structuring for Positional Information Preservation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Unstructured domain-specific documents, such as summary plan documents and technical specifications, are difficult to interpret and extract information from due to their complex and lengthy nature, often resulting in loss of positional information during conversion to plaintext files, making it challenging to map extracted information back to the original document.
Innovation Solution
A method and system that convert unstructured documents to structured documents using natural language processing engines and domain-specific ontologies, maintaining contextual information and linking extracted information to its original position, allowing for the generation of navigable structures and accurate information extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If unstructured documents are converted to plaintext files for information extraction, then information can be processed by NLP engines, but positional information is lost making it difficult to map extracted information back to the original document
Solution Approach 1:
The system performs preliminary action by converting the unstructured document to a structured format (such as HTML or XML with annotations) before information extraction, preserving positional information through character offsets, bounding boxes, or span annotations. This preliminary structuring enables both accurate information extraction and traceability back to the original document locations.
Solution Approach 2:
The patent introduces an intermediary structured document format that acts as a mediator between the unstructured original document and the NLP processing pipeline. This intermediary structure preserves positional information through metadata annotations while maintaining the content in a machine-processable format, thus solving the contradiction between information extraction capability and positional information preservation.
2Measurement precision
If domain-specific ontologies and NLP engines are used to extract precise information, then measurement precision improves, but device complexity increases
Solution Approach 1:
The system segments the complex information extraction task into multiple components: document preprocessing, structure annotation, NLP processing, and result mapping. By dividing the workflow into discrete, manageable segments with specialized handling for each aspect, the system achieves high extraction accuracy while managing complexity through modular architecture.
Solution Approach 2:
The patent applies parameter changes by transforming the document representation from unstructured text to structured format with explicit positional parameters (character offsets, bounding boxes). This parameter transformation enables precise information extraction through standardized data structures while managing system complexity through consistent parameter handling across the processing pipeline.
Data Source
AI summary
Aspects of the present disclosure describe techniques for generating a machine learning model for extracting information from textual content. The method generally includes receiving a training data set including a plurality of documents having related textual strings. A relevancy model is generated from the training data set. The relevancy model is generally configured to generate relevance scores for a plurality of words extracted from the plurality of documents. A knowledge graph model illustrating relationships between the plurality of words extracted from the plurality of documents is generated from the training data set. The relevancy model and the knowledge graph model are aggregated into a complimentary model including a plurality of nodes from the knowledge graph model and weights associated with edges between connected nodes, wherein the weights comprise relevance scores generated from the relevancy model, and the complimentary model is deployed for use in analyzing documents.