Structured Data Extraction from Unstructured Documents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods fail to effectively extract structured data from unstructured electronic documents due to a lack of correspondence between data fields and their layout, especially in documents without a consistent template, leading to challenges in identifying and segmenting layout elements that correspond to structured data.
Innovation Solution
A method and system that perform layout analysis to generate layout structures, tag textual elements with content tags based on predefined heuristics, generate a content-layout model to associate tags with repetitive layout positions, and extract structured data by inferring missing tags, allowing for the identification and extraction of structured data from unstructured documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If geometric page layout analysis algorithms are used to segment pages into text blocks and image blocks, then the document can be divided into structural elements, but the correspondence between these layout elements and structured data fields cannot be reliably established in documents without consistent templates
Solution Approach 1:
The patent introduces an intermediary layer of semantic tags that bridge the gap between visual layout elements and structured data fields. Instead of directly mapping layout blocks to data fields, the system first tags elements with semantic meaning (e.g., 'invoice_number', 'total_amount') based on their visual characteristics and content, then uses these tags to establish reliable correspondences. This intermediary tagging system resolves the contradiction by adding a layer of interpretation that connects form to function.
Solution Approach 2:
The system changes parameters by considering multiple attributes of layout elements simultaneously - not just their geometric properties but also their textual content, positional relationships, and visual hierarchy. By analyzing multiple parameters (font size, position, content patterns) rather than single geometric features, the system can reliably identify which layout elements correspond to which structured data fields even in variable document formats.
2Adaptability or versatility
If a generic algorithm is used to segment documents, then it can process any document format, but it cannot systematically provide segmentation where found layout elements correspond to a single structured data field
Solution Approach 1:
The patent applies segmentation by breaking down the document analysis process into distinct stages: first segmenting the document into layout blocks, then segmenting those blocks into candidate elements, and finally segmenting candidates into confirmed structured data fields. This multi-level segmentation approach allows the generic algorithm to maintain versatility across document formats while achieving precision in field identification through progressive refinement at each segment level.
Solution Approach 2:
The system dynamically adapts its analysis depth and criteria based on the document type and detected patterns. For documents with consistent templates, the algorithm can use simpler, faster matching. For variable-format documents, it dynamically switches to more complex analysis involving multiple parameters and iterative refinement. This dynamic behavior enables both versatility across formats and precision in identification.
3Ease of operation
If layout analysis is performed to generate layout structures, then structural elements can be identified, but the lack of correspondence between these elements and structured data fields remains unresolved in unstructured documents
Solution Approach 1:
The system performs preliminary tagging of layout elements with semantic information before the actual structured data extraction process. By pre-identifying and tagging elements that likely contain specific data types (e.g., tagging a prominently displayed numeric field as 'total_amount'), the system preserves the correspondence information that would otherwise be lost. This preliminary action ensures that when structured data is extracted, the mapping information is already in place.
Solution Approach 2:
The system uses feedback mechanisms where the results of initial structured data extraction are fed back into the layout analysis process. If correspondence is successfully established for certain element types, this feedback reinforces the mapping rules for similar elements. If correspondence fails, the feedback triggers re-analysis with adjusted parameters. This feedback loop prevents loss of correspondence information by continuously refining the mappings.
Data Source
AI summary
This disclosure provides an exemplary method and system for extracting structured data from an unstructured textual document. According to an exemplary method, initially a layout analysis is performed resulting in one or more alternatives for grouping and ordering the page elements of interest. Next, the content of these page elements are tagged based on application-specific heuristics. Finally, a sequence-based method is applied to the tags for identifying repetitive contiguous patterns.