MiBAT Data Record Extraction Using Anchor Trees
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data record extraction techniques assume uniform structure in user-generated content (UGC), which is not applicable to unstructured formats, leading to inefficiencies in extracting data records from documents containing diverse UGC.
Innovation Solution
The MiBAT process distinguishes structured and unstructured parts of data records, representing documents as trees to locate anchor trees based on domain constraints, determining record boundaries, and extracting data records without assuming similar overall structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data record extraction techniques assume uniform structure in user-generated content, then extraction process is simplified, but extraction accuracy deteriorates due to inability to handle unstructured formats
Solution Approach 1:
The patent segments data records into structured portions (with uniform patterns like author, publication date) and unstructured UGC portions (free-format content). This segmentation allows the system to apply uniform extraction rules to structured parts while handling unstructured parts separately, resolving the contradiction between process simplicity and extraction accuracy.
Solution Approach 2:
The patent applies different extraction strategies to different parts of data records: uniform structure assumptions are applied locally to structured portions (metadata, headers) while flexible unstructured handling is applied locally to UGC portions. This local differentiation maintains simplicity where applicable while ensuring accuracy where needed.
2Measurement precision
If data record extraction techniques handle diverse unstructured UGC formats, then extraction accuracy is improved, but extraction process complexity increases
Solution Approach 1:
By dividing data records into structured and unstructured portions, the patent reduces overall process complexity. The structured portion can be extracted using simple uniform rules, while only the unstructured portion requires complex handling, thus improving accuracy without proportionally increasing overall complexity.
Solution Approach 2:
The patent extracts and isolates the unstructured UGC portion from the overall data record structure. This extraction allows the system to apply specialized handling only where needed, rather than applying complex unstructured processing to entire records, thereby improving accuracy while controlling complexity.
3Productivity
If traditional extraction methods are used on unstructured UGC, then processing speed is maintained, but extraction relevance deteriorates due to inability to distinguish structured from unstructured parts
Solution Approach 1:
The patent extracts the unstructured UGC portion separately from structured data. This allows efficient processing of structured parts using fast uniform rules while applying more sophisticated relevance-based extraction only to the unstructured portion, maintaining overall processing speed while improving extraction relevance.
Solution Approach 2:
The patent applies different processing qualities to different parts: fast uniform processing for structured portions and more thorough relevance-based processing for unstructured portions. This local differentiation maintains high processing speed for the majority of structured data while ensuring high relevance for unstructured content.
Data Source
AI summary
Described herein are techniques for extracting data records containing user-generated content from documents. The documents may be processed into document trees in which sub-trees represent the data records of the document. Domain constraints may be used to locate structured portions of the document tree. For example, anchor trees may be located as being sets of sibling sub-trees with similar tag paths that contain the domain constraints. The anchor trees may then be used to determine a record boundary (e.g., the start offset and length) of the data records. Finally, the data records may be extracted based on the anchor trees and the record boundaries.