Correlation-Based Data Extraction from Markup Documents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for extracting information from web pages are unreliable and require substantial processing, as they often rely on regular expression matching or structural information that may not generalize across similar web pages, leading to inconsistent data extraction.
Innovation Solution
The method combines information from multiple markup language documents of similar structure to generate extraction rules, using rule clustering to identify and merge rules with high correlation, thereby improving the reliability of data extraction and accommodating structural differences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If regular expression matching is used for information extraction, then extraction can be automated, but reliability deteriorates and processing requirements increase substantially
Solution Approach 1:
The patent replaces mechanical regular expression matching with a correlation-based system that uses structural graph representations and correlation analysis to identify relationships between nodes in markup language documents, achieving more reliable extraction without substantial processing overhead
Solution Approach 2:
The patent introduces an intermediary correlation analysis layer between the document structure and extraction rules. By calculating correlations between node pairs and using these to generate extraction rules, the system achieves reliable automated extraction without relying on fragile regular expressions
2Measurement precision
If structural information is used to improve extraction accuracy, then measurement precision improves, but device complexity increases
Solution Approach 1:
The patent extracts only the essential structural information needed for correlation analysis from the full document structure. By focusing on node pairs and their relationships rather than processing the entire document structure, the system achieves high extraction accuracy with reduced processing complexity
Solution Approach 2:
The patent applies partial action by using correlation analysis only for specific node pairs and relationships rather than analyzing every possible structural element. This selective approach maintains extraction accuracy while significantly reducing the complexity of processing
3Ease of manufacture
If extraction rules are generated from a single sampled web page, then ease of manufacture improves, but reliability deteriorates due to inability to generalize
Solution Approach 1:
The patent merges multiple extraction rules generated from different sampled web pages into a unified correlation-based extraction system. By combining rules from multiple samples and using correlation analysis to identify consistent patterns, the system achieves both ease of generation and high reliability across similar pages
Solution Approach 2:
The patent creates a universal extraction system that can handle multiple web page structures through correlation analysis. The correlation-based rules generated from sampled pages generalize to handle variations in similar pages, making the system multi-functional and reliable across different instances
Data Source
AI summary
An extraction-rule generation and training system uses information obtained from multiple markup language documents (e.g. web pages) of similar structure to generate an extraction rule for extracting datapoints from markup language documents. By using information extracted from multiple documents of similar structure, including information regarding correlations between such documents, the method produces data extraction rules that provide improved datapoint extraction reliability. Where the structures of two or more documents are not sufficiently similar, the system maintains separate extraction rules for the same datapoint, and applies these separate extraction rules in combination to particular markup language documents to extract the datapoint.