Meta-Path Graph Web Extraction for Adaptive HTML Change Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional web data extraction methods face scalability issues due to reliance on predefined templates and rigid rules, struggle with HTML syntax variations, and fail to discriminate between relevant and inconsequential changes on web pages.

Innovation Solution

A method and system using meta-path graphs to convert HTML data into an HTML graph, identify areas of interest, refine sub-structures with recurring patterns, and extract data items, while detecting significant changes and notifying users via a user interface.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional web data extraction methods use predefined templates and rigid rules, then data extraction can be performed systematically, but scalability is limited and maintenance overhead increases

Engineering Contradiction:

Improvedata extraction efficiencyVSAvoidmaintenance overhead

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent transforms static template-based extraction into a dynamic graph neural network approach. The system learns extraction patterns from data rather than following fixed rules, allowing the extraction logic to adapt automatically to different webpage structures without manual reconfiguration of templates or rules.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent replaces the mechanical template-matching system with a machine learning-based graph neural network. Instead of manually defining extraction rules and templates, the system automatically learns extraction patterns from webpage graphs, eliminating the need for manual feature engineering and rule definition.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Adaptability or versatility

If HTML parsing handles syntax variations and shortcuts, then web pages can be designed flexibly, but parsing errors and structure ambiguity increase

Engineering Contradiction:

ImproveHTML design flexibilityVSAvoidparsing accuracy

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent introduces an HTML graph as an intermediary representation between the raw HTML syntax and the data extraction process. The graph structure normalizes various HTML syntax forms into a unified representation, where elements are nodes and relationships are edges, making the extraction process robust to syntax variations while maintaining accuracy.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Loss of information

If web data extraction processes entire web pages, then all data can be captured, but processing resources increase and significant changes are harder to identify

Engineering Contradiction:

Improvedata capture completenessVSAvoidprocessing resource consumption

Core Design Contradiction:

Loss of informationVSUse of energy by moving object

Solution Approach 1:

The patent extracts and focuses on specific sub-structures within the HTML graph that are relevant to data extraction. By identifying and processing only the necessary sub-graphs rather than entire web pages, the system reduces computational resources while maintaining data capture effectiveness for the target information.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent segments the HTML graph into meaningful sub-structures based on the graph neural network's learned patterns. This segmentation allows the system to process and analyze specific relevant portions of the webpage independently, reducing overall processing requirements while capturing all necessary data through targeted extraction.

Inventive Principle:

Principle #1Segmentation

4Quantity of substance

If conventional methods extract all data from web pages, then comprehensive data is obtained, but discrimination between relevant and inconsequential changes becomes difficult

Engineering Contradiction:

Improvedata extraction volumeVSAvoidchange detection accuracy

Core Design Contradiction:

Quantity of substanceVSMeasurement precision

Solution Approach 1:

The patent applies different processing and attention levels to different parts of the HTML graph. The graph neural network learns to identify and focus on locally relevant sub-structures that contain meaningful data, while ignoring inconsequential elements. This local quality approach enables precise change detection by comparing only the relevant extracted data across different pages.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20250265305A1Method and System for Web Data Extraction Using Meta-Path Graph

Publication Date: 2025.08.21 S&P GLOBAL INC

AI summary

A computer-implemented method for web data extraction is provided. The method includes receiving an HTML file containing HTML data and converting the HTML data into an HTML graph. Elements in the HTML file are represented by nodes in the HTML graph and relationships among the elements are represented by meta-paths. The method includes generating feature sets for the nodes in the HTML graph and identifying areas of interest in the HTML graph based on the feature sets of the nodes. The method includes refining the identified areas of interest by segregating sub-structures having recurring patterns or sequences. The method includes extracting data items from the segregated sub-structures, storing the extracted data items and monitoring them over time for any significant updates.