DOM Tree Partitioning for Web Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated information retrieval from dynamic web pages is challenging due to small variations in content, causing existing algorithms like XPath to fail, and machine learning solutions require expertise and are often inaccurate.
Innovation Solution
A method that partitions DOM trees into matched and unmatched paths, using a tree matching algorithm to identify similarities and a partial path matching approach to extract data from target web pages efficiently, reducing computational effort and improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If XPath algorithms are used for automated information retrieval, then extraction efficiency is improved, but accuracy deteriorates when small variations in web page content occur
Solution Approach 1:
The patent segments the DOM tree into matched paths and unmatched branches, allowing the algorithm to focus only on relevant portions of the tree that contain data of interest. This segmentation enables efficient processing while maintaining accuracy by ignoring variations in unmatched branches.
Solution Approach 2:
The patent applies different processing strategies to different parts of the DOM tree. Matched paths receive precise matching algorithms while unmatched branches are excluded from processing. This local differentiation allows the system to maintain high accuracy for critical data extraction while improving overall efficiency.
2Adaptability or versatility
If machine learning techniques are used for data extraction, then adaptability to dynamic web pages is improved, but system complexity and cost increase due to requiring expert knowledge
Solution Approach 1:
The patent enables the system to automatically adapt to dynamic web pages without requiring machine learning experts. The algorithm self-adjusts by identifying matched paths and unmatched branches, performing the adaptation function that would otherwise require complex machine learning models and expert knowledge.
3Measurement precision
If full DOM tree matching is performed, then comprehensive data extraction is achieved, but computational effort increases significantly
Solution Approach 1:
The patent extracts and isolates only the matched paths from the full DOM tree, removing unnecessary unmatched branches from processing. This extraction approach maintains comprehensive data extraction capability while significantly reducing computational effort by focusing only on relevant portions of the DOM tree.
Data Source
AI summary
Disclosed is a method of automatically extracting data from a target web page, comprising selecting (302) data in a source web page; determining (304) the respective DOM (document object model) trees of the source and target web page, and identifying the one or more nodes comprising the selected data in the source web page DOM tree; determining (306) matching paths in the respective DOM trees; for selected data in a node of an unmatched branch of the source web page DOM tree, identifying (308) the nearest matched path in the source web page; identifying (310) the unmatched branch nearest to the corresponding matched path in the target web page; determining (312) if said identified unmatched branch in the target web page DOM tree comprises a target node matching the selected data node; and if so: extracting (322) data from the target node if the mismatch between the respective unmatched branches does not exceed a predefined threshold. A computer program product and system implementing this method are also disclosed.