Web Page Data Extraction via Object Model Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Unstructured data on web pages makes it difficult to extract and compare information across multiple pages, as it is formatted for reading rather than analysis, leading to challenges in data sorting and classification for search engines.
Innovation Solution
The method involves creating a tree representation of web pages, comparing them to identify format similarities, and applying labels to extract data from matching pages into a structured format, using a computer-implemented system with a template generator, labeler, comparison module, and data extractor to store data in a structured database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is formatted for reading like a book, then readability is improved, but data extraction and analysis become difficult
Solution Approach 1:
The patent segments web page content into structured data fields by identifying and extracting specific information elements (such as pricing data, product details) from the unstructured HTML format. This segmentation allows the data to be separated from the formatting, enabling both readability preservation and efficient data extraction for analysis.
2Loss of information
If data is spread across multiple web pages, then information completeness is improved, but data comparison and understanding become difficult
Solution Approach 1:
The patent merges data from multiple web pages into a unified structured format by extracting information elements across pages and consolidating them into a single dataset. This combining approach maintains the completeness of information distributed across multiple sources while enabling easy comparison and analysis of the consolidated data.
3Ease of operation
If each good appears on a separate web page, then page organization is improved, but price comparison between goods becomes difficult
Solution Approach 1:
The patent copies pricing data and other information elements from individually organized product pages into a centralized structured database. This copying mechanism preserves the organizational structure of having each good on its own page while creating a consolidated view that enables easy price comparison across all products without requiring users to navigate multiple pages.
Data Source
AI summary
Systems and techniques for extracting data from unstructured documents are described. One such method involves assigning one or more labels to one or more nodes in a first object model of a first web page; comparing a second object model of a second web page to the first object model; if the first object model matches the second object model to a determined degree, extracting from the second web page data associated with nodes in the second object model that match labeled nodes in the first object model; and providing the extracted data for storage in a structured database in a manner associated with the labels.